woods 1.6.1 → 2.0.0.beta2

data/docs/PUBLISHED_INDEX.md

@@ -0,0 +1,213 @@
+# Reading a published index from Ruby
+
+`Woods::PublishedIndex` is the stable, read-only API for tools that are not MCP clients: RuboCop cops, CI gate scripts, and the `woods:check:*` tasks. It needs no Rails, opens one published generation, and never moves off it for the life of the reader.
+
+```ruby
+require 'woods/published_index'
+
+index = Woods::PublishedIndex.new(Rails.root.join('tmp/woods'))
+index.generation_number            # => 42
+index.unit('Order')                # => { "type" => "model", "identifier" => "Order", ... }
+index.units(type: 'model')         # => index entries with a "type" key
+index.edges(via: 'belongs_to')     # => [{ from: "Comment", to: "Post", via: "belongs_to", through: nil, through_db: nil, disable_joins: false }]
+index.dependents_of('Post')        # => ["Comment", "PostsController"]
+index.table_database_map           # => { "orders" => "primary", "events" => "analytics" }
+index.external_dependency_checksum # => "9f2c81ad..." (changes on every publish)
+index.close
+```
+
+Prefer the block form so the lock in the next section is always released:
+
+```ruby
+Woods::PublishedIndex.open(Rails.root.join('tmp/woods')) do |index|
+  index.table_database_map
+end
+```
+
+## One generation, pinned for the reader's whole life
+
+Unlike `Woods::MCP::IndexReader`, a `PublishedIndex` never refreshes between calls. It resolves one generation at `.new`/`.open` time and every fact it returns, `units`, the table map, `generation_number`, `external_dependency_checksum`, comes from that one generation for as long as the reader is open. There is no `reload` and no auto-refresh: open a new reader to see a later publish.
+
+Opening a numbered generation acquires that generation's `manifest.json` lock through the same retention protocol `Woods::PayloadStore#prune` respects (a shared advisory `flock`, taken read-only). The lock is held for the reader's whole lifetime, not just one read, so a publish's retention pass cannot remove the payload out from under a long-lived reader: `PayloadStore#prune` takes a non-blocking exclusive lock on the same file before removing a generation, and skips any generation it cannot lock. `#close` releases it (`ensure` in the block form), so a reader that is done reading stops holding the generation open.
+
+**Not thread-safe.** A `PublishedIndex` instance is meant for one script or cop process reading one generation; it keeps no mutex around its lock file or its underlying `Woods::MCP::IndexReader`. Give each thread its own reader rather than sharing one.
+
+## API
+
+| Method | Returns | Notes |
+|---|---|---|
+| `.new(index_dir, generation: nil)` | reader, holding a lock | `generation:` pins a published `payloads/gen-N`; omit for the currently published one |
+| `.open(index_dir, generation: nil) { |index| ... }` | the block's value | releases the lock in `ensure`, on return or exception |
+| `.available_generations(index_dir)` | `Array<Integer>` | Published generations, ascending: see below |
+| `#generation_number` | Integer | 0 for an index written flat (pre-2.0 layout) |
+| `#close` | nil | Releases the retention lock; safe to call more than once |
+| `#unit(identifier, type: nil)` | Hash or nil | The unit JSON, string keys; see the collision note below |
+| `#units(type: nil)` | `Array<Hash>` | `_index.json` entries plus `"type"` |
+| `#edges(via: nil)`, `#each_edge` | `Array<Hash>` | Every forward edge with `through`, `through_db`, and `disable_joins`; an identifier shared by more than one type contributes one edge per owning type, never folded into a single deduplicated entry |
+| `#dependents_of(identifier, via: nil)` | `Array<String>` | Reverse index |
+| `#table_database_map` | `Hash` | Model table to database; empty on Rails 6.0 extractions |
+| `#external_dependency_checksum` | String | SHA-256 of the pinned payload's `manifest.json` |
+
+The reader wraps `Woods::MCP::IndexReader` with `auto_refresh: false`; the unit and graph shapes are the ones documented in [Extractor reference](EXTRACTOR_REFERENCE.md#extractedunit-field-reference).
+
+### `#unit`: an identifier shared across types
+
+`Woods::MCP::IndexReader#find_unit` keys its identifier map on identifier alone. If two type directories both list the same identifier (a model and a service both named `Foo`, for example), whichever type sorts last in `Woods::MCP::IndexReader::TYPE_DIRS` silently wins, and `unit(identifier)` returns that one. Pass `type:` to read a specific type's unit file directly and skip the collision entirely; `#table_database_map` always does this internally (`type: 'model'`), so a same-named non-model unit can never shadow a model's `table_name`/`database`.
+
+### `available_generations`: published means published
+
+A generation is listed only when both hold:
+
+* its number is at or below the pointer `generation.json` currently names;
+* its `payloads/gen-N` directory holds a `manifest.json`.
+
+A directory numbered above the pointer (a payload built but never bumped to) and a directory missing its manifest (an interrupted or corrupted publish) are never listed, and `.new(index_dir, generation: N)` raises `ArgumentError` for either. Retention itself is bounded by `WOODS_PAYLOAD_RETENTION` (default 3).
+
+A *missing* `generation.json` is not an error: it means a flat (pre-2.0) index, generation 0. A `generation.json` that **exists but will not parse** is different, a corrupt install, not an empty index, so `.available_generations` and `.new`/`.open` raise `Woods::PublishedIndex::CorruptPointerError` naming the file's path instead of silently reporting zero published generations.
+
+### Durability: the pointer is the commit point
+
+The pointer is not only what makes a generation *visible*, it is what makes it *durable*.
+
+Payload files are written through `Woods::AtomicFile.write` with `durable: false`: a temp file, a chmod to the final mode, and a rename, but no `fsync`. Immediately before `generation.json` is written, the writer flushes the whole payload directory once with `Woods::AtomicFile.sync_directory_tree`, and then writes the pointer durably.
+
+So the contract a reader can rely on is:
+
+> When `generation.json` is durable, every file in the payload it names is durable.
+
+What that gives up is an individual payload file being durable *before* the pointer exists. Nothing reads a payload file in that window: every reader resolves through the pointer, and a crash there leaves an unreferenced partial payload that the next run prunes. The saving is large, since two forced flushes per file is about 8.9ms each on btrfs, or 71s for 8000 units against 1s for one flush.
+
+`sync_directory_tree` tries `syncfs(2)` through Fiddle, then `sync -f <dir>`, then a bare `sync`, then an `fsync` on every file in the tree. The last resort is what keeps this honest: the chain never silently does nothing, it only gets slower.
+
+A file whose readers do **not** resolve through the pointer keeps its own `fsync`: `generation.json` itself, the watch daemon's `watch_status.json`, the update check cache, the Obsidian and Unblocked exports, Notion sync state, temporal snapshots, `checkpoint.json`, and MCP task records. Set `durable_payload_writes = true` to pay the per-file `fsync` on payload files as well; it cannot disable the publish flush.
+
+## Keying a RuboCop cache on the index
+
+RuboCop caches offenses per file and invalidates the cache when a cop's `external_dependency_checksum` changes. `rubocop-rails` uses this to re-run schema-aware cops when `db/schema.rb` changes. The same pattern works against Woods: the pinned payload's `manifest.json` is rewritten through `AtomicFile` on every publish, so its digest is the checksum.
+
+### Worked example: `Multidb/ForeignKeyAcrossDatabases`
+
+The cop flags an `add_foreign_key` in a migration whose two tables resolve to different databases. The table-to-database map comes from the index (`metadata.database` on model units), so the cop never boots Rails. A `:through` association's edge carries the same join rule under `through_db`: the through model can resolve to a database that differs from either end, so a check built on `#edges` rather than `#table_database_map` needs that value to catch the same class of cross-database join.
+
+```ruby
+# lib/rubocop/cop/multidb/foreign_key_across_databases.rb
+# frozen_string_literal: true
+
+require 'woods/published_index'
+
+module RuboCop
+  module Cop
+    module Multidb
+      # Flags `add_foreign_key :from, :to` when the two tables live in
+      # different databases. MySQL and PostgreSQL both refuse the constraint
+      # at the database level; catching it in review is cheaper.
+      class ForeignKeyAcrossDatabases < Base
+        MSG = 'Foreign key from `%<from>s` (%<from_db>s) to `%<to>s` (%<to_db>s) crosses databases.'
+        RESTRICT_ON_SEND = %i[add_foreign_key].freeze
+
+        def_node_matcher :foreign_key_tables, <<~PATTERN
+          (send nil? :add_foreign_key ${sym str} ${sym str} ...)
+        PATTERN
+
+        def on_send(node)
+          foreign_key_tables(node) do |from_node, to_node|
+            from = from_node.value.to_s
+            to = to_node.value.to_s
+            from_db = table_databases[from]
+            to_db = table_databases[to]
+            next if from_db.nil? || to_db.nil? || from_db == to_db
+
+            add_offense(node, message: format(MSG, from: from, from_db: from_db, to: to, to_db: to_db))
+          end
+        end
+
+        # RuboCop re-runs the cop on every file when this changes, and the
+        # pinned manifest changes on every Woods publish.
+        def external_dependency_checksum
+          index.external_dependency_checksum
+        rescue ArgumentError
+          'no-woods-index'
+        end
+
+        private
+
+        def table_databases
+          @table_databases ||= index.table_database_map
+        rescue ArgumentError
+          {}
+        end
+
+        def index
+          @index ||= Woods::PublishedIndex.new(File.join(Dir.pwd, 'tmp/woods'))
+        end
+      end
+    end
+  end
+end
+```
+
+Register it in `.rubocop.yml` with `require: ./lib/rubocop/cop/multidb/foreign_key_across_databases` and enable `Multidb/ForeignKeyAcrossDatabases` for `db/migrate/**/*.rb` and `db/*_migrate/**/*.rb`.
+
+Keep the index current in CI so the cop sees the same generation the app runs on:
+
+```ruby
+# config/ci.rb (Rails 8.1)
+step "Woods: refresh", "bin/rails woods:incremental"
+step "Style", "bin/rubocop"
+```
+
+A memoized cop instance holds its `PublishedIndex` (and its retention lock) for the process's lifetime; RuboCop runs each cop once per process, so there is no explicit `close` call above.
+
+## Comparing two generations
+
+`generation:` pins a retained payload. Two readers over two generations are the base for [`woods:check:moved_messages`](#moved-message-check), and for any check that asks "what changed between these two publishes".
+
+```ruby
+before = Woods::PublishedIndex.new(dir, generation: 41)
+after = Woods::PublishedIndex.new(dir, generation: 42)
+added = after.units.map { |u| u['identifier'] } - before.units.map { |u| u['identifier'] }
+before.close
+after.close
+added
+```
+
+Retention is `WOODS_PAYLOAD_RETENTION` generations (default 3). Raise it on a CI runner that compares against an older baseline, or the older side of the comparison may no longer be published.
+
+## Moved-message check
+
+`woods:check:moved_messages` opens two retained generations through `Woods::PublishedIndex` (block form, so both readers' retention locks always release) and reports every public method name that looks like it moved from one unit to another while a `:test_coverage` edge did not follow.
+
+Each row is a **candidate move into a unit without mapped tests**, never a proven coverage loss: the check matches on method name and kind alone, so two unrelated methods that happen to share both look identical to a real move. Reported only when the source unit was covered before and the destination is not covered after; a method that was never covered is not a regression this check owns. `WOODS_CHECK_STRICT=1` exits 1 on any finding, but the check stays heuristic either way, strict mode changes the exit code, not the confidence of a row.
+
+```bash
+bin/rails woods:check:moved_messages              # previous retained generation vs the published one
+bin/rails "woods:check:moved_messages[41,42]"     # explicit generations
+WOODS_CHECK_STRICT=1 bin/rails woods:check:moved_messages   # exit 1 on findings, for CI (still heuristic)
+WOODS_CHECK_JSON=1 bin/rails woods:check:moved_messages     # also print findings as JSON
+```
+
+With no `[from,to]` given, the generations default to the latest two from `available_generations` (a pure function, `Woods::Checks::GenerationResolution`, picks them); an index with fewer than two retained generations exits 1 with a message naming `WOODS_PAYLOAD_RETENTION` as the fix. A `generation.json` that will not parse raises `Woods::PublishedIndex::CorruptPointerError`, named by the task before it propagates, rather than being swallowed as "no findings".
+
+### Which key holds which method list
+
+The check reads three metadata keys and normalizes them to one `[name, kind]` shape before matching, so an instance method never collides with a class method of the same bare name:
+
+| Extractor family | Key | Shape |
+|---|---|---|
+| Services, POROs, managers, decorators, policies, validators, concerns, Pundit policies, lib units | `public_methods` | Regex-extracted; a class method is a bare name **prefixed with `self.`** (`"self.build"`), an instance method is bare (`"run"`) |
+| Services, POROs, managers, decorators, policies, validators, concerns, lib units | `class_methods` | Regex-extracted `def self.foo` names, prefix already stripped |
+| Concerns, models | `instance_methods` | Bare names; models get this from runtime introspection (`instance_methods(false)`, filtered), a wider list than the regex-based extractors produce |
+| Phlex components, ViewComponents | `public_methods` | Runtime introspection (`public_instance_methods(false)`); always bare, since these extractors only ever report instance methods |
+
+Model units carry both `instance_methods` and `class_methods` from runtime introspection, no `public_methods` key, and it is the widest of the method lists: every instance and class method the model responds to, not just the ones a source-regex can see.
+
+| Situation | Reported |
+|---|---|
+| `total` left `Checkout` (covered) and appeared in `Pricing` (not covered) | yes |
+| Same move, `Pricing` covered by a test_mapping unit | no |
+| `total` was never covered before the move | no |
+| Method renamed in place | no (nothing gained it) |
+| An instance method and a same-named class method swap units | no (kind mismatch) |
+| A `self.build` public method reappears as a `class_methods` entry named `build` | yes, `kind: :class` (normalized to the same signature) |
+
+Temporal snapshots (`snapshot_diff`, `unit_history`) store content hashes only, so they cannot see a moved method; that is why the check reads payload generations. Keep at least two retained generations (`WOODS_PAYLOAD_RETENTION`, default 3).

data/docs/README.md

@@ -0,0 +1,94 @@
+# Woods documentation
+
+Woods extracts runtime-accurate Rails context and serves it to coding agents through MCP. Start with the task you are trying to complete; you do not need to read the documentation in order.
+
+## What are you trying to do?
+
+| Task | Start here | You will finish with |
+|---|---|---|
+| Install Woods in a Rails app | [Getting started](GETTING_STARTED.md) | A validated codebase index with the 14 packaged-default tools connected |
+| Ask an agent to install or configure Woods | [Agent setup runbook](AGENT_SETUP.md) | A safe, reviewable install with an agent handoff report |
+| Configure an MCP client or Docker path | [MCP servers](MCP_SERVERS.md) | A working Index Server and, if authorized, an optional Console Server |
+| Use Woods tools as an agent | [Agent guide](AGENT_GUIDE.md) | A repeatable query workflow for code context, flows, and blast radius |
+| Keep the index current automatically | [Watch daemon](WATCH_DAEMON.md) | A resident development process that catches up changes and republishes the index |
+| Upgrade from Woods 1.x | [Upgrade to Woods 2.0](UPGRADING_TO_2.md) | A backed-up, re-indexed, verified v2 installation |
+| Diagnose an error | [Troubleshooting](TROUBLESHOOTING.md) | Symptom-to-cause checks for extraction, MCP, embeddings, storage, and Docker |
+| Contribute to Woods | [Contributing](../CONTRIBUTING.md) | A tested change with synchronized docs and plugin guidance |
+
+Claude Code users can optionally install the packaged setup/configuration
+workflows with `/plugin marketplace add lost-in-the/plugins`, followed by
+`/plugin install woods-plugin@lost-in-the-plugins`. Woods and its MCP servers
+do not otherwise depend on Claude Code; other clients use the same runbooks
+and stdio or Streamable HTTP endpoints directly.
+
+## First-time setup
+
+- [Getting started](GETTING_STARTED.md): install, review generated configuration, remove the unused legacy application migration unless an older/custom integration requires it, extract, validate, and connect the Index Server.
+- [Agent setup runbook](AGENT_SETUP.md): the same result with version checks, repository safety, approval boundaries, and a copyable agent prompt.
+- [Docker setup](DOCKER_SETUP.md): extraction inside the container, container-first MCP launch, optional host launch, and path translation.
+- [Configuration reference](CONFIGURATION_REFERENCE.md): every supported option, default, and environment variable.
+- [Backend matrix](BACKEND_MATRIX.md): choose structural-only, local Ollama, pgvector/OpenAI, Qdrant/OpenAI, or shared-filesystem deployment.
+
+## MCP and agents
+
+- [MCP servers](MCP_SERVERS.md): choose the pre-extracted Index Server or live-data Console Server; configure clients; understand the callable 14 and 9/11 tool surfaces.
+- [Agent guide](AGENT_GUIDE.md): start with `woods_status`, discover with `search`, inspect with `lookup`, and follow dependencies or flows.
+- [MCP tool cookbook](MCP_TOOL_COOKBOOK.md): scenario-based calls with parameters and expected response shapes.
+- [Console MCP setup](CONSOLE_MCP_SETUP.md): Console transports, blocked tables, credential scanning, redaction, SQL validation, and production safeguards.
+- [MCP HTTP transport](MCP_HTTP_TRANSPORT.md): shared/remote Index Server transport, authentication, origins, and protocol details.
+- [MCP worktree setup](MCP_WORKTREE_SETUP.md): register Woods correctly when agents work in linked git worktrees.
+
+## Index lifecycle
+
+- [Retrieval guide](RETRIEVAL_GUIDE.md): configure embeddings and understand semantic retrieval, ranking, and token budgets.
+- [Embedding models](EMBEDDING_MODELS.md): choose and size local Ollama models.
+- [Upgrade to Woods 2.0](UPGRADING_TO_2.md): identifier changes, atomic payloads, durable-store reconciliation, and rollback.
+
+## Reference
+
+- [Why Woods](WHY_WOODS.md): the problems runtime introspection solves.
+- [Internals](INTERNALS.md): extraction, publication, graph, storage, retrieval, and MCP components.
+- [Extractor reference](EXTRACTOR_REFERENCE.md): what each extractor produces and the edge cases it handles.
+- [Reading a published index from Ruby](PUBLISHED_INDEX.md): the `Woods::PublishedIndex` Ruby API for cops, gate scripts, and `woods:check:*` tasks (including the moved-message check).
+- [Evaluation](EVALUATION.md): retrieval scoring, baselines, and the agent-level index on/off ablation.
+- [Backend matrix](BACKEND_MATRIX.md): implemented provider/store combinations and their operational requirements.
+- [Token benchmark](TOKEN_BENCHMARK.md): evidence behind Woods token-estimation defaults.
+- [FAQ](FAQ.md): short answers and links to the canonical guides.
+
+## Exports and integrations
+
+- [Notion integration](NOTION_INTEGRATION.md)
+- [Obsidian integration](OBSIDIAN_INTEGRATION.md)
+- [Unblocked integration](UNBLOCKED_INTEGRATION.md)
+
+## Maintainer material
+
+Historical build-phase documents are not user guides. Source checkouts also contain the [MCP protocol decision record](https://github.com/lost-in-the/woods/blob/main/docs/design/MCP_2026_STRATEGY.md), [generated self-analysis diagrams](https://github.com/lost-in-the/woods/tree/main/docs/self-analysis), and the maintainer work ledger in `backlog.json`; these maintainer-only paths are not packaged with the gem.
+
+Contract records that read as maintainer reference rather than guides:
+
+- [Incremental extraction](INCREMENTAL_EXTRACTION.md): the incremental/full equivalence contract and dispatch inventory.
+- [Watch daemon](WATCH_DAEMON.md): the resident daemon's design contract (the user-facing entry point is the task table above).
+
+The release flow, the version states `main` moves through, and the tag and dispatch runbook live in [CONTRIBUTING.md](../CONTRIBUTING.md#release-flow).
+
+## Canonical owners
+
+Use this map when changing behavior or documentation. Update the owner first; other pages should summarize and link instead of copying full instructions.
+
+| Fact or workflow | Canonical owner |
+|---|---|
+| Install and first successful run | [GETTING_STARTED.md](GETTING_STARTED.md) |
+| Agent-operated install/configuration | [AGENT_SETUP.md](AGENT_SETUP.md) |
+| Configuration keys and defaults | [CONFIGURATION_REFERENCE.md](CONFIGURATION_REFERENCE.md) |
+| MCP server setup and callable tool surface | [MCP_SERVERS.md](MCP_SERVERS.md) |
+| Agent tool-selection workflow | [AGENT_GUIDE.md](AGENT_GUIDE.md) |
+| Console security and transports | [CONSOLE_MCP_SETUP.md](CONSOLE_MCP_SETUP.md) |
+| v1-to-v2 migration | [UPGRADING_TO_2.md](UPGRADING_TO_2.md) |
+| Failure diagnosis | [TROUBLESHOOTING.md](TROUBLESHOOTING.md) |
+| Contributor policy | [CONTRIBUTING.md](../CONTRIBUTING.md) |
+| Coding-agent repository instructions | [AGENTS.md](https://github.com/lost-in-the/woods/blob/main/AGENTS.md) |
+| Non-MCP Ruby access to a published index | [PUBLISHED_INDEX.md](PUBLISHED_INDEX.md) |
+| Evaluation harnesses | [EVALUATION.md](EVALUATION.md) |
+
+The current public surface is generated from 35 extractors. Counts and capability claims must match `.Codex/release-v2/surface-inventory.json`, which is generated from the code and verified in CI.

data/docs/RETRIEVAL_GUIDE.md

@@ -0,0 +1,267 @@
+# Retrieval Guide
+
+Woods retrieval combines semantic search (vector similarity), keyword search (identifier/text matching), and graph traversal (dependency edges), fusing results with Reciprocal Rank Fusion (RRF) before assembling them into a token-budgeted context string. This is distinct from `search` (exact name/pattern lookup) or `lookup` (direct identifier fetch): retrieval is designed for natural-language questions about behavior, relationships, or concepts that span multiple code units.
+
+---
+
+## The Pipeline at a Glance
+
+```
+query
+  └─▶ QueryClassifier        classify intent, scope, target type
+        └─▶ SearchExecutor   select strategy, run parallel search
+              ├── vector search     (semantic similarity)
+              ├── keyword search    (identifier/text matching)
+              └── graph traversal  (dependency edges)
+                └─▶ Ranker         RRF fusion + weighted signal scoring
+                      └─▶ ContextAssembler  token-budgeted context string
+                            └─▶ RetrievalResult
+```
+
+| Stage | Class | Responsibility |
+|-------|-------|----------------|
+| Classification | `Woods::Retrieval::QueryClassifier` | Detects intent, scope, target type, and framework context from the query text |
+| Search | `Woods::Retrieval::SearchExecutor` | Maps classification to a strategy (`:vector`, `:keyword`, `:graph`, `:hybrid`, `:direct`) and executes it |
+| Ranking | `Woods::Retrieval::Ranker` | Applies RRF across sources, then weighted signal scoring (semantic, keyword, recency, importance, type match, diversity) |
+| Assembly | `Woods::Retrieval::ContextAssembler` | Fills a token budget with ranked units, sectioned into structural / primary / supporting / framework blocks |
+| Orchestration | `Woods::Retriever` | Coordinates all four stages; returns a `RetrievalResult` with `context`, `sources`, `strategy`, `tokens_used`, and `trace` |
+
+### Search strategies
+
+`SearchExecutor` selects one of five strategies based on query classification:
+
+| Strategy | When selected | What it does |
+|----------|--------------|-------------|
+| `:vector` | `understand`, `debug`, `implement` intents | Embeds query, searches vector store by cosine similarity |
+| `:keyword` | `locate`, `reference` intents; `framework` queries | Searches metadata store by extracted keywords |
+| `:graph` | `trace` intent | Finds seed identifiers, then walks forward and reverse dependency edges |
+| `:hybrid` | `comprehensive` or `exploratory` scope | Runs vector + keyword + graph expansion, deduplicates |
+| `:direct` | `locate`/`reference` + `pinpoint` scope | Looks up identifiers directly in metadata store; falls back to keyword |
+
+Keyword results are scored by how many distinct fields matched (identifier, source, metadata), not by the store's result order. Each matched field adds 0.25, capped at 1.0, so a result matching on identifier and source scores higher than one matching source alone.
+
+---
+
+## Configuring Retrieval
+
+Retrieval requires an embedding provider and a vector store. Set these in `config/initializers/woods.rb`.
+
+### Presets (recommended)
+
+Four named presets cover the supported deployment scenarios:
+
+```ruby
+# Local development: in-memory vectors + SQLite metadata + Ollama.
+# Requires sqlite3, a running Ollama service, and a pulled embedding model.
+Woods.configure_with_preset(:local)
+
+# Separate embed/query processes sharing output_dir. No sqlite3 gem.
+# Requires a running Ollama service and a filesystem visible to both processes.
+Woods.configure_with_preset(:shared_filesystem)
+
+# PostgreSQL: pgvector + SQLite metadata + OpenAI.
+# Requires pgvector, sqlite3, and an OpenAI API key.
+Woods.configure_with_preset(:postgresql) do |config|
+  config.embedding_options = { api_key: ENV.fetch('OPENAI_API_KEY') }
+  config.vector_store_options = { connection: ActiveRecord::Base.connection }
+end
+
+# Production: Qdrant + SQLite metadata + OpenAI.
+# Requires Qdrant, sqlite3, and an OpenAI API key.
+Woods.configure_with_preset(:production) do |config|
+  config.embedding_options = { api_key: ENV.fetch('OPENAI_API_KEY') }
+  config.vector_store_options = {
+    url: ENV.fetch('QDRANT_URL'),
+    collection: ENV.fetch('WOODS_QDRANT_COLLECTION', 'woods'),
+    allow_private_hosts: true # only when QDRANT_URL is deliberately private
+  }
+end
+```
+
+Presets accept a block for overrides:
+
+```ruby
+Woods.configure_with_preset(:local) { |config| config.max_context_tokens = 12_000 }
+```
+
+### Manual configuration
+
+**MySQL host app (Qdrant required. MySQL has no native vector extension):**
+
+```ruby
+Woods.configure do |config|
+  config.vector_store         = :qdrant
+  config.vector_store_options = {
+    url: ENV.fetch('QDRANT_URL'),
+    collection: 'myapp',
+    allow_private_hosts: true # required for trusted localhost/RFC1918 URLs
+  }
+  config.metadata_store       = :sqlite
+  config.embedding_provider   = :openai
+  config.embedding_options    = { api_key: ENV['OPENAI_API_KEY'] }
+  config.embedding_model      = 'text-embedding-3-small'
+end
+```
+
+**PostgreSQL host app (pgvector, all-in-one):**
+
+```ruby
+Woods.configure do |config|
+  config.vector_store         = :pgvector
+  # pgvector takes a live PostgreSQL connection object, not a URL.
+  config.vector_store_options = { connection: ActiveRecord::Base.connection }
+  config.metadata_store       = :sqlite
+  config.embedding_provider   = :openai
+  config.embedding_options    = { api_key: ENV['OPENAI_API_KEY'] }
+  config.embedding_model      = 'text-embedding-3-small'
+end
+```
+
+After configuring, generate embeddings before running retrieval:
+
+```bash
+bundle exec rake woods:extract
+bundle exec rake woods:embed
+```
+
+---
+
+## Running Retrieval
+
+### MCP tool: `codebase_retrieve`
+
+The primary interface for agents. Available in the Index Server when an embedding provider is configured and `rake woods:embed` has been run.
+
+```
+codebase_retrieve(query: "how does billing work?")
+codebase_retrieve(query: "what callbacks run when an order is placed?", budget: 12000)
+```
+
+Parameters:
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `query` | string | required | Natural-language question |
+| `budget` | integer | 8000 | Token budget for context assembly |
+
+The tool returns a formatted context string ready for use in a prompt, along with source attributions. Use `search` for exact name/pattern lookups; use `codebase_retrieve` for conceptual or behavioral questions.
+
+### Ruby API
+
+```ruby
+retriever = Woods::Retriever.new(
+  vector_store:       vector_store,
+  metadata_store:     metadata_store,
+  graph_store:        graph_store,
+  embedding_provider: embedding_provider
+)
+
+result = retriever.retrieve("How does the User model work?")
+
+result.context      # => "Codebase: 42 units...\n\n---\n\n## User (model)\n..."
+result.strategy     # => :hybrid
+result.tokens_used  # => 4200
+result.sources      # => [{ identifier: "User", type: "model", score: 0.91, ... }]
+result.trace        # => RetrievalTrace with elapsed_ms, candidate_count, etc.
+```
+
+Override the token budget per call:
+
+```ruby
+result = retriever.retrieve("explain the checkout flow", budget: 16_000)
+```
+
+`Woods.build_retriever` instantiates a retriever from the current configuration:
+
+```ruby
+Woods.configure_with_preset(:postgresql) do |config|
+  config.embedding_options = { api_key: ENV.fetch('OPENAI_API_KEY') }
+  config.vector_store_options = { connection: ActiveRecord::Base.connection }
+end
+retriever = Woods.build_retriever
+result    = retriever.retrieve("what validations does Order have?")
+```
+
+---
+
+## Degradation Tiers
+
+Retrieval degrades gracefully when components are unavailable. The Retriever itself does not implement explicit fallback tiers, degradation happens naturally through how each component handles errors:
+
+- **Embedding provider unavailable**: `codebase_retrieve` returns a structured configuration error. Check `woods_status` for retrieval readiness.
+- **Vector store unavailable**: vector and hybrid strategies fail at query time. Keyword and graph strategies remain available for direct calls to `SearchExecutor`.
+- **Metadata store error**: the structural context overview (unit counts by type) is silently omitted; `Retriever#build_structural_context` rescues `StandardError` and returns `nil`. The retrieval result is still returned without the overview.
+- **Graph store unavailable**: graph expansion in hybrid strategy produces no graph candidates; vector and keyword candidates are still ranked and returned.
+
+In all cases, errors in individual components produce empty candidate sets for that source rather than raising through the `Retriever`. Configure circuit breakers via `Woods::Resilience::CircuitBreaker` on external providers (Qdrant, OpenAI) for production deployments.
+
+---
+
+## Tuning
+
+### `similarity_threshold`
+
+Controls which vector search results are considered. Range: `0.0`–`1.0`. Default: `0.7`.
+
+```ruby
+config.similarity_threshold = 0.6  # Include less similar results (broader)
+config.similarity_threshold = 0.8  # Require higher similarity (narrower)
+```
+
+Lower values return more candidates, which can improve recall for broad queries at the cost of precision. Raise it if results seem loosely related.
+
+### `max_context_tokens`
+
+Sets the default token budget for context assembly. Default: `8000`. The `budget` parameter on `codebase_retrieve` and `Retriever#retrieve` overrides this per call.
+
+```ruby
+config.max_context_tokens = 12_000  # More context per retrieval
+```
+
+The `ContextAssembler` carves off 10% for the structural overview first, then splits what's left:
+
+- **Framework context active** (the query mentions Rails/framework keywords, `rails`, `activerecord`, `middleware`, etc.): primary 55%, supporting 25%, framework 20%.
+- **No framework context**: primary 65%, supporting 35%, the framework section gets nothing, and its share is not proportionally folded into the other two; the fractions are just different, not rescaled.
+
+Separately, if the supporting section ends up with no candidates (it only ever holds `:graph_expansion` results), its reserved budget is reclaimed into primary rather than wasted.
+
+### `context_format`
+
+Controls how assembled units are formatted. Default: `:markdown`. Valid values: `:claude`, `:markdown`, `:plain`, `:json`.
+
+```ruby
+config.context_format = :claude # XML-wrapped output for Claude-style context
+config.context_format = :json   # Machine-readable output
+```
+
+### Switching embedding models
+
+The embedding model must match between `rake woods:embed` and retrieval. Different models produce vectors with different dimensionalities. Woods raises `Woods::MCP::DimensionMismatch` when they disagree, at embed time for durable stores and at MCP boot for dumps. After changing `embedding_model`, drop the vector store and re-run full extraction and embedding:
+
+```bash
+bundle exec rake woods:extract
+bundle exec rake woods:embed
+```
+
+**OpenAI model dimensions:**
+
+| Model | Dimensions |
+|-------|-----------|
+| `text-embedding-3-small` (default) | 1536 |
+| `text-embedding-3-large` | 3072 |
+
+**Ollama default model:** `nomic-embed-text`. Dimensions are detected dynamically on first embed.
+
+---
+
+## Troubleshooting
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| `codebase_retrieve` returns no results | Embeddings not generated, or embedding provider not configured | Run `rake woods:embed`; verify `embedding_provider` is set and API key is valid |
+| Results are stale or missing recent changes | Index not updated after code changes | Run `rake woods:incremental` (or `rake woods:extract` for route/event changes) |
+| Dimension mismatch warning in logs | `embedding_model` changed after embedding was generated | Re-run `rake woods:extract && rake woods:embed` with the new model |
+| Empty results for a known class name | Keyword strategy not finding the identifier | Try a conceptual query with `codebase_retrieve`; or use `search` for exact name lookup |
+| Very slow retrieval | Large vector index without HNSW index, or Qdrant cold start | For pgvector: create an HNSW index (see `BACKEND_MATRIX.md`). For Qdrant: check collection status |
+| `codebase_retrieve` tool listed but disabled | Embedding provider not configured or API key missing | Set `embedding_provider`, run `woods:embed`, and check `woods_status` |
+| Results clustered around one type | Diversity penalty insufficient for codebase shape | Lower `similarity_threshold` slightly and widen the query scope |

data/docs/TOKEN_BENCHMARK.md

@@ -0,0 +1,68 @@
+# Token Estimation Benchmark
+
+> **Single source of truth:** `Woods::TokenUtils.chars_per_token_for(provider)`
+> in `lib/woods/token_utils.rb`. Production code uses **4.0 chars/token** for
+> the OpenAI path and **1.5 chars/token** for the Ollama / WordPiece path.
+> Both are applied consistently by `Woods::Builder#chars_per_token_for`,
+> `ContextAssembler`, `TextPreparer`, and `ExtractedUnit#estimated_tokens`.
+> The cost-model layer (`lib/woods/cost_model/`) is the one exception, it
+> uses its own pre-aggregated `TOKENS_PER_CHUNK` constant (450) rather than
+> a per-string ratio, since it measures a different thing (per-chunk average
+> vs. per-string chars/token).
+>
+> When the optional [`tokenizers`](https://github.com/ankane/tokenizers-ruby)
+> gem is installed, the Ollama path uses the real BERT WordPiece tokenizer
+> (`Woods::Embedding::TokenCounter`) instead of this heuristic. The 4.0
+> divisor below is what the gem falls back to everywhere else.
+
+This is a historical record of the benchmark that picked 4.0 over the
+original 3.5 divisor. It is cited from five places in `lib/` as the evidence
+for that choice, keep the numbers below intact if you edit this doc.
+
+## What was measured
+
+- **Corpus**: 19 Ruby source files from `lib/woods/` (1.4 KB–33.4 KB): extractors, retrieval pipeline, MCP servers, and utility modules.
+- **Reference tokenizer**: tiktoken_ruby with cl100k_base (GPT-4) and o200k_base (GPT-4o).
+- **Heuristic under test**: `(content.length / N).ceil`, comparing divisors 3.0–4.0.
+
+## Results
+
+| Divisor | Mean Abs Error (cl100k) | Max Abs Error |
+|--------:|------------------------:|--------------:|
+| 3.0 | 47.2% | 80.5% |
+| 3.2 | 38.0% | 69.2% |
+| 3.5 (previous default) | 26.2% | 54.7% |
+| 3.8 | 16.2% | 42.5% |
+| **4.0 (shipped)** | **10.6%** | **35.4%** |
+
+Mean chars/token across the corpus was **4.41** (range 3.94–5.42). The
+heuristic always overestimated, never underestimated, across all 19 files,
+which is what makes it safe for token-limit enforcement even at its worst
+case. Code lines and comment/YARD lines had similar ratios (4.38 vs. 4.27
+chars/token), no separate handling needed for either.
+
+## What shipped
+
+**The divisor changed from 3.5 to 4.0.** It roughly halves the mean
+overestimate (26.2% → 10.6%) while keeping the conservative
+always-overestimates property, at zero new runtime dependencies. The
+constant lives in one place now (`Woods::TokenUtils::CHARS_PER_TOKEN_BY_PROVIDER`),
+not scattered across call sites, see `lib/woods/token_utils.rb` for the
+current definition and `docs/EMBEDDING_MODELS.md` for the Ollama-side ratio.
+
+**tiktoken_ruby was deliberately not added as a runtime dependency.** A 10.6%
+mean error is acceptable for chunking decisions, budget estimates, and
+truncation; a native-extension dependency for marginal accuracy gains wasn't
+worth it. The optional `tokenizers` gem covers the case where exact counts
+matter more (see above).
+
+## Reproducing this benchmark
+
+```bash
+gem install tiktoken_ruby  # not a Gemfile dependency, benchmark only
+```
+
+`spec/token_estimation_benchmark_spec.rb` keeps this honest going forward: it
+checks the heuristic is self-consistent (monotonically increasing with
+content length) and, when tiktoken_ruby is installed, that overestimation
+stays bounded with no underestimation greater than 5%.