woods 2.0.0.beta2 → 2.0.0.beta3

data/docs/CONFIGURATION_REFERENCE.md

@@ -108,8 +108,9 @@ Columns:
 | `output_dir` | Pathname/String | `Rails.root.join('tmp/woods')` | user-settable | Directory where extracted data is written |
 | `extractors` | Array<Symbol> | `[:models, :controllers, :services, ...]` | accepted, not implemented | Does not select which extractors run. See [Extractors](#extractors) below. |
 | `pretty_json` | Boolean | `true` | user-settable | Format extracted JSON with indentation |
-| `max_context_tokens` | Integer | `8000` | user-settable | Maximum tokens for retrieval context windows |
-| `similarity_threshold` | Float | `0.7` | user-settable | Minimum similarity score (0.0-1.0) for retrieval results |
+| `retrieval_mode` | Symbol | `:semantic` | user-settable | `:semantic` uses configured embeddings; explicit `:lexical` ranks published text without a provider/vector store. See [retrieval modes](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval). |
+| `max_context_tokens` | Integer | `8000` | user-settable | Default context-assembly token budget captured when a retriever is built; per-call budget overrides it |
+| `similarity_threshold` | Float | `0.7` | deprecated, inert | Accepted for compatibility; setting it warns. It does not filter or change retrieval ranking |
 | `context_format` | Symbol | `:markdown` | user-settable | Output format for retrieval: `:claude`, `:markdown`, `:plain`, `:json` |
 | `include_framework_sources` | Boolean | `true` | user-settable | Extract Rails and gem source code |
 | `concurrent_extraction` | Boolean | `false` | user-settable | Enable parallel extraction (experimental) |
@@ -135,6 +136,17 @@ config.embedding_options = {
 }
 ```
 
+OpenAI embedding batches are sent in slices of at most 36 texts, preserving
+input order. For inputs within the API's 8,192-token per-text limit, this stays
+below both the 2,048-input limit and the 300,000-token total request limit.
+See the [OpenAI embedding request contract](https://developers.openai.com/api/reference/resources/embeddings/methods/create).
+Woods retains its conservative 8,191-token chunking ceiling; slicing does not
+make an individually oversized text valid. This bound avoids relying on token
+estimates and adds HTTP requests for batches containing many short chunks.
+All slices must validate, including consistent vector dimensions, before the
+provider returns any vectors for the batch. Ollama and custom providers keep
+their existing batching behavior.
+
 ### Ollama embeddings
 
 ```ruby
@@ -224,6 +236,15 @@ config.metadata_store_options = {
 }
 ```
 
+Without an explicit `database` option, SQLite uses
+`<output_dir>/metadata.sqlite3`. For `woods:embed` and
+`woods:embed_incremental`, `WOODS_OUTPUT` overrides that directory together
+with the index and embedding dumps. An explicit `database` path (including
+`:memory:`) still takes precedence, so configure a separate path for each
+worktree when overriding it. Existing databases at the old configured output
+path are not moved or deleted; run `woods:embed` for the selected index after
+upgrading to populate its default metadata database.
+
 Requires the `sqlite3` gem in your host bundle. Rails apps backed by
 MySQL or PostgreSQL won't have it by default, selecting `:sqlite`
 without it raises `Woods::ConfigurationError` with install
@@ -236,10 +257,13 @@ unless cross-process metadata persistence matters.
 config.metadata_store = :in_memory
 ```
 
-Pure-Ruby hash-backed store. No external dependencies, no persistence, vectors and metadata both live in the building process and die with
-it. The `_index.json` manifest under `output_dir` is the durable
-metadata for the index MCP server, so this is a reasonable default
-for hosts that don't bundle `sqlite3`.
+Pure-Ruby hash-backed store with no external dependencies. For local vector
+presets, embedding runs persist it as `metadata.msgpack` alongside `vectors.bin`
+in the promoted dump; the index MCP server loads that snapshot at startup or
+reload. Incremental embedding publishes changes to paths, dependencies, and
+other unit metadata even when unchanged source needs no new embedding. A run
+with no content or metadata changes keeps the existing dump and retention
+window. This is a reasonable default for hosts that don't bundle `sqlite3`.
 
 ## Retrieval cache options
 
@@ -278,6 +302,13 @@ overrides the wrapper defaults for `:embeddings` (24 hours) and `:context`
 (15 minutes). `:memory` accepts `max_entries` (default 500); it ignores
 `default_ttl` because each wrapper write supplies its domain TTL.
 
+`Woods::Cache.cache_key` length-prefixes every component, including a single
+component, so different argument counts cannot share a response. Existing
+multi-component keys used by Woods' wrappers remain unchanged. Custom callers
+using single-component keys must clear their affected persistent cache domain
+when upgrading, since older unprefixed entries can alias the new encoding;
+subsequent calls refill it normally. Namespace clearing still covers both formats.
+
 ## Deployment shapes
 
 Woods supports three deployment shapes, pick the preset that matches yours.
@@ -358,12 +389,41 @@ end
 | `extract_navigation_edges` | Boolean | `true` | Extract `link_to`, `redirect_to`, and `form_action` navigation edges from views and controllers |
 | `enable_snapshots` | Boolean | `false` | Enable temporal snapshots. Woods automatically migrates its internal output-directory SQLite store; if SQLite is unavailable, it uses the JSON snapshot store. No Rails migration is required. |
 | `volatile_dependency_ratio` | Float | `3.0` | A dependency whose commit count (last 365 days) exceeds the dependent's by this ratio appears in the `volatile_dependencies` report (top 20, ranked by PageRank). Must be greater than 1. Report only, never a gate. |
+| `volatile_dependency_limit_per_target` | Integer or `nil` | `nil` | Optional maximum report edges per dependency (type and identifier), applied after ranking and before the global top 20. Positive integers only; `nil` leaves the default report unchanged. Distinct relationships consume separate slots. |
 | `graph_cycle_limit` | Integer or `nil` | `500` | How many distinct cycles `GraphAnalyzer` enumerates before it stops. Cycle detection finds one cycle per DFS back-edge, so a dense graph has tens of thousands of them and enumerating every one is the largest single cost of the analysis that runs on every extraction. Set to `nil` for exhaustive enumeration. |
 | `graph_cycle_max_length` | Integer or `nil` | `50` | The longest cycle recorded, in distinct nodes. A back-edge deep in the DFS closes a cycle as long as the path, which on a large graph is thousands of nodes: unreadable as a report and expensive to canonicalize. Set to `nil` to record a cycle of any length. |
 
 | `incremental_blast_radius_depth` | Integer or `nil` | `nil` | How many reverse hops an incremental run walks from a changed file before it stops re-extracting dependents. `nil` keeps the unbounded transitive closure. See the note below before setting it. |
 | `durable_payload_writes` | Boolean | `false` | Force an `fsync` on every payload file as it is written, on top of the single flush every publish already performs. See the note below before setting it. |
 
+**Tuning volatile dependency reports.** Start with
+`graph_analysis.json`'s `stats.volatile_dependency_count`: it counts every
+qualifying edge before either cap, while the array normally keeps only 20.
+Raise `volatile_dependency_ratio` until the remaining candidates are useful
+for your application. A measured 7k-unit app had 544 qualifying edges at 3.0;
+8–10 was a useful ratio there, not a universal recommendation. Commit counts
+cover the last 365 days and need complete git history; missing git data is not
+proof of stability.
+
+If one hot dependency still fills the list, set
+`volatile_dependency_limit_per_target` to a small positive integer such as 3.
+Each dependency keeps its highest-ranked edges before the global limit applies,
+allowing other dependencies into the report. Different relationship labels
+remain separate edges and consume separate slots. With this option enabled,
+`stats.volatile_dependencies_limit_per_target` records the setting and
+`stats.volatile_dependency_reported_count` counts the final persisted array;
+`stats.volatile_dependency_count` still counts all qualifying edges. The extra
+stats are absent at the default `nil`. This limits the report, not the work of
+finding qualifying edges. Run extraction again after changing either setting;
+MCP reads the published report. These findings remain informational, never a
+release or architecture gate.
+
+When the JSON snapshot fallback is in use, malformed JSON, top-level values
+other than objects, and files that cannot be read (including concurrent retention
+removals) are warned about and treated as absent. Snapshot lists and unit history
+omit them; direct lookup returns no snapshot, and a diff with an unavailable
+snapshot returns empty added, modified, and deleted lists.
+
 `incremental_blast_radius_depth` is unbounded by default because a unit two hops
 out really can have content that depends on the changed file. An STI grandchild
 (`SportsCar < Car < Vehicle`) inherits its grandparent's associations,
@@ -408,11 +468,54 @@ require 'woods/session_tracer/file_store' # the stores are not autoloaded
 
 config.session_tracer_enabled = true
 config.session_store = Woods::SessionTracer::FileStore.new(
-  Rails.root.join('tmp/session_traces')
+  base_dir: Rails.root.join('tmp/session_traces')
 )
 config.session_exclude_paths = ['/health', '/metrics', '/assets']
 ```
 
+### Redis session index compatibility
+
+`RedisStore` lists and clears both legacy SET indexes and recency ZSET indexes.
+Listing removes expired members and orders summaries by their last request.
+Reads and clears preserve a legacy SET, so upgrading only readers does not
+break older writers. Type checks and index operations run atomically in Redis
+and tolerate a concurrent writer converting the index.
+
+The first `record` from a newer writer converts the SET to a ZSET atomically.
+Upgrade writers together: older SET writers cannot write after that conversion.
+Migrated members receive score zero, so they evict in lexical order at the
+retention limit until recorded again; new records use their request timestamp.
+Session list contents and TTLs are preserved by the index conversion.
+
+### Solid Cache session retention and compatibility
+
+`Woods::SessionTracer::SolidCacheStore` accepts a direct `SolidCache::Store`.
+Its coordination layer uses private Solid Cache APIs for uncached, per-key
+routing and atomic ownership. Missing APIs raise
+`Woods::SessionTracer::SolidCacheCoordination::BackendError`; a gem's version
+constraint alone does not establish compatibility. The live contract suite
+validates SQLite, PostgreSQL, and MySQL (including MySQL's insert path without
+`RETURNING`). See [the version-validation procedure](../CONTRIBUTING.md#solid-cache-session-compatibility)
+before upgrading Solid Cache.
+
+Session traces are best-effort diagnostic data. The slot directory and record
+rings are bounded by `max_sessions` and `max_requests_per_session`, but the
+following crash/eviction limits remain:
+
+- A crashed admission can strand an active-session mapping outside the directory.
+  These mappings can accumulate across distinct session IDs, with a bounded
+  amount per ID. Recording that session again reclaims its mapping; the normal
+  directory bound is not a hard bound on these stranded keys.
+- If the backend evicts a slot counter while deep crash-orphan records remain
+  in an unoccupied slot, new records can be silently dropped until the counter
+  advances past them. This loss is bounded and self-healing, but the affected
+  requests are not recovered.
+- Losing the epoch key fences every live session. Old traces become unreadable,
+  and each session is admitted again on its next record. This deliberately
+  favors privacy over retention so previously cleared data stays cleared.
+
+Do not use this store as an audit log or the only record of a request.
+
 ## Gem indexing
 
 `config.add_gem` is accepted for forward compatibility but **not implemented**: nothing in the
@@ -525,8 +628,9 @@ deployment guide including defense layers.
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `console_mcp_enabled` | Boolean | `false` | Master switch. When `false`, the Railtie does not mount the Console MCP middleware. |
-| `console_mcp_token` | String | `ENV['WOODS_CONSOLE_MCP_TOKEN']` or `nil` | Bearer token required on every Console HTTP request. **Required in production**: the Railtie raises `Woods::ConfigurationError` when `console_mcp_enabled` is true but no token is set. In non-production a missing token warns at boot and every Console request fails closed with `401 Unauthorized`. A configured token shorter than 32 characters raises `Woods::ConfigurationError` at boot in every environment. Generate with `SecureRandom.hex(32)`. |
+| `console_mcp_enabled` | Boolean | `false` | Master switch. When `false`, stdio exits and the mounted Console middleware passes requests through to Rails. |
+| `console_mcp_http_enabled` | Boolean | `true` | HTTP transport switch; effective only while the master switch is on. Set `false` for stdio-only use without HTTP token validation or an active HTTP endpoint. Read at request time. |
+| `console_mcp_token` | String | `ENV['WOODS_CONSOLE_MCP_TOKEN']` or `nil` | Bearer token required on every enabled Console HTTP request. With both Console flags enabled, production boot raises on a missing token; other environments warn and requests fail closed with 401. A configured token shorter than 32 characters raises at boot while HTTP is enabled. Explicit stdio-only configurations skip HTTP token validation. Generate with `SecureRandom.hex(32)`. |
 | `console_mcp_allowed_origins` | Array\<String\> | `%w[http://localhost http://127.0.0.1 http://[::1]]` | `OriginGuard` allowlist. Port is stripped before comparison, so `http://localhost` matches any localhost port. Override for tunneled / internal-dashboard access. |
 | `console_mcp_path` | String | `/mcp/console` | URL path the Rack middleware responds on. |
 | `console_embedded_read_tools` | Boolean | `false` | Register `console_sql` and `console_query` in supported stdio and Rack modes. |
@@ -547,13 +651,14 @@ These variables are read by the gem and its MCP servers at runtime. They complem
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
+| `WOODS_RETRIEVAL_MODE` | `semantic` | Explicit packaged MCP retrieval mode: `semantic` or `lexical`. Lexical reads extraction unit JSON without provider autodetection, credentials or vector artifacts. |
 | `WOODS_DIR` | `Dir.pwd` | Path to the extraction output directory. |
-| `WOODS_REQUIRE_INDEX` | unset | Set to `"1"` to fail closed: the server refuses to boot (raises `MissingArtifact`) unless a real index (`woods.json`) is present. By default an extract-only host boots in pattern/structural mode without it. |
+| `WOODS_REQUIRE_INDEX` | unset | Set to `"1"` to fail closed: the server refuses to boot (raises `MissingArtifact`) unless a real index (`woods.json`) is present. By default an extract-only host boots in pattern/structural mode without it. Explicit lexical mode requires a valid published extraction index, not `woods.json`. |
 | `WOODS_ALLOW_AUTODETECT` | unset | **Deprecated no-op.** Auto-detect is now the default; accepted for backward compatibility only. |
 | `WOODS_SEARCH_MAX_SCAN` | `500` | Cap on unit files loaded during a phase-2 (metadata/source_code) `search`. Hitting the cap sets `partial: true` in the response. |
 | `WOODS_SNAPSHOTS` | unset | Set to `"true"` to force-enable temporal snapshot storage, even without a pre-existing SQLite database. |
 | `WOODS_ALLOW_PURGE` | unset | Set to `"1"` to override the 30%-deletion purge guard in `woods:embed`/`woods:embed_incremental`. |
-| `WOODS_PAYLOAD_RETENTION` | `3` | How many past generations' payload directories (`payloads/gen-N/`) to retain, and — when the JSON snapshot store is in use — how many temporal snapshots (`snapshots/`) to keep. A payload pinned by an active reader process is kept temporarily beyond this bound and reconsidered after the pin is released. |
+| `WOODS_PAYLOAD_RETENTION` | `3` | How many past generations' payload directories (`payloads/gen-N/`) to retain, and — when the JSON snapshot store is in use — how many temporal snapshots (`snapshots/`) to keep. JSON snapshot retention counts corrupt or unreadable SHA-named files and prunes them first; the just-captured snapshot is protected. Cleanup failures are non-fatal. A payload pinned by an active reader process is kept temporarily beyond this bound and reconsidered after the pin is released. |
 | `WOODS_MCP_CACHE_TTL_MS` | `10000` | Cache TTL advertised in tool result `_meta`. `0` disables caching. |
 | `WOODS_NO_UPDATE_CHECK` | unset | Set to `"1"` to skip the `woods_status` RubyGems version check. |
 | `XDG_CACHE_HOME` | `~/.cache` | Base directory for the best-effort update-check cache (`$XDG_CACHE_HOME/woods/update_check.json`). An unset or empty value uses `~/.cache`; if the home directory cannot be resolved, Woods falls back to the system temporary directory. |
@@ -590,24 +695,52 @@ These variables are read by the gem and its MCP servers at runtime. They complem
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
-| `WOODS_IGNORE_WATCH` | unset | Set to `"1"` to make `woods:incremental`/`woods:clean` proceed even when a daemon is (or claims to be) running. For `woods:incremental` this removes daemon coverage: a git range that fails to resolve then exits 1 instead of standing down (see [Incremental Extraction](./INCREMENTAL_EXTRACTION.md#exit-behavior-in-ci-chains)). |
+| `WOODS_IGNORE_WATCH` | unset | Set to `"1"` to make `woods:incremental`/`woods:clean`/`woods:hook_refresh` proceed even when a daemon is (or claims to be) running. For `woods:incremental` this removes daemon coverage: a git range that fails to resolve then exits 1 instead of standing down (see [Incremental Extraction](./INCREMENTAL_EXTRACTION.md#exit-behavior-in-ci-chains)). |
 | `WOODS_LOCK_WAIT` | `Watch::Daemon::LOCK_STALE_TIMEOUT` (600s) | How long a rake writer waits for `PipelineLock` before exiting non-zero. |
 | `WOODS_WATCH_POLL` | auto-detected | Set to `"1"`/`"0"` to force/disable polling mode (vs. `listen` gem, e.g. in a container without inotify). |
+| `WOODS_WATCH_POLL_INTERVAL` | `1.0` (seconds) | Positive, finite delay between polling scans; also used on native-watcher fallback. Does not force polling. Larger values reduce scan frequency and can delay detection and shutdown. |
 | `WOODS_WATCH_DEBOUNCE` | `0.4` (seconds) | Delay before processing a batch of file-change events. |
 | `WOODS_WATCH_FULL_THRESHOLD` | `50` | Number of changed paths in one batch that triggers a full extraction instead of incremental. |
 | `WOODS_WATCH_IDLE_TIMEOUT` | unset (no timeout) | Seconds of inactivity before the daemon exits. |
 | `WOODS_WATCH_CATCH_UP` | `1` (enabled) | Set to `"0"` to skip generation-watermark catch-up on daemon start. |
+| `WOODS_WATCH_TRUST_FOREIGN_HOST` | unset (disabled) | Set to `"1"` in each task/MCP reader to trust a foreign daemon's heartbeat for up to 15 minutes, without a local pid check. See [cross-host liveness](WATCH_DAEMON.md#cross-host-liveness) for clock bounds, degraded coverage, and startup limitations. |
+
+### Opt-in plugin refresh hooks
+
+These settings control the plugin shell worker. Check installed
+`woods:hook_refresh` support first; the task is unreleased after 2.0.0.beta2.
+See [hook coverage and retry](WATCH_DAEMON.md#hooks-for-agent-sessions) and
+[optional context limits](WATCH_DAEMON.md#optional-bounded-context-hints).
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `WOODS_HOOKS_ENABLED` | unset (disabled) | Exact `1` enables the refresh/session hooks when an index exists. |
+| `WOODS_HOOKS_DISABLED` | unset | Exact `1` disables both refresh and optional context hooks. |
+| `WOODS_HOOK_CONTEXT_ENABLED` | unset (disabled) | Exact `1` enables separate bounded Claude orientation/impact hints; independent of refresh enablement. |
+| `WOODS_HOOK_CONTEXT_COMMAND` | `bundle exec woods-hook-context` | Installed helper argv prefix; startup counts toward the fixed context deadline. Use a wrapper for quoting/container environment. |
+| `WOODS_HOOK_CONTEXT_ROOT` | payload cwd | Explicit runtime-visible application root for context path mapping; set inside a container when host paths differ. |
+| `WOODS_HOOK_RAKE` | `bundle exec rake` | Application command prefix; supports `docker compose exec -T app bundle exec rake`. Use a wrapper for shell quoting or explicit container environment. |
+| `WOODS_SOURCE_CAPTURE` | internal | Private, one-use `woods-extract` child handoff. Do not set or persist this variable manually; see [source freshness](SOURCE_FRESHNESS.md). |
+| `WOODS_HOOK_TIMEOUT_SECONDS` | `600` | PostToolUse worker deadline (SessionStart uses a fixed ten seconds), integer 1–3600 seconds; failed/deferred batches remain queued. Docker-side cancellation requires separate verification. |
+| `WOODS_HOOK_LOCK_STALE_SECONDS` | `1800` | Age used only to reclaim legacy empty mkdir locks; live PID owners are never reclaimed merely by age. |
+
+`woods:hook_refresh[<base64 JSON>]` is the internal plugin transport. Version 1
+contains `output` and an `events` array of `{path, operation}` records; paths are
+application-relative and operations are `add`, `update`, `delete`, or `move`.
+The task validates inputs, defers active daemons with exit 75 before Rails boot,
+and checks publication failure before acknowledging work. Use ordinary extraction
+tasks for manual refreshes; hook transport is not a general shell execution API.
 
 ### Extraction rake tasks
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
-| `WOODS_OUTPUT` | `Woods.configuration.output_dir` | Overrides the output directory for `woods:extract`/`woods:incremental`/`woods:watch` without editing the initializer. |
+| `WOODS_OUTPUT` | `Woods.configuration.output_dir` | Overrides the output directory for extraction/watch and embedding tasks without editing the initializer; embedding also places its default SQLite metadata database there. Explicit database options take precedence. |
 | `CHANGED_FILES` | unset | Comma-separated explicit changed-path list for `woods:incremental`; when set, git range resolution is skipped entirely. |
 | `CI_COMMIT_BEFORE_SHA`, `CI_COMMIT_SHA` | unset (GitLab) | Build the diff range `<before>..<after>` for `woods:incremental`. A zero before-SHA (new branch) makes the range unresolvable, which exits 1 unless a running daemon covers the index. |
 | `GITHUB_BASE_REF` | unset (GitHub Actions) | Build the diff range `origin/<ref>...HEAD` for `woods:incremental`; an unfetched ref makes the range unresolvable, same exit behavior. |
 | `RAILS_ENV` | `development` | Rails environment the rake tasks boot in. |
-| `WOODS_PROFILE` | unset | Set to `"1"` to log one `[Woods] [profile] <phase> in N.NNs` line per run phase (payload seed, previous graph load, eager load, extraction or blast radius and re-extraction, type index, graph analysis, flows, manifest and summary, publish). Complements the per-extractor timing lines, which cover extraction only. Off by default and free when off. |
+| `WOODS_PROFILE` | unset | Set to `"1"` to log disjoint `[Woods] [profile] <phase> in N.NNs` durations, including git enrichment, reconciliation, payload sync, pointer publication (`publish`) and retention (`payload prune`). Separate `[profile total]` lines report whole extraction wall time, including unprofiled setup and failed runs; never add these totals to phase durations. Excludes process/Rails boot before extraction. Off by default. |
 | `WOODS_GIT_DIR` | unset | Absolute path to the canonical git directory. Wins over the repository Woods would otherwise find, at all three of its git call sites: per-unit `commit_count`/`change_frequency` (enrichment), `manifest.json`'s `git_branch`/`git_sha` (provenance), and the `woods:incremental` diff range. All three build their command line with `Woods::GitCommand.argv`. |
 | `GIT_BRANCH`, `GIT_SHA` | unset | Provenance for a checkout with no `.git` at all (a source tarball, a Docker `COPY` that excludes it). Ignored when a `.git` is present but unresolvable, so a stale build arg cannot mask a worktree. |
 
@@ -645,6 +778,49 @@ WOODS_GIT_DIR=/canonical-git bundle exec rake woods:extract
 
 The `woods-mcp` bootstrapper emits a one-line STDERR banner at startup indicating whether semantic search is enabled and which provider is active. If no key/instance is found, pattern search still works and `codebase_retrieve` surfaces an actionable fix message.
 
+## Git enrichment history
+
+Current source requires **Git 2.31 or newer** for optional per-unit git
+metadata. Extraction still succeeds when git is unavailable or history cannot
+be read completely. Git enrichment is omitted in either case; a failed or
+incomplete streamed history read logs a warning.
+This requirement and the history policy below are unreleased after 2.0.0.beta2.
+
+Per-unit enrichment also requires a non-shallow repository. A shallow checkout
+or a failed repository-depth probe omits enrichment with one warning per
+extractor instance. Fetch complete history (`git fetch --unshallow`, or
+`actions/checkout` with `fetch-depth: 0`) and run full extraction to refresh
+retained metadata. If depth cannot be verified, check git access and version.
+A source archive without a repository remains quiet.
+
+Full and incremental extraction use one streamed `HEAD` history walk, restricted
+to the last 365 days by git's `--since` traversal. Only requested app-owned paths
+are retained. Commit counts and contributors describe **HEAD-reachable touched-path
+events**, independent of which other paths are requested:
+
+- A root commit compares with an empty tree; a normal commit compares with its parent.
+- A merge compares with its first parent, while traversal still visits all parents.
+  A normal merge can therefore count both a side commit and the merge that introduces
+  its change. An `ours` merge touches no paths itself, but its side commits remain
+  reachable and count. A conflict-resolution merge counts when its result differs
+  from the first parent.
+- Renames are deletion/addition events at exact current names; Woods never follows
+  previous names. Unmerged branches, remote-only refs, and checkpoint refs are excluded.
+- `change_frequency` uses total events and events newer than 90 days. Contributors
+  and recent commits retain their existing top-five limit; recent commits and
+  `last_modified` follow git's traversal order, not a separate global timestamp sort.
+
+**Compatibility:** merge-related counts, authors, recent commits, and derived
+churn analysis can differ from the old 500-path batches, which used git's
+pathspec history simplification. This is an intentional semantics change.
+Run a full extraction after upgrading to replace retained incremental metadata
+consistently. No configuration key enables the new policy; rollback uses the
+previous gem plus a full extraction. Shallow checkouts still provide truncated
+history; fetch the complete history for complete 365-day evidence.
+
+The single walk removes repeated pathspec matching. Its end-to-end speedup on
+the original #305 host has not yet been measured.
+
 ## Database compatibility
 
 All storage options work with both MySQL and PostgreSQL, except:
@@ -653,3 +829,12 @@ All storage options work with both MySQL and PostgreSQL, except:
 - **SQLite metadata store**: uses a standalone SQLite database file, independent of your app's database
 
 See [BACKEND_MATRIX.md](BACKEND_MATRIX.md) for the full compatibility matrix.
+
+### Explicit retrieval and discovery scope
+
+On a server whose tool schema advertises them, `packages` and `source_paths` narrow
+`search` and `codebase_retrieve` before candidate limits. These are per-call
+arguments, not configuration settings. Inspect applied scope and completeness;
+a narrow graph query can omit relevant cross-boundary dependencies. See the
+[scope contract](RETRIEVAL_GUIDE.md#explicit-package-and-source-path-scopes) for
+root/nested ownership, path normalization, errors, storage support, and cost.

data/docs/CONSOLE_MCP_SETUP.md

@@ -26,13 +26,23 @@ The simplest setup. The `woods:console` rake task boots Rails, then starts the e
 ```ruby
 Woods.configure do |config|
   config.console_mcp_enabled = true
-  config.console_mcp_token = ENV["WOODS_CONSOLE_MCP_TOKEN"]
+  config.console_mcp_http_enabled = false # stdio-only; no HTTP endpoint
 end
 ```
 
-The stdio and Docker entry points exit with status 1 while this setting is false. Enabling it grants the MCP process live read access under the blocked-table, redaction, and credential-scanning controls described below.
+The stdio and Docker entry points exit with status 1 while `console_mcp_enabled` is false. Enabling it grants the MCP process live read access under the blocked-table, redaction, and credential-scanning controls described below.
 
-The token authenticates HTTP requests; a stdio client does not send it. Production Rails boot nevertheless requires a token of at least 32 characters whenever Console MCP is enabled, even for a stdio-only setup. Store `WOODS_CONSOLE_MCP_TOKEN` in the application's normal secret store. Outside production a missing token warns and leaves Console HTTP guarded with 401 — the boot warning names both transports, so a stdio-only setup can tell that the 401 is not its symptom and that its own session still works.
+`console_mcp_http_enabled` defaults to `true` to preserve existing HTTP
+setups. Set it to `false` for stdio-only use: HTTP guards and the Console
+middleware pass requests through to Rails, and boot skips HTTP token checks,
+including in production. Stdio does not send or consume a bearer token.
+
+If both Console flags are enabled, HTTP still requires a token of at least
+32 characters. Missing tokens warn outside production and every guarded
+request returns 401; production refuses to boot. A configured short token
+raises at boot in every environment while HTTP is enabled. Store the token
+in the application's normal secret store. Before enabling HTTP later, set
+its token, allowed origins and TLS as described in [Option C](#option-c-http-rack-middleware).
 
 ### How it works
 
@@ -172,6 +182,7 @@ In an initializer (`config/initializers/woods.rb`):
 ```ruby
 Woods.configure do |config|
   config.console_mcp_enabled = true
+  config.console_mcp_http_enabled = true
   config.console_mcp_token = ENV.fetch('WOODS_CONSOLE_MCP_TOKEN')
   config.console_mcp_allowed_origins = [
     'https://rails.internal.example', # public Rails/MCP Host
@@ -483,6 +494,14 @@ Until this flag is `true`, none of the transports route traffic:
 
 Keep the flag off in environments where the Console isn't needed (production web tier, CI). Flip it on per-environment, e.g. in `config/environments/development.rb` or a staging-only initializer, once the layers below are configured for that environment's threat model.
 
+### `console_mcp_http_enabled` (HTTP transport gate)
+
+Defaults to `true` for compatibility. Set to `false` alongside
+`console_mcp_enabled = true` to retain stdio access without enabling HTTP or
+requiring an HTTP token. Both flags are read at request time, so settings in
+`config/initializers/woods.rb` take effect. Disabling HTTP does not disable
+blocked-table, redaction, credential-scanning or rollback controls on stdio.
+
 ### `console_blocked_tables` (layer 1: table gate)
 
 Entries are lowercased table names. A tool call is rejected at dispatch time when:
@@ -536,7 +555,11 @@ registered in a supported mode, so this setting does not enable eval.
 Woods::Console::Server.rebuild_credential_index(rails_app: Rails.application)
 ```
 
-This rebuilds the index from the current credentials and hot-swaps it into the active scanner. The swap is atomic on MRI, in-flight scans see either the old or the new index, never a partial one. The method is a no-op (returns `nil`) when no server has been built yet or when `console_credential_defense_enabled` is `false`.
+This reads a fresh encrypted-file and key snapshot, without changing Rails' cached application credentials, and replaces the index in every live embedded Console server in this process. Each response scan keeps one complete index; a rotation cannot change its index halfway through a response. Servers released by their transports are not retained by this registry. A later server construction also reads fresh credentials.
+
+A refresh failure (including missing files or keys, failed decryption, and invalid YAML) raises and leaves every existing index intact. Treat a failed rebuild as an operational failure: resolve the credentials deployment and retry, or restart after verification. A valid empty credential mapping intentionally clears the index. A successful rebuild replaces the old set rather than retaining removed secret values. Custom `rails_app:` collaborators that expose `credentials.config` remain supported; those collaborators own freshness of their returned config.
+
+The method is a no-op (returns `nil`) when no live scanner remains or when `console_credential_defense_enabled` is `false`. Normal server boot still permits unavailable credentials and falls back to the other configured defenses; this permissive boot behavior does not apply to an explicit rebuild.
 
 **Rotation warning.** At boot time, Woods checks whether any credentials file (`config/credentials.yml.enc`, `config/credentials/<env>.yml.enc`) was modified *after* the process started. When it detects this, it emits a `console.credential_index.stale` warn-level structured log line with the file path, mtime, and a hint to restart or call `rebuild_credential_index`. This check is on by default; disable it with:
 
@@ -658,7 +681,8 @@ touching ActiveRecord, and neither is registered in `tools/list`.
 ```ruby
 # config/initializers/woods.rb
 Woods.configure do |config|
-  config.console_mcp_enabled           = true     # mount the Rack middleware via Railtie
+  config.console_mcp_enabled           = true     # master switch
+  config.console_mcp_http_enabled      = true     # false for stdio-only use
   config.console_mcp_token             = ENV.fetch('WOODS_CONSOLE_MCP_TOKEN')
   config.console_embedded_read_tools   = true     # unlock console_sql / console_query
   config.console_redacted_columns      = Woods::DEFAULT_CONSOLE_REDACTED_COLUMNS
@@ -753,6 +777,12 @@ For `console_query`, a schema-qualified column reference such as `orders.total`
 
 Scope hashes accept Ransack-style predicate suffixes (`_eq`, `_not_eq`, `_gt`, `_gteq`, `_lt`, `_lteq`, `_in`, `_not_in`, `_null`, `_not_null`, `_present`, `_blank`, `_matches`), see the [cookbook](MCP_TOOL_COOKBOOK.md#scope-predicates) for the full table. Every column name in a suffixed key is validated before an Arel predicate is built, so SQL injection via column names is not possible.
 
+The internal scope-array defense also uses the connected adapter's dialect and
+MySQL session quote modes when rejecting subqueries and forbidden keywords.
+This protects direct/legacy executor callers; supported tool schemas continue
+to enforce their narrower parameterized scope grammar.
+
+
 ---
 
 ## Troubleshooting

data/docs/DOCKER_SETUP.md

@@ -87,6 +87,12 @@ docker compose exec app bundle exec rake woods:extract_framework
 
 Run the watcher as its own development service or process-manager entry, not as a one-off terminal command. Docker Desktop bind mounts may not deliver reliable native filesystem events; set `WOODS_WATCH_POLL=1` for polling when needed. The watcher updates structural generations automatically, while semantic vectors still require `woods:embed_incremental`.
 
+When host-side tasks or one-off containers read the daemon's shared index,
+`WOODS_WATCH_TRUST_FOREIGN_HOST=1` lets those readers trust its recent heartbeat.
+Set it in each reader process; Docker does not forward host variables by default.
+See [cross-host liveness](WATCH_DAEMON.md#cross-host-liveness) for the 15-minute
+crash-detection bound and single-supervisor requirement.
+
 ### Index persistence
 
 Persist `tmp/woods/` if the index should survive container replacement. A bind mount also makes it available to optional host-side tools:
@@ -185,6 +191,16 @@ Use this only when the application bundle, a supported Ruby, and the Woods execu
 | **Survives container replacement** | Only with a bind/named volume | Yes, on host disk |
 | **Needs Ruby/Woods bundle on host** | No | Yes |
 
+### Index filesystem performance
+
+Bind mounts backed by virtiofs or FUSE can make each hardlink, rename and
+metadata lookup costly. Profile `payload seed`, writes and `payload prune`
+before tuning extraction. A container volume can reduce these costs, but it
+changes host visibility: use a distinct index location per worktree, run MCP
+where that path is visible, and update export/archive paths together. A single
+shared volume for every worktree would mix their indexes. See
+[incremental profiling](INCREMENTAL_EXTRACTION.md#profiling-fixed-costs).
+
 ## Console Server Setup
 
 The Console Server queries live Rails state. There are two launch paths for the same embedded server.
@@ -194,13 +210,16 @@ Before either path can start, deliberately enable live-data access in the Rails
 ```ruby
 Woods.configure do |config|
   config.console_mcp_enabled = true
-  config.console_mcp_token = ENV["WOODS_CONSOLE_MCP_TOKEN"]
+  config.console_mcp_http_enabled = false # stdio-only
 end
 ```
 
 The process exits with status 1 while this master switch is false. Review [Console MCP setup and security](CONSOLE_MCP_SETUP.md) before enabling it.
 
-Stdio does not send the bearer token, but production Rails boot still requires `WOODS_CONSOLE_MCP_TOKEN` to contain at least 32 characters whenever Console is enabled. Provide it to the container through the application's normal secret mechanism. Outside production, omitting it warns and leaves the Console HTTP endpoint guarded with 401.
+This explicitly disables HTTP Console while retaining stdio access; no HTTP
+token is needed at boot. Existing configurations default to HTTP enabled.
+For HTTP deployment, enable the HTTP flag and configure its token, origins
+and TLS using the [Console setup guide](CONSOLE_MCP_SETUP.md#option-c-http-rack-middleware).
 
 ### Comparison