woods 2.0.0.beta2 → 2.0.0.beta4

data/docs/CONFIGURATION_REFERENCE.md

@@ -108,8 +108,9 @@ Columns:
 | `output_dir` | Pathname/String | `Rails.root.join('tmp/woods')` | user-settable | Directory where extracted data is written |
 | `extractors` | Array<Symbol> | `[:models, :controllers, :services, ...]` | accepted, not implemented | Does not select which extractors run. See [Extractors](#extractors) below. |
 | `pretty_json` | Boolean | `true` | user-settable | Format extracted JSON with indentation |
-| `max_context_tokens` | Integer | `8000` | user-settable | Maximum tokens for retrieval context windows |
-| `similarity_threshold` | Float | `0.7` | user-settable | Minimum similarity score (0.0-1.0) for retrieval results |
+| `retrieval_mode` | Symbol | `:semantic` | user-settable | `:semantic` uses configured embeddings; explicit `:lexical` ranks published text without a provider/vector store. See [retrieval modes](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval). |
+| `max_context_tokens` | Integer | `8000` | user-settable | Default context-assembly token budget captured when a retriever is built; per-call budget overrides it |
+| `similarity_threshold` | Float | `0.7` | deprecated, inert | Accepted for compatibility; setting it warns. It does not filter or change retrieval ranking |
 | `context_format` | Symbol | `:markdown` | user-settable | Output format for retrieval: `:claude`, `:markdown`, `:plain`, `:json` |
 | `include_framework_sources` | Boolean | `true` | user-settable | Extract Rails and gem source code |
 | `concurrent_extraction` | Boolean | `false` | user-settable | Enable parallel extraction (experimental) |
@@ -135,6 +136,17 @@ config.embedding_options = {
 }
 ```
 
+OpenAI embedding batches are sent in slices of at most 36 texts, preserving
+input order. For inputs within the API's 8,192-token per-text limit, this stays
+below both the 2,048-input limit and the 300,000-token total request limit.
+See the [OpenAI embedding request contract](https://developers.openai.com/api/reference/resources/embeddings/methods/create).
+Woods retains its conservative 8,191-token chunking ceiling; slicing does not
+make an individually oversized text valid. This bound avoids relying on token
+estimates and adds HTTP requests for batches containing many short chunks.
+All slices must validate, including consistent vector dimensions, before the
+provider returns any vectors for the batch. Ollama and custom providers keep
+their existing batching behavior.
+
 ### Ollama embeddings
 
 ```ruby
@@ -196,6 +208,25 @@ config.vector_store_options = {
 }
 ```
 
+Woods uses an HNSW index over pgvector's `vector` representation, which supports
+**1–2,000 dimensions** ([pgvector's HNSW limits](https://github.com/pgvector/pgvector#hnsw)).
+Unreleased after `2.0.0.beta3`: the adapter rejects wider dimensions before any
+SQL, and `woods:pgvector` rejects invalid widths before writing a migration.
+There is no automatic vector truncation or half-precision conversion.
+
+The default `text-embedding-3-large` output is 3,072 dimensions. With pgvector,
+explicitly request a supported provider output width, for example:
+
+```ruby
+config.embedding_model = 'text-embedding-3-large'
+config.embedding_options = { dimensions: 1536 }
+```
+
+Keep the provider, `vector_store_options[:dimensions]` (when set), and generated
+migration width equal. Changing a stored width requires a compatible new table
+or an intentional index rebuild; changing the setting does not resize old data.
+Use another backend if you need the full 3,072-dimensional output.
+
 Requires the pgvector extension. Run the generator to create migrations:
 
 ```bash
@@ -224,6 +255,15 @@ config.metadata_store_options = {
 }
 ```
 
+Without an explicit `database` option, SQLite uses
+`<output_dir>/metadata.sqlite3`. For `woods:embed` and
+`woods:embed_incremental`, `WOODS_OUTPUT` overrides that directory together
+with the index and embedding dumps. An explicit `database` path (including
+`:memory:`) still takes precedence, so configure a separate path for each
+worktree when overriding it. Existing databases at the old configured output
+path are not moved or deleted; run `woods:embed` for the selected index after
+upgrading to populate its default metadata database.
+
 Requires the `sqlite3` gem in your host bundle. Rails apps backed by
 MySQL or PostgreSQL won't have it by default, selecting `:sqlite`
 without it raises `Woods::ConfigurationError` with install
@@ -236,10 +276,18 @@ unless cross-process metadata persistence matters.
 config.metadata_store = :in_memory
 ```
 
-Pure-Ruby hash-backed store. No external dependencies, no persistence, vectors and metadata both live in the building process and die with
-it. The `_index.json` manifest under `output_dir` is the durable
-metadata for the index MCP server, so this is a reasonable default
-for hosts that don't bundle `sqlite3`.
+Pure-Ruby hash-backed store with no external dependencies. For local vector
+presets, embedding runs persist it as `metadata.msgpack` alongside `vectors.bin`
+in the promoted dump; the index MCP server loads that snapshot at startup or
+reload. Incremental embedding publishes changes to paths, dependencies, and
+other unit metadata even when unchanged source needs no new embedding. A run
+with no content or metadata changes keeps the existing dump and retention
+window. Unreleased after `2.0.0.beta3`: a full `Indexer#index_all` run replaces
+the published corpus even when a custom caller reuses in-memory vector and
+metadata stores. Deleted units, including metadata-only records, are removed;
+an empty full rebuild publishes an empty dump. Failed embedding leaves the
+previous promoted dump and checkpoint intact. Incremental purge guards remain
+unchanged. This is a reasonable default for hosts that don't bundle `sqlite3`.
 
 ## Retrieval cache options
 
@@ -278,6 +326,13 @@ overrides the wrapper defaults for `:embeddings` (24 hours) and `:context`
 (15 minutes). `:memory` accepts `max_entries` (default 500); it ignores
 `default_ttl` because each wrapper write supplies its domain TTL.
 
+`Woods::Cache.cache_key` length-prefixes every component, including a single
+component, so different argument counts cannot share a response. Existing
+multi-component keys used by Woods' wrappers remain unchanged. Custom callers
+using single-component keys must clear their affected persistent cache domain
+when upgrading, since older unprefixed entries can alias the new encoding;
+subsequent calls refill it normally. Namespace clearing still covers both formats.
+
 ## Deployment shapes
 
 Woods supports three deployment shapes, pick the preset that matches yours.
@@ -304,7 +359,7 @@ The embed run writes `woods.json` + `dumps/<ISO8601>/vectors.bin` + `metadata.ms
 
 Requirements:
 - `output_dir` must be set and readable by both the embed process and the MCP server.
-- The MCP server must know the same `output_dir` (pass via `woods-mcp <DIR>` or set `WOODS_DIR`).
+- The MCP server must know the same `output_dir` (pass via `woods-mcp <DIR>` or set `WOODS_DIR`; see MCP path precedence below).
 
 ## Presets
 
@@ -358,12 +413,56 @@ end
 | `extract_navigation_edges` | Boolean | `true` | Extract `link_to`, `redirect_to`, and `form_action` navigation edges from views and controllers |
 | `enable_snapshots` | Boolean | `false` | Enable temporal snapshots. Woods automatically migrates its internal output-directory SQLite store; if SQLite is unavailable, it uses the JSON snapshot store. No Rails migration is required. |
 | `volatile_dependency_ratio` | Float | `3.0` | A dependency whose commit count (last 365 days) exceeds the dependent's by this ratio appears in the `volatile_dependencies` report (top 20, ranked by PageRank). Must be greater than 1. Report only, never a gate. |
+| `volatile_dependency_limit_per_target` | Integer or `nil` | `nil` | Optional maximum report edges per dependency (type and identifier), applied after ranking and before the global top 20. Positive integers only; `nil` leaves the default report unchanged. Distinct relationships consume separate slots. |
 | `graph_cycle_limit` | Integer or `nil` | `500` | How many distinct cycles `GraphAnalyzer` enumerates before it stops. Cycle detection finds one cycle per DFS back-edge, so a dense graph has tens of thousands of them and enumerating every one is the largest single cost of the analysis that runs on every extraction. Set to `nil` for exhaustive enumeration. |
 | `graph_cycle_max_length` | Integer or `nil` | `50` | The longest cycle recorded, in distinct nodes. A back-edge deep in the DFS closes a cycle as long as the path, which on a large graph is thousands of nodes: unreadable as a report and expensive to canonicalize. Set to `nil` to record a cycle of any length. |
 
 | `incremental_blast_radius_depth` | Integer or `nil` | `nil` | How many reverse hops an incremental run walks from a changed file before it stops re-extracting dependents. `nil` keeps the unbounded transitive closure. See the note below before setting it. |
 | `durable_payload_writes` | Boolean | `false` | Force an `fsync` on every payload file as it is written, on top of the single flush every publish already performs. See the note below before setting it. |
 
+**Tuning volatile dependency reports.** Start with
+`graph_analysis.json`'s `stats.volatile_dependency_count`: it counts every
+qualifying edge before either cap, while the array normally keeps only 20.
+Raise `volatile_dependency_ratio` until the remaining candidates are useful
+for your application. A measured 7k-unit app had 544 qualifying edges at 3.0;
+8–10 was a useful ratio there, not a universal recommendation. Commit counts
+cover the last 365 days and need complete git history; missing git data is not
+proof of stability.
+
+If one hot dependency still fills the list, set
+`volatile_dependency_limit_per_target` to a small positive integer such as 3.
+Each dependency keeps its highest-ranked edges before the global limit applies,
+allowing other dependencies into the report. Different relationship labels
+remain separate edges and consume separate slots. With this option enabled,
+`stats.volatile_dependencies_limit_per_target` records the setting and
+`stats.volatile_dependency_reported_count` counts the final persisted array;
+`stats.volatile_dependency_count` still counts all qualifying edges. The extra
+stats are absent at the default `nil`. This limits the report, not the work of
+finding qualifying edges. Run extraction again after changing either setting;
+MCP reads the published report. These findings remain informational, never a
+release or architecture gate.
+
+When the JSON snapshot fallback is in use, malformed JSON, invalid snapshot
+shapes, and files that cannot be read (including concurrent retention removals)
+are warned about and treated as absent. A snapshot needs a hexadecimal string
+`git_sha` matching its filename; `extracted_at` may be a string, null, or omitted.
+When present and non-null, `units` must be an object whose records are objects.
+A malformed record invalidates the entire snapshot, rather than exposing partial
+history. Legacy bare identifier keys, omitted/null unit collections, and optional per-unit hash
+fields remain supported; timestamp strings are not restricted to a new format.
+Unit history limits count matching unit records, not the most recent snapshots
+searched (JSON fallback correction unreleased after `2.0.0.beta3`). A unit
+missing from newer snapshots can still have retained history.
+Snapshot lists and unit history omit unusable files; direct lookup returns no
+snapshot, and a diff with an unavailable snapshot returns empty added, modified,
+and deleted lists. An empty diff in this case is not proof that nothing changed.
+New captures compare against the latest usable snapshot. Unusable SHA-named files
+still count toward retention and are pruned first when the limit is exceeded;
+reading alone does not delete them. Valid legacy snapshots with null or omitted
+timestamps are retained ahead of corrupt files, then treated as oldest among
+usable snapshots. Direct lookup and diff still reject invalid
+caller-supplied SHA paths with an argument error.
+
 `incremental_blast_radius_depth` is unbounded by default because a unit two hops
 out really can have content that depends on the changed file. An STI grandchild
 (`SportsCar < Car < Vehicle`) inherits its grandparent's associations,
@@ -408,11 +507,66 @@ require 'woods/session_tracer/file_store' # the stores are not autoloaded
 
 config.session_tracer_enabled = true
 config.session_store = Woods::SessionTracer::FileStore.new(
-  Rails.root.join('tmp/session_traces')
+  base_dir: Rails.root.join('tmp/session_traces')
 )
 config.session_exclude_paths = ['/health', '/metrics', '/assets']
 ```
 
+### File session retention
+
+`FileStore` accepts `ttl:` in seconds (default `nil`, expiration disabled),
+`max_sessions:` (default `1000`), and `max_requests_per_session:` (default `1000`).
+TTL expires a file when the store clock reaches its modification time plus the
+TTL. Recording after expiry starts a fresh history; expired events are discarded
+before appending or migrating legacy filenames, under the same store lock.
+When legacy and encoded files coexist, each expires independently before any
+surviving histories are merged. Clearing a session is idempotent for supported
+IDs, including Unicode and punctuation, and removes both filename formats when
+applicable.
+
+### Redis session index compatibility
+
+`RedisStore` lists and clears both legacy SET indexes and recency ZSET indexes.
+Listing removes expired members and orders summaries by their last request.
+Reads and clears preserve a legacy SET, so upgrading only readers does not
+break older writers. Type checks and index operations run atomically in Redis
+and tolerate a concurrent writer converting the index.
+
+The first `record` from a newer writer converts the SET to a ZSET atomically.
+Upgrade writers together: older SET writers cannot write after that conversion.
+Migrated members receive score zero, so they evict in lexical order at the
+retention limit until recorded again; new records use their request timestamp.
+Session list contents and TTLs are preserved by the index conversion.
+
+### Solid Cache session retention and compatibility
+
+`Woods::SessionTracer::SolidCacheStore` accepts a direct `SolidCache::Store`.
+Its coordination layer uses private Solid Cache APIs for uncached, per-key
+routing and atomic ownership. Missing APIs raise
+`Woods::SessionTracer::SolidCacheCoordination::BackendError`; a gem's version
+constraint alone does not establish compatibility. The live contract suite
+validates SQLite, PostgreSQL, and MySQL (including MySQL's insert path without
+`RETURNING`). See [the version-validation procedure](../CONTRIBUTING.md#solid-cache-session-compatibility)
+before upgrading Solid Cache.
+
+Session traces are best-effort diagnostic data. The slot directory and record
+rings are bounded by `max_sessions` and `max_requests_per_session`, but the
+following crash/eviction limits remain:
+
+- A crashed admission can strand an active-session mapping outside the directory.
+  These mappings can accumulate across distinct session IDs, with a bounded
+  amount per ID. Recording that session again reclaims its mapping; the normal
+  directory bound is not a hard bound on these stranded keys.
+- If the backend evicts a slot counter while deep crash-orphan records remain
+  in an unoccupied slot, new records can be silently dropped until the counter
+  advances past them. This loss is bounded and self-healing, but the affected
+  requests are not recovered.
+- Losing the epoch key fences every live session. Old traces become unreadable,
+  and each session is admitted again on its next record. This deliberately
+  favors privacy over retention so previously cleared data stays cleared.
+
+Do not use this store as an audit log or the only record of a request.
+
 ## Gem indexing
 
 `config.add_gem` is accepted for forward compatibility but **not implemented**: nothing in the
@@ -525,8 +679,9 @@ deployment guide including defense layers.
 
 | Key | Type | Default | Description |
 |---|---|---|---|
-| `console_mcp_enabled` | Boolean | `false` | Master switch. When `false`, the Railtie does not mount the Console MCP middleware. |
-| `console_mcp_token` | String | `ENV['WOODS_CONSOLE_MCP_TOKEN']` or `nil` | Bearer token required on every Console HTTP request. **Required in production**: the Railtie raises `Woods::ConfigurationError` when `console_mcp_enabled` is true but no token is set. In non-production a missing token warns at boot and every Console request fails closed with `401 Unauthorized`. A configured token shorter than 32 characters raises `Woods::ConfigurationError` at boot in every environment. Generate with `SecureRandom.hex(32)`. |
+| `console_mcp_enabled` | Boolean | `false` | Master switch. When `false`, stdio exits and the mounted Console middleware passes requests through to Rails. |
+| `console_mcp_http_enabled` | Boolean | `true` | HTTP transport switch; effective only while the master switch is on. Set `false` for stdio-only use without HTTP token validation or an active HTTP endpoint. Read at request time. |
+| `console_mcp_token` | String | `ENV['WOODS_CONSOLE_MCP_TOKEN']` or `nil` | Bearer token required on every enabled Console HTTP request. With both Console flags enabled, production boot raises on a missing token; other environments warn and requests fail closed with 401. A configured token shorter than 32 characters raises at boot while HTTP is enabled. Explicit stdio-only configurations skip HTTP token validation. Generate with `SecureRandom.hex(32)`. |
 | `console_mcp_allowed_origins` | Array\<String\> | `%w[http://localhost http://127.0.0.1 http://[::1]]` | `OriginGuard` allowlist. Port is stripped before comparison, so `http://localhost` matches any localhost port. Override for tunneled / internal-dashboard access. |
 | `console_mcp_path` | String | `/mcp/console` | URL path the Rack middleware responds on. |
 | `console_embedded_read_tools` | Boolean | `false` | Register `console_sql` and `console_query` in supported stdio and Rack modes. |
@@ -547,13 +702,15 @@ These variables are read by the gem and its MCP servers at runtime. They complem
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
-| `WOODS_DIR` | `Dir.pwd` | Path to the extraction output directory. |
-| `WOODS_REQUIRE_INDEX` | unset | Set to `"1"` to fail closed: the server refuses to boot (raises `MissingArtifact`) unless a real index (`woods.json`) is present. By default an extract-only host boots in pattern/structural mode without it. |
+| `WOODS_RETRIEVAL_MODE` | `semantic` | Explicit packaged MCP retrieval mode: `semantic` or `lexical`. Lexical reads extraction unit JSON without provider autodetection, credentials or vector artifacts. |
+| `WOODS_DIR` | unset | MCP extraction-index path, after a positional argument and before `WOODS_OUTPUT`. See precedence below. |
+| `WOODS_OUTPUT` | unset | MCP index-path fallback when neither a positional path nor `WOODS_DIR` is set; unreleased after `2.0.0.beta3`. |
+| `WOODS_REQUIRE_INDEX` | unset | Set to `"1"` to fail closed: the server refuses to boot (raises `MissingArtifact`) unless a real index (`woods.json`) is present. By default an extract-only host boots in pattern/structural mode without it. Explicit lexical mode requires a valid published extraction index, not `woods.json`. |
 | `WOODS_ALLOW_AUTODETECT` | unset | **Deprecated no-op.** Auto-detect is now the default; accepted for backward compatibility only. |
 | `WOODS_SEARCH_MAX_SCAN` | `500` | Cap on unit files loaded during a phase-2 (metadata/source_code) `search`. Hitting the cap sets `partial: true` in the response. |
 | `WOODS_SNAPSHOTS` | unset | Set to `"true"` to force-enable temporal snapshot storage, even without a pre-existing SQLite database. |
 | `WOODS_ALLOW_PURGE` | unset | Set to `"1"` to override the 30%-deletion purge guard in `woods:embed`/`woods:embed_incremental`. |
-| `WOODS_PAYLOAD_RETENTION` | `3` | How many past generations' payload directories (`payloads/gen-N/`) to retain, and — when the JSON snapshot store is in use — how many temporal snapshots (`snapshots/`) to keep. A payload pinned by an active reader process is kept temporarily beyond this bound and reconsidered after the pin is released. |
+| `WOODS_PAYLOAD_RETENTION` | `3` | How many past generations' payload directories (`payloads/gen-N/`) to retain, and — when the JSON snapshot store is in use — how many temporal snapshots (`snapshots/`) to keep. JSON snapshot retention counts corrupt or unreadable SHA-named files and prunes them first; the just-captured snapshot is protected. Cleanup failures are non-fatal. A payload pinned by an active reader process is kept temporarily beyond this bound and reconsidered after the pin is released. |
 | `WOODS_MCP_CACHE_TTL_MS` | `10000` | Cache TTL advertised in tool result `_meta`. `0` disables caching. |
 | `WOODS_NO_UPDATE_CHECK` | unset | Set to `"1"` to skip the `woods_status` RubyGems version check. |
 | `XDG_CACHE_HOME` | `~/.cache` | Base directory for the best-effort update-check cache (`$XDG_CACHE_HOME/woods/update_check.json`). An unset or empty value uses `~/.cache`; if the home directory cannot be resolved, Woods falls back to the system temporary directory. |
@@ -563,6 +720,20 @@ These variables are read by the gem and its MCP servers at runtime. They complem
 | `WOODS_QDRANT_URL`, `WOODS_QDRANT_COLLECTION`, `WOODS_QDRANT_API_KEY` | n/a | Override/require Qdrant connection settings when a pgvector/Qdrant-backed index is served outside its host application (no `Woods.configuration` available). |
 | `WOODS_PG_URL` | n/a | Required when a pgvector-backed index is served outside its host application. |
 
+**MCP index path precedence (unreleased after `2.0.0.beta3`):** positional
+argument → `WOODS_DIR` → `WOODS_OUTPUT` → current directory for `woods-mcp`
+and `woods-mcp-http`. `woods-mcp-start` still requires one of the first three;
+it never silently selects the current directory. An explicitly empty
+`WOODS_DIR` remains an invalid override rather than falling through. Earlier
+versions accept the positional path or `WOODS_DIR`; use an explicit path for
+portable client configuration.
+
+Paths are resolved in the MCP process's working directory and filesystem.
+A published index can have `generation.json` pointing to a payload's
+`manifest.json`; a root `manifest.json` is only the legacy flat layout. If
+startup cannot find a manifest, check the reported directory and point at the
+existing index before deciding another extraction is needed.
+
 ### Rake tasks
 
 | Variable | Default | Purpose |
@@ -590,24 +761,52 @@ These variables are read by the gem and its MCP servers at runtime. They complem
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
-| `WOODS_IGNORE_WATCH` | unset | Set to `"1"` to make `woods:incremental`/`woods:clean` proceed even when a daemon is (or claims to be) running. For `woods:incremental` this removes daemon coverage: a git range that fails to resolve then exits 1 instead of standing down (see [Incremental Extraction](./INCREMENTAL_EXTRACTION.md#exit-behavior-in-ci-chains)). |
+| `WOODS_IGNORE_WATCH` | unset | Set to `"1"` to make `woods:incremental`/`woods:clean`/`woods:hook_refresh` proceed even when a daemon is (or claims to be) running. For `woods:incremental` this removes daemon coverage: a git range that fails to resolve then exits 1 instead of standing down (see [Incremental Extraction](./INCREMENTAL_EXTRACTION.md#exit-behavior-in-ci-chains)). |
 | `WOODS_LOCK_WAIT` | `Watch::Daemon::LOCK_STALE_TIMEOUT` (600s) | How long a rake writer waits for `PipelineLock` before exiting non-zero. |
 | `WOODS_WATCH_POLL` | auto-detected | Set to `"1"`/`"0"` to force/disable polling mode (vs. `listen` gem, e.g. in a container without inotify). |
+| `WOODS_WATCH_POLL_INTERVAL` | `1.0` (seconds) | Positive, finite delay between polling scans; also used on native-watcher fallback. Does not force polling. Larger values reduce scan frequency and can delay detection and shutdown. |
 | `WOODS_WATCH_DEBOUNCE` | `0.4` (seconds) | Delay before processing a batch of file-change events. |
 | `WOODS_WATCH_FULL_THRESHOLD` | `50` | Number of changed paths in one batch that triggers a full extraction instead of incremental. |
 | `WOODS_WATCH_IDLE_TIMEOUT` | unset (no timeout) | Seconds of inactivity before the daemon exits. |
 | `WOODS_WATCH_CATCH_UP` | `1` (enabled) | Set to `"0"` to skip generation-watermark catch-up on daemon start. |
+| `WOODS_WATCH_TRUST_FOREIGN_HOST` | unset (disabled) | Set to `"1"` in each task/MCP reader to trust a foreign daemon's heartbeat for up to 15 minutes, without a local pid check. See [cross-host liveness](WATCH_DAEMON.md#cross-host-liveness) for clock bounds, degraded coverage, and startup limitations. |
+
+### Opt-in plugin refresh hooks
+
+These settings control the plugin shell worker. Check installed
+`woods:hook_refresh` support first; the task is unreleased after 2.0.0.beta2.
+See [hook coverage and retry](WATCH_DAEMON.md#hooks-for-agent-sessions) and
+[optional context limits](WATCH_DAEMON.md#optional-bounded-context-hints).
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `WOODS_HOOKS_ENABLED` | unset (disabled) | Exact `1` enables the refresh/session hooks when an index exists. |
+| `WOODS_HOOKS_DISABLED` | unset | Exact `1` disables both refresh and optional context hooks. |
+| `WOODS_HOOK_CONTEXT_ENABLED` | unset (disabled) | Exact `1` enables separate bounded Claude orientation/impact hints; independent of refresh enablement. |
+| `WOODS_HOOK_CONTEXT_COMMAND` | `bundle exec woods-hook-context` | Installed helper argv prefix; startup counts toward the fixed context deadline. Use a wrapper for quoting/container environment. |
+| `WOODS_HOOK_CONTEXT_ROOT` | payload cwd | Explicit runtime-visible application root for context path mapping; set inside a container when host paths differ. |
+| `WOODS_HOOK_RAKE` | `bundle exec rake` | Application command prefix; supports `docker compose exec -T app bundle exec rake`. Use a wrapper for shell quoting or explicit container environment. |
+| `WOODS_SOURCE_CAPTURE` | internal | Private, one-use `woods-extract` child handoff. Do not set or persist this variable manually; see [source freshness](SOURCE_FRESHNESS.md). |
+| `WOODS_HOOK_TIMEOUT_SECONDS` | `600` | PostToolUse worker deadline (SessionStart uses a fixed ten seconds), integer 1–3600 seconds; failed/deferred batches remain queued. Docker-side cancellation requires separate verification. |
+| `WOODS_HOOK_LOCK_STALE_SECONDS` | `1800` | Age used only to reclaim legacy empty mkdir locks; live PID owners are never reclaimed merely by age. |
+
+`woods:hook_refresh[<base64 JSON>]` is the internal plugin transport. Version 1
+contains `output` and an `events` array of `{path, operation}` records; paths are
+application-relative and operations are `add`, `update`, `delete`, or `move`.
+The task validates inputs, defers active daemons with exit 75 before Rails boot,
+and checks publication failure before acknowledging work. Use ordinary extraction
+tasks for manual refreshes; hook transport is not a general shell execution API.
 
 ### Extraction rake tasks
 
 | Variable | Default | Purpose |
 |----------|---------|---------|
-| `WOODS_OUTPUT` | `Woods.configuration.output_dir` | Overrides the output directory for `woods:extract`/`woods:incremental`/`woods:watch` without editing the initializer. |
+| `WOODS_OUTPUT` | `Woods.configuration.output_dir` | Overrides the output directory for extraction/watch and embedding tasks without editing the initializer; embedding also places its default SQLite metadata database there. Explicit database options take precedence. |
 | `CHANGED_FILES` | unset | Comma-separated explicit changed-path list for `woods:incremental`; when set, git range resolution is skipped entirely. |
 | `CI_COMMIT_BEFORE_SHA`, `CI_COMMIT_SHA` | unset (GitLab) | Build the diff range `<before>..<after>` for `woods:incremental`. A zero before-SHA (new branch) makes the range unresolvable, which exits 1 unless a running daemon covers the index. |
 | `GITHUB_BASE_REF` | unset (GitHub Actions) | Build the diff range `origin/<ref>...HEAD` for `woods:incremental`; an unfetched ref makes the range unresolvable, same exit behavior. |
 | `RAILS_ENV` | `development` | Rails environment the rake tasks boot in. |
-| `WOODS_PROFILE` | unset | Set to `"1"` to log one `[Woods] [profile] <phase> in N.NNs` line per run phase (payload seed, previous graph load, eager load, extraction or blast radius and re-extraction, type index, graph analysis, flows, manifest and summary, publish). Complements the per-extractor timing lines, which cover extraction only. Off by default and free when off. |
+| `WOODS_PROFILE` | unset | Set to `"1"` to log disjoint `[Woods] [profile] <phase> in N.NNs` durations, including git enrichment, reconciliation, payload sync, pointer publication (`publish`) and retention (`payload prune`). Separate `[profile total]` lines report whole extraction wall time, including unprofiled setup and failed runs; never add these totals to phase durations. Excludes process/Rails boot before extraction. Off by default. |
 | `WOODS_GIT_DIR` | unset | Absolute path to the canonical git directory. Wins over the repository Woods would otherwise find, at all three of its git call sites: per-unit `commit_count`/`change_frequency` (enrichment), `manifest.json`'s `git_branch`/`git_sha` (provenance), and the `woods:incremental` diff range. All three build their command line with `Woods::GitCommand.argv`. |
 | `GIT_BRANCH`, `GIT_SHA` | unset | Provenance for a checkout with no `.git` at all (a source tarball, a Docker `COPY` that excludes it). Ignored when a `.git` is present but unresolvable, so a stale build arg cannot mask a worktree. |
 
@@ -645,6 +844,49 @@ WOODS_GIT_DIR=/canonical-git bundle exec rake woods:extract
 
 The `woods-mcp` bootstrapper emits a one-line STDERR banner at startup indicating whether semantic search is enabled and which provider is active. If no key/instance is found, pattern search still works and `codebase_retrieve` surfaces an actionable fix message.
 
+## Git enrichment history
+
+Current source requires **Git 2.31 or newer** for optional per-unit git
+metadata. Extraction still succeeds when git is unavailable or history cannot
+be read completely. Git enrichment is omitted in either case; a failed or
+incomplete streamed history read logs a warning.
+This requirement and the history policy below are unreleased after 2.0.0.beta2.
+
+Per-unit enrichment also requires a non-shallow repository. A shallow checkout
+or a failed repository-depth probe omits enrichment with one warning per
+extractor instance. Fetch complete history (`git fetch --unshallow`, or
+`actions/checkout` with `fetch-depth: 0`) and run full extraction to refresh
+retained metadata. If depth cannot be verified, check git access and version.
+A source archive without a repository remains quiet.
+
+Full and incremental extraction use one streamed `HEAD` history walk, restricted
+to the last 365 days by git's `--since` traversal. Only requested app-owned paths
+are retained. Commit counts and contributors describe **HEAD-reachable touched-path
+events**, independent of which other paths are requested:
+
+- A root commit compares with an empty tree; a normal commit compares with its parent.
+- A merge compares with its first parent, while traversal still visits all parents.
+  A normal merge can therefore count both a side commit and the merge that introduces
+  its change. An `ours` merge touches no paths itself, but its side commits remain
+  reachable and count. A conflict-resolution merge counts when its result differs
+  from the first parent.
+- Renames are deletion/addition events at exact current names; Woods never follows
+  previous names. Unmerged branches, remote-only refs, and checkpoint refs are excluded.
+- `change_frequency` uses total events and events newer than 90 days. Contributors
+  and recent commits retain their existing top-five limit; recent commits and
+  `last_modified` follow git's traversal order, not a separate global timestamp sort.
+
+**Compatibility:** merge-related counts, authors, recent commits, and derived
+churn analysis can differ from the old 500-path batches, which used git's
+pathspec history simplification. This is an intentional semantics change.
+Run a full extraction after upgrading to replace retained incremental metadata
+consistently. No configuration key enables the new policy; rollback uses the
+previous gem plus a full extraction. Shallow checkouts still provide truncated
+history; fetch the complete history for complete 365-day evidence.
+
+The single walk removes repeated pathspec matching. Its end-to-end speedup on
+the original #305 host has not yet been measured.
+
 ## Database compatibility
 
 All storage options work with both MySQL and PostgreSQL, except:
@@ -653,3 +895,12 @@ All storage options work with both MySQL and PostgreSQL, except:
 - **SQLite metadata store**: uses a standalone SQLite database file, independent of your app's database
 
 See [BACKEND_MATRIX.md](BACKEND_MATRIX.md) for the full compatibility matrix.
+
+### Explicit retrieval and discovery scope
+
+On a server whose tool schema advertises them, `packages` and `source_paths` narrow
+`search` and `codebase_retrieve` before candidate limits. These are per-call
+arguments, not configuration settings. Inspect applied scope and completeness;
+a narrow graph query can omit relevant cross-boundary dependencies. See the
+[scope contract](RETRIEVAL_GUIDE.md#explicit-package-and-source-path-scopes) for
+root/nested ownership, path normalization, errors, storage support, and cost.

data/docs/CONSOLE_MCP_SETUP.md

@@ -26,13 +26,23 @@ The simplest setup. The `woods:console` rake task boots Rails, then starts the e
 ```ruby
 Woods.configure do |config|
   config.console_mcp_enabled = true
-  config.console_mcp_token = ENV["WOODS_CONSOLE_MCP_TOKEN"]
+  config.console_mcp_http_enabled = false # stdio-only; no HTTP endpoint
 end
 ```
 
-The stdio and Docker entry points exit with status 1 while this setting is false. Enabling it grants the MCP process live read access under the blocked-table, redaction, and credential-scanning controls described below.
+The stdio and Docker entry points exit with status 1 while `console_mcp_enabled` is false. Enabling it grants the MCP process live read access under the blocked-table, redaction, and credential-scanning controls described below.
 
-The token authenticates HTTP requests; a stdio client does not send it. Production Rails boot nevertheless requires a token of at least 32 characters whenever Console MCP is enabled, even for a stdio-only setup. Store `WOODS_CONSOLE_MCP_TOKEN` in the application's normal secret store. Outside production a missing token warns and leaves Console HTTP guarded with 401 — the boot warning names both transports, so a stdio-only setup can tell that the 401 is not its symptom and that its own session still works.
+`console_mcp_http_enabled` defaults to `true` to preserve existing HTTP
+setups. Set it to `false` for stdio-only use: HTTP guards and the Console
+middleware pass requests through to Rails, and boot skips HTTP token checks,
+including in production. Stdio does not send or consume a bearer token.
+
+If both Console flags are enabled, HTTP still requires a token of at least
+32 characters. Missing tokens warn outside production and every guarded
+request returns 401; production refuses to boot. A configured short token
+raises at boot in every environment while HTTP is enabled. Store the token
+in the application's normal secret store. Before enabling HTTP later, set
+its token, allowed origins and TLS as described in [Option C](#option-c-http-rack-middleware).
 
 ### How it works
 
@@ -172,6 +182,7 @@ In an initializer (`config/initializers/woods.rb`):
 ```ruby
 Woods.configure do |config|
   config.console_mcp_enabled = true
+  config.console_mcp_http_enabled = true
   config.console_mcp_token = ENV.fetch('WOODS_CONSOLE_MCP_TOKEN')
   config.console_mcp_allowed_origins = [
     'https://rails.internal.example', # public Rails/MCP Host
@@ -187,6 +198,12 @@ the Rails server environment. The middleware stack registers automatically via
 the gem's Railtie and requires `Authorization: Bearer <token>` on every Console
 request. Missing or incorrect tokens receive `401 Unauthorized`.
 
+The HTTP authentication scheme is ASCII case-insensitive (`Bearer`, `bearer`,
+or `BEARER`); the token remains case-sensitive and must match exactly after one
+space. This applies to both Console HTTP and `woods-mcp-http`. Case-insensitive
+scheme support is unreleased after `2.0.0.beta3`; use the canonical `Bearer`
+spelling in client configuration for compatibility with earlier releases.
+
 For non-loopback access, `console_mcp_allowed_origins` must include the public
 Rails/MCP host. If a browser-based client sends an `Origin` header from a
 different host, include that exact origin too. This allow-list controls both
@@ -483,6 +500,14 @@ Until this flag is `true`, none of the transports route traffic:
 
 Keep the flag off in environments where the Console isn't needed (production web tier, CI). Flip it on per-environment, e.g. in `config/environments/development.rb` or a staging-only initializer, once the layers below are configured for that environment's threat model.
 
+### `console_mcp_http_enabled` (HTTP transport gate)
+
+Defaults to `true` for compatibility. Set to `false` alongside
+`console_mcp_enabled = true` to retain stdio access without enabling HTTP or
+requiring an HTTP token. Both flags are read at request time, so settings in
+`config/initializers/woods.rb` take effect. Disabling HTTP does not disable
+blocked-table, redaction, credential-scanning or rollback controls on stdio.
+
 ### `console_blocked_tables` (layer 1: table gate)
 
 Entries are lowercased table names. A tool call is rejected at dispatch time when:
@@ -536,7 +561,11 @@ registered in a supported mode, so this setting does not enable eval.
 Woods::Console::Server.rebuild_credential_index(rails_app: Rails.application)
 ```
 
-This rebuilds the index from the current credentials and hot-swaps it into the active scanner. The swap is atomic on MRI, in-flight scans see either the old or the new index, never a partial one. The method is a no-op (returns `nil`) when no server has been built yet or when `console_credential_defense_enabled` is `false`.
+This reads a fresh encrypted-file and key snapshot, without changing Rails' cached application credentials, and replaces the index in every live embedded Console server in this process. Each response scan keeps one complete index; a rotation cannot change its index halfway through a response. Servers released by their transports are not retained by this registry. A later server construction also reads fresh credentials.
+
+A refresh failure (including missing files or keys, failed decryption, and invalid YAML) raises and leaves every existing index intact. Treat a failed rebuild as an operational failure: resolve the credentials deployment and retry, or restart after verification. A valid empty credential mapping intentionally clears the index. A successful rebuild replaces the old set rather than retaining removed secret values. Custom `rails_app:` collaborators that expose `credentials.config` remain supported; those collaborators own freshness of their returned config.
+
+The method is a no-op (returns `nil`) when no live scanner remains or when `console_credential_defense_enabled` is `false`. Normal server boot still permits unavailable credentials and falls back to the other configured defenses; this permissive boot behavior does not apply to an explicit rebuild.
 
 **Rotation warning.** At boot time, Woods checks whether any credentials file (`config/credentials.yml.enc`, `config/credentials/<env>.yml.enc`) was modified *after* the process started. When it detects this, it emits a `console.credential_index.stale` warn-level structured log line with the file path, mtime, and a hint to restart or call `rebuild_credential_index`. This check is on by default; disable it with:
 
@@ -658,7 +687,8 @@ touching ActiveRecord, and neither is registered in `tools/list`.
 ```ruby
 # config/initializers/woods.rb
 Woods.configure do |config|
-  config.console_mcp_enabled           = true     # mount the Rack middleware via Railtie
+  config.console_mcp_enabled           = true     # master switch
+  config.console_mcp_http_enabled      = true     # false for stdio-only use
   config.console_mcp_token             = ENV.fetch('WOODS_CONSOLE_MCP_TOKEN')
   config.console_embedded_read_tools   = true     # unlock console_sql / console_query
   config.console_redacted_columns      = Woods::DEFAULT_CONSOLE_REDACTED_COLUMNS
@@ -736,12 +766,12 @@ Each transaction sets a statement timeout before any query runs. The default is
 
 `SqlValidator` rejects non-read-only SQL at the string level, before any database interaction.
 
-Validation runs **once**, inside the executor, with the dialect of the live adapter. There is deliberately no earlier dialect-blind pre-check in the tool handler: a validator built without a dialect is the conservative MySQL+PostgreSQL union, and running it first meant a MySQL host rejected statements whose `\'`/backtick grammar produces a spuriously forbidden PostgreSQL view — the adapter-aware acceptance below could never be reached on a real transport. The executor raises `SqlValidationError` for anything it refuses, which the dispatch pipeline renders as a tool error, so nothing is ungated.
+Validation runs **once**, inside the executor, with the dialect of the live adapter. There is deliberately no earlier dialect-blind pre-check in the tool handler: a validator built without a dialect is the conservative union of supported dialects, and running it first meant a MySQL host rejected statements whose `\'`/backtick grammar produces a spuriously forbidden PostgreSQL view — the adapter-aware acceptance below could never be reached on a real transport. The executor raises `SqlValidationError` for anything it refuses, which the dispatch pipeline renders as a tool error, so nothing is ungated.
 
 
 - **Allowed prefixes:** `SELECT`, `WITH...SELECT`, and plain `EXPLAIN`. `EXPLAIN ANALYZE` is rejected, it executes the query rather than just planning it (both the whitespace and `EXPLAIN (ANALYZE, …)` option-list spellings).
 - **Rejected prefixes:** `INSERT`, `UPDATE`, `DELETE`, `MERGE`, `DROP`, `ALTER`, `TRUNCATE`, `CREATE`, `GRANT`, `REVOKE`
-- **Rejected anywhere in query:** `UNION`, `INTO`, `COPY`; row-lock clauses (`FOR UPDATE`, `FOR NO KEY UPDATE`, `FOR SHARE`, `FOR KEY SHARE`, `FOR UPDATE NOWAIT`/`SKIP LOCKED`, MySQL `LOCK IN SHARE MODE`) — these take live row locks even inside the rolled-back transaction. The lock check is adapter-aware: `console_sql` validates with the active adapter's dialect, including MySQL double-quoted strings/backtick identifiers and PostgreSQL quoted identifiers/E-strings. Unknown adapters conservatively scan both normalizations. Every view is scanned under both MySQL executable-comment (`/*!...*/`) semantics, so `#` comments and version-guarded comments cannot split a clause apart.
+- **Rejected anywhere in query:** `UNION`, `INTO`, `COPY`; row-lock clauses (`FOR UPDATE`, `FOR NO KEY UPDATE`, `FOR SHARE`, `FOR KEY SHARE`, `FOR UPDATE NOWAIT`/`SKIP LOCKED`, MySQL `LOCK IN SHARE MODE`) — these take live row locks even inside the rolled-back transaction. The lock check is adapter-aware: `console_sql` validates with the active adapter's dialect, including MySQL double-quoted strings/backtick identifiers and PostgreSQL quoted identifiers/E-strings. Unknown adapters conservatively scan all supported normalizations. Every view is scanned under both MySQL executable-comment (`/*!...*/`) semantics, so `#` comments and version-guarded comments cannot split a clause apart.
 - **Function allowlist (the authoritative function control):** every function-call-shaped identifier must appear in `ALLOWED_FUNCTIONS`, a conservative set of pure read-only functions (aggregates, window functions, string/number/date/JSON readers) kept portable across MySQL, PostgreSQL, and SQLite. Anything else is rejected by name, quoted forms (`"pg_terminate_backend"(…)`) included. This is an allowlist because a denylist cannot enumerate every side-effecting function (`nextval`, `pg_advisory_lock`, `pg_terminate_backend`, …). A legacy `DANGEROUS_FUNCTIONS` denylist (`pg_sleep`, `lo_import`, `lo_export`, `pg_read_file`, `pg_write_file`, `load_file`, `sleep`, `benchmark`) still runs first as belt-and-suspenders.
 - **Rejected patterns:** multiple statements (semicolons), writable CTEs (every `AS (...)` body is checked, so a writable CTE in any WITH position is refused — `WITH a AS (SELECT 1), b AS (DELETE FROM users RETURNING *) SELECT * FROM b`), a CTE list attached to top-level DML (`WITH a AS (SELECT 1) DELETE FROM users RETURNING *`), comment-hidden injections
 
@@ -753,6 +783,12 @@ For `console_query`, a schema-qualified column reference such as `orders.total`
 
 Scope hashes accept Ransack-style predicate suffixes (`_eq`, `_not_eq`, `_gt`, `_gteq`, `_lt`, `_lteq`, `_in`, `_not_in`, `_null`, `_not_null`, `_present`, `_blank`, `_matches`), see the [cookbook](MCP_TOOL_COOKBOOK.md#scope-predicates) for the full table. Every column name in a suffixed key is validated before an Arel predicate is built, so SQL injection via column names is not possible.
 
+The internal scope-array defense also uses the connected adapter's dialect and
+MySQL session quote modes when rejecting subqueries and forbidden keywords.
+This protects direct/legacy executor callers; supported tool schemas continue
+to enforce their narrower parameterized scope grammar.
+
+
 ---
 
 ## Troubleshooting
@@ -823,7 +859,44 @@ quote and comment rules. MySQL also reads the executing session's `ANSI_QUOTES`
 and `NO_BACKSLASH_ESCAPES` settings for validation, protected-column scanning, and
 table gating; adjacent subtraction operators are not assumed to begin a comment.
 Direct scanner callers without session settings use conservative quote-mode scans.
+SQLite read SQL accepts simple ASCII bare, double-quoted, or backtick identifiers
+(letters, digits, and underscores, starting with a letter or underscore).
+Use whitespace after `FROM` and `JOIN`, and use `SELECT` subqueries rather than
+parenthesized table groups. Bracket-quoted and string-quoted names, quoted names
+containing punctuation, and unsupported table-reference syntax are refused before
+execution, because they cannot be reliably checked against the configured table
+policy. Ordinary string literals remain supported. This restriction is part of
+2.0.0.beta4; keep read tools disabled on older versions when this policy is
+needed. Confirm that the release is available before selecting it. The Rails-version
+integration lane checks these boundaries on real SQLite.
+
 The contributor live-backend lane exercises these boundaries
 through Console requests against PostgreSQL and MySQL. Keep read tools disabled
 unless live SQL access is needed, and retain the configured blocked-table and
 redaction policies when diagnosing a rejected request.
+
+## Console policy corrections in 2.0.0.beta4
+
+In `2.0.0.beta4`, the default model-reading tools check the resolved relation
+against `console_blocked_tables` before fetching records or counts. This includes
+application-defined default scopes and the parent lookup for association counts.
+The checked relation is reused for execution so a dynamic default scope is not
+resolved twice. These checks do not change which Console tools are enabled.
+
+Response handling redacts protected fields before invoking serializers, converts
+the remaining response to JSON-compatible values, then redacts and scans that
+normalized tree before either JSON or Markdown rendering. Symbol values and custom
+JSON serializers therefore receive the same credential checks as ordinary strings.
+Custom values in Markdown now use their JSON-compatible representation. Numbers,
+booleans, nulls, and ordinary record shapes retain their existing meanings.
+
+For SQLite SQL, keyword spellings receive function-policy exceptions only where
+supported query grammar requires them. PostgreSQL reserved-keyword grammar is
+preserved. Unsupported parenthesized offset expressions on other dialects may
+require a plain numeric offset. Keep the configured access and credential policies
+in place when adjusting a query.
+
+These corrections require `2.0.0.beta4` or a reviewed development revision that
+contains them. Confirm that a patched release is available before selecting it.
+On affected versions, disable Console where these policies are required; Index MCP
+can stay enabled because it reads the published code index separately.