woods 2.0.0.beta4 → 2.0.1

data/docs/MCP_HTTP_TRANSPORT.md

@@ -86,16 +86,68 @@ Clients must send `Authorization: Bearer $WOODS_MCP_HTTP_TOKEN` on every request
 
 ### Browser origins (DNS rebinding defense)
 
-A second middleware, `Woods::MCP::OriginGuard`, rejects requests whose `Origin` header is outside an allow-list. Requests without an `Origin` header (curl, MCP stdio clients, server-to-server) pass through, bearer auth still gates them.
+**Unreleased diagnostic correction:** invalid origin encodings also refuse before
+binding HTTP. `woods-mcp-http` exits 2 with one bounded `ConfigurationError`
+message naming the invalid entry; re-enter that origin using an ASCII hostname
+(or its Punycode form). No allowlist keeps the existing defaults. Explicit entries
+are literal origins, not wildcard patterns.
+Retain the actual request Host through a reverse proxy; forwarded headers do not
+replace it. When authentication is configured, requests without Host still pass
+through bearer authentication.
+
+In the 2.0.1 maintenance patch, preflight and SDK dispatch share an immutable
+normalized policy. A configured list replaces the default browser origins;
+include loopback explicitly if needed. Cross-origin ports must match an entry,
+while omitted default HTTP(S) ports match their explicit 80/443 forms. A portless
+entry permits same-authority traffic rather than every cross-port browser origin.
+Console defaults remain HTTP loopback; Index HTTP defaults include HTTP and HTTPS
+loopback. When no list is set, these defaults are unchanged. Invalid entries fail
+at boot, naming the offending entry. Restart after changing the configuration.
+
+A second middleware, `Woods::MCP::OriginGuard`, rejects requests whose `Origin` header is outside an allow-list. Requests without an `Origin` header still pass through Host validation and configured bearer authentication.
 
 | Scenario          | `WOODS_MCP_HTTP_ALLOWED_ORIGINS`      | Origins accepted                                                |
 |--------------------|-----------------------------------------|-------------------------------------------------------------------|
-| default            | unset                                    | `http(s)://localhost`, `127.0.0.1`, `::1` (any port)              |
+| default            | unset                                    | HTTP(S) loopback, matching request authority; explicit entries for cross-port origins              |
 | explicit list      | `https://app.example.com`                | exactly `https://app.example.com`, loopback no longer allowed  |
 | multiple origins   | `https://a.example,https://b.example`    | each listed origin                                                |
 
 `OPTIONS` preflights are answered with the matching `Access-Control-Allow-*` headers; successful responses carry `Access-Control-Allow-Origin` and `Vary: Origin`. `Access-Control-Expose-Headers: Mcp-Session-Id` appears only in legacy session mode (`WOODS_MCP_HTTP_STATELESS=0`).
 
+### Origin configuration compatibility
+
+These details apply to the supporting security-patch revisions described above.
+
+The Index HTTP launcher names `WOODS_MCP_HTTP_ALLOWED_ORIGINS` and the offending
+entry when refusing malformed origin configuration, including under a POSIX
+(`C`) locale. It emits one diagnostic before index resolution or HTTP binding.
+
+Configured origins are literal values, with lowercasing, one trailing slash
+removed and HTTP(S) default-port normalization. Wildcard-looking hostnames are
+literal hostnames, not patterns. A nonempty explicit list replaces browser-origin
+defaults, including loopback; add the loopback browser origins you need. Console's
+configured defaults are HTTP loopback; the Index defaults include HTTP and HTTPS
+loopback. Loopback **Host** acceptance is separate from browser-origin acceptance.
+
+Ruby-configured 2.x origin entries reject surrounding whitespace; 1.6.4 trims it.
+The `woods-mcp-http` comma-separated environment setting trims each entry on both
+lines. Use whitespace-free values for portable configuration. The exact-port and
+same-authority rules above still apply; literal matching does not imply wildcard
+or arbitrary cross-port access.
+
+The guards read the actual `Origin` and `Host`; `Forwarded` and
+`X-Forwarded-*` do not replace them. Preserve the public Host through a proxy.
+A genuinely absent Host is accepted by the Host policy, but any supplied Origin
+must still pass its own check. Console requests and token-configured Index
+requests still require bearer authentication before dispatch. The loopback-only
+Index mode without a token retains its documented unauthenticated behavior;
+CORS preflights do not dispatch tools.
+
+Malformed Origin/Host values receive constant `403` responses without reflecting
+the values. Invalid Authorization receives `401` when it reaches the auth guard.
+Malformed HTTP framing can instead receive Puma's `400` before Rack runs. Other
+HTTP servers may reject framing at their own boundary.
+
 ### TLS termination
 
 The server speaks plain HTTP. Any deployment beyond a single trusted host should front it with a reverse proxy that handles TLS, HTTP/2, and connection limits.

data/docs/MCP_SERVERS.md

@@ -56,6 +56,12 @@ Prefer the application's bundle and a project-scoped configuration:
 
 `woods-mcp-start` checks that the directory and published manifest exist, then replaces itself with `woods-mcp`. It does not install dependencies or restart a crashed process.
 
+The MCP client owns this reader process. Automatic index maintenance has its own
+[watcher startup](WATCH_DAEMON.md#managed-development-startup): configuring MCP
+does not enable it. A running reader sees new published generations on subsequent
+calls without reconnecting. Verify a real edit, not only a successful connection;
+see [automatic maintenance](AUTOMATIC_MAINTENANCE.md).
+
 `woods_status.index.woods_version` identifies the last publisher of the served
 manifest; `server.version` identifies the running MCP reader. Missing writer
 provenance is `null`. See [manifest writer provenance](PUBLISHED_INDEX.md#manifest-writer-provenance).
@@ -133,12 +139,23 @@ before using `codebase_retrieve`, including after reload. The guidance grants
 no extraction, configuration-change, or Console authorization. Detailed usage
 belongs in the [agent guide](AGENT_GUIDE.md).
 
-This addition is unreleased after `2.0.0.beta2`. The SDK omits `instructions`
+This addition is included in Woods `2.0.0`. The SDK omits `instructions`
 when negotiating protocol `2024-11-05`; that behavior is preserved. Older gems,
 legacy clients, and clients that do not show server instructions can use the
 agent guide or investigation skill. Leave protocol negotiation enabled rather
 than pinning a newer version solely to obtain guidance.
 
+### Structural and semantic readiness
+
+`ready` describes the published structural index. A reachable embedding provider
+and bootstrap state `hydrated` do not establish that semantic stores contain
+data. Supporting readers also report `retriever.corpus`: locally known vector
+and metadata entry counts, counts by type, and whether those stores are empty,
+populated, or unknown. These are stored records (including chunks), not a count
+of extracted units with verified embedding coverage. Positive counts do not
+prove complete coverage or working provider access. See
+[semantic corpus diagnostics](RETRIEVAL_GUIDE.md#semantic-corpus-diagnostics).
+
 ### Tools (29 — 14 registered in the packaged default)
 
 The Index Server defines 29 schemas across core and conditional capabilities. The normal packaged executable registers the 14 tools below; the remaining schemas require the specialized wiring described afterward.
@@ -194,7 +211,7 @@ process after publishing a new embedded index.
 
 ### Graph-analysis pages
 
-Unreleased after `2.0.0.beta3`: `graph_analysis` enforces its advertised default
+Included in Woods `2.0.0`: `graph_analysis` enforces its advertised default
 of 20 rows per section. Pass `limit` and `offset` to page one selected `analysis`
 or each section of `analysis: "all"`. Explicit limits also bound nested hub
 `dependents` lists. Older servers may return every section row when `limit` is
@@ -214,7 +231,7 @@ they do not establish complete source-reference coverage.
 Search responses retain `query`, `result_count`, and `results`; `result_count`
 is the number returned, not an estimated total. The additive `completeness`
 object describes the requested types, literal filters, and fields in the pinned
-generation. This contract is unreleased after `2.0.0.beta2`.
+generation. This contract is included in Woods `2.0.0`.
 
 | `reason` | `status` | `has_more` | `total_matches` |
 |---|---|---|---|
@@ -260,16 +277,16 @@ Supporting servers expose the annotated, paginated traversal result in
 stdio and HTTP servers. Read `data.total_is_exact`, `data.graph_coverage`, budget
 counters and optional explanation witnesses there; `content[0].text` and
 `structuredContent.text` keep the same human-readable rendering. No `format`
-tool argument is needed or accepted. This additive data payload is unreleased
-after `2.0.0.beta3`; verify the installed response before relying on it. Older
+tool argument is needed or accepted. This additive data payload is included in
+Woods `2.0.0`; verify the installed response before relying on it. Older
 human-renderer responses can carry only text. The structured nodes and witnesses
 cover the same page, not an additional traversal or an unpaginated graph.
 
 Successful responses carry `graph_coverage` with `scope: "published_relationships"`,
 `source_references: "not_exhaustive"`, and a human-readable `notice`. Text formats
 show the same notice, including compact, root-only and empty-page responses.
-This response metadata and the total exactness field below are unreleased after
-Woods `2.0.0.beta3`; older servers need the same conservative interpretation.
+This response metadata and the total exactness field below are included in Woods
+`2.0.0`; older servers need the same conservative interpretation.
 
 ### Dependency traversal budgets
 
@@ -310,14 +327,14 @@ for stable pages. No wall-clock deadline is used, so cutoffs are deterministic.
 Budgets cover traversal work after per-generation graph loading and cache
 preparation (JSON parsing, typed-edge normalization, node types and database
 metadata). They do not cap that initial load, elapsed time, or total process
-memory. These arguments are unreleased in Woods 2.0.0.beta2; check the connected
+memory. These arguments are included in Woods `2.0.0`; check the connected
 server's tool schema before sending them to an older installation.
 
 ### Traversal explanations
 
 Supporting development versions accept `explain: true` on `dependencies` and
-`dependents`. Check the connected schema first; this option is unreleased after
-2.0.0.beta2. Omitted or false keeps the existing compact response.
+`dependents`. Check the connected schema first; this option is included in Woods
+`2.0.0`. Omitted or false keeps the existing compact response.
 
 The additive `explanation` object contains:
 
@@ -343,7 +360,7 @@ it describes identifier-level reachability, never a uniquely typed path.
 A true value means only that identities along this witness have unambiguous
 types. It does not establish source-reference coverage or observed execution.
 Text labels this `witness types unambiguous=yes/no`; the JSON key and its meaning
-remain unchanged. The text label change is unreleased after `2.0.0.beta3`.
+remain unchanged. The text label change is included in Woods `2.0.0`.
 `types` filters retain the compact traversal's identifier-level semantics: any
 registered type can qualify a name, while edge evidence keeps its actual source
 owner. Multiple relationship kinds between the same endpoints remain separate.

data/docs/MCP_TOOL_COOKBOOK.md

@@ -356,7 +356,7 @@ column.
 Use `lookup` on the returned identifier for source, actions, and routes. Search
 `source_code` for textual matches beyond names. Supporting versions distinguish
 exact totals from a bounded result prefix; `partial` means this is discovery,
-not an exhaustive list. Completeness metadata is unreleased after `2.0.0.beta2`;
+not an exhaustive list. Completeness metadata is included in Woods `2.0.0`;
 see the [search contract](MCP_SERVERS.md#search-completeness).
 
 ---

data/docs/MCP_WORKTREE_SETUP.md

@@ -66,7 +66,19 @@ The published payload's `manifest.json` records the extraction's `git_branch` an
 
 Woods uses worktree-aware Git commands. If a present `.git` cannot be resolved, provenance is `"unknown"`; stale `GIT_BRANCH`/`GIT_SHA` values are not substituted. Those environment variables are fallbacks only when the root has no `.git` or Git is unavailable. Temporal snapshots skip an unknown SHA.
 
-For extraction in a container, make the canonical Git directory and the worktree's pointer resolvable there. Mounting only the private worktree Git directory can leave its shared object store unreachable. Follow the [Git provenance troubleshooting guide](TROUBLESHOOTING.md) for mount and `WOODS_GIT_DIR` guidance, and the [published index layout](INDEX_LAYOUT.md) when locating the manifest.
+For extraction in a container, mount the complete shared Git layout at the
+original path so the worktree's `.git` pointer resolves without an override,
+or select `/mounted-common/worktrees/<id>` with `WOODS_GIT_DIR` inside a
+relocated complete mount. Derive `<id>` from Git metadata, not the branch name.
+Selecting the shared root instead uses the primary checkout's HEAD, affecting
+provenance, per-file history, and incremental paths. Follow the
+[worktree mount and verification steps](TROUBLESHOOTING.md#git-directory-mounts-for-linked-worktrees)
+and the [published index layout](INDEX_LAYOUT.md) when locating the manifest.
+
+Compare the branch and exact SHA in the extraction environment with the intended
+worktree, then verify the published manifest after full extraction. A commit
+alone may not trigger the source-file watcher; use full `woods:extract` when
+current Git history is required.
 
 ## Troubleshooting
 

data/docs/PUBLISHED_INDEX.md

@@ -111,7 +111,7 @@ The reader wraps `Woods::MCP::IndexReader` with `auto_refresh: false`; the unit
 
 ### Actual unit types and directory families
 
-Unreleased after `2.0.0.beta3`: `unit` and `units` accept actual published
+Included in Woods `2.0.0`: `unit` and `units` accept actual published
 `graphql_type`, `graphql_mutation`, `graphql_resolver`, `graphql_query`, and
 `gem_source` types. Enumeration preserves each unit's actual type rather than
 labeling every GraphQL member `graphql` or every gem source `rails_source`.

data/docs/README.md

@@ -10,7 +10,7 @@ Woods extracts runtime-accurate Rails context and serves it to coding agents thr
 | Ask an agent to install or configure Woods | [Agent setup runbook](AGENT_SETUP.md) | A safe, reviewable install with an agent handoff report |
 | Configure an MCP client or Docker path | [MCP servers](MCP_SERVERS.md) | A working Index Server and, if authorized, an optional Console Server |
 | Use Woods tools as an agent | [Agent guide](AGENT_GUIDE.md) | A repeatable query workflow for code context, flows, and blast radius |
-| Keep the index current automatically | [Watch daemon](WATCH_DAEMON.md) | A resident development process that catches up changes and republishes the index |
+| Keep the index current automatically | [Automatic maintenance guide](AUTOMATIC_MAINTENANCE.md) | A setup with supervised indexing, automatic reader refresh, and clear hook ownership |
 | Upgrade from Woods 1.x | [Upgrade to Woods 2.0](UPGRADING_TO_2.md) | A backed-up, re-indexed, verified v2 installation |
 | Diagnose an error | [Troubleshooting](TROUBLESHOOTING.md) | Symptom-to-cause checks for extraction, MCP, embeddings, storage, and Docker |
 | Contribute to Woods | [Contributing](../CONTRIBUTING.md) | A tested change with synchronized docs and plugin guidance |
@@ -41,6 +41,7 @@ and stdio or Streamable HTTP endpoints directly.
 
 ## Index lifecycle
 
+- [Automatic maintenance guide](AUTOMATIC_MAINTENANCE.md): recommended development workflow and a section-by-section map of watcher, indexing, MCP, and hook documentation.
 - [Source freshness](SOURCE_FRESHNESS.md): verify dirty source against a served generation, establish a fresh-process baseline, and understand bounded unknown results.
 
 - [Retrieval guide](RETRIEVAL_GUIDE.md): configure embeddings and understand semantic retrieval, ranking, and token budgets.

data/docs/RETRIEVAL_GUIDE.md

@@ -291,16 +291,65 @@ result    = retriever.retrieve("what validations does Order have?")
 
 ---
 
-## Degradation Tiers
-
-Retrieval degrades gracefully when components are unavailable. The Retriever itself does not implement explicit fallback tiers, degradation happens naturally through how each component handles errors:
+## Semantic corpus diagnostics
+
+Extraction and embedding publish different data. `woods_status.ready` describes
+the structural index; neither that flag nor bootstrap `hydrated` proves that
+semantic retrieval has indexed records. Provider detection alone can succeed
+before the first embedding run.
+
+Supporting readers report `woods_status.retriever.corpus`:
+
+- `state`: `empty`, `metadata_only`, `vectors_only`, `nonempty`, or `unknown`.
+- `vectors` and `metadata`: each has `count`, `by_type`, and `untyped_count`.
+  Counts describe stored entries, including chunks; they are not distinct
+  extracted-unit counts or a completeness certificate. Missing type labels are
+  reported separately rather than assigned to a guessed type.
+- Unknown counts are `null`. Diagnostics use an explicit local-store capability;
+  they do not query remote stores to discover their counts. A missing capability
+  does not mean the backend is empty or broken.
+
+When both semantic stores are known empty, `codebase_retrieve` reports
+`empty_index` with recovery guidance instead of presenting an empty match as
+evidence that application code is absent. Run `woods:embed` in the application
+with the intended provider and storage configuration, then reload or restart
+the reader. Restart after changing provider or store configuration; reload
+refreshes the stores of the existing retriever. Alternatively, explicitly select `WOODS_RETRIEVAL_MODE=lexical`
+in the MCP process and restart for ranked retrieval over the published units.
+Woods does not change retrieval mode automatically.
+
+Metadata-only stores can still answer some keyword, direct, and graph queries;
+they are not blocked by this diagnostic. Source-empty units deliberately retain
+metadata without vectors, so the two counts need not match. Nonempty stores can
+still have missing types, stale vectors, or provider failures. Inspect the
+query result and embedding evidence before claiming coverage. The type-rank
+table's metadata count describes the retrieval metadata store, not all units
+in the structural index.
+
+Older readers may omit `retriever.corpus`; record the reader revision separately
+from the index writer version and verify the embedding artifacts directly.
+Explicit lexical mode does not use or report semantic corpus counts.
 
-- **Embedding provider unavailable**: `codebase_retrieve` returns a structured configuration error. Check `woods_status` for retrieval readiness.
-- **Vector store unavailable**: vector and hybrid strategies fail at query time. Keyword and graph strategies remain available for direct calls to `SearchExecutor`.
-- **Metadata store error**: the structural context overview (unit counts by type) is silently omitted; `Retriever#build_structural_context` rescues `StandardError` and returns `nil`. The retrieval result is still returned without the overview.
-- **Graph store unavailable**: graph expansion in hybrid strategy produces no graph candidates; vector and keyword candidates are still ranked and returned.
+## Degradation Tiers
 
-In all cases, errors in individual components produce empty candidate sets for that source rather than raising through the `Retriever`. Configure circuit breakers via `Woods::Resilience::CircuitBreaker` on external providers (Qdrant, OpenAI) for production deployments.
+The MCP boundary distinguishes missing configuration, empty stores, and failed
+stores. A failure is not evidence that no application code matches:
+
+- **No embedding provider configured:** `codebase_retrieve` reports a configuration
+  error with embedding and explicit lexical-mode options.
+- **Both semantic stores known empty:** the tool reports `empty_index`; see the
+  [corpus diagnostics](#semantic-corpus-diagnostics) above.
+- **Failed dump hydration:** the tool reports `degraded_index` rather than serving
+  a clean empty result. Repair or regenerate the named embedding artifact.
+- **Query-time storage failures:** vector, metadata, and graph adapter exceptions
+  are translated into store errors and reported as `degraded_index` by MCP.
+  This includes failures building the metadata overview; that failure is not
+  silently omitted. Direct Ruby callers should handle `Woods::Retriever::StoreError`.
+
+Provider failures and missing metadata for returned candidates have their own
+error paths. Preserve their diagnostics; do not silently switch modes or treat
+an exception as an empty match. Positive corpus counts do not override these
+checks.
 
 ---
 

data/docs/SOURCE_FRESHNESS.md

@@ -4,7 +4,7 @@
 application inputs with source bytes visible to the reader. It is separate from
 index age, HEAD equality, daemon liveness and external database/runtime state.
 
-This capability is unreleased after `2.0.0.beta2`. Check the installed gem's
+This capability is included in Woods `2.0.0`. Check the installed gem's
 `woods-extract --help`, `rake -T woods:source_status`, and `woods_status` schema
 before using it; upgrading the plugin alone does not upgrade Woods.
 

data/docs/TOKEN_BENCHMARK.md

@@ -13,7 +13,8 @@
 > When the optional [`tokenizers`](https://github.com/ankane/tokenizers-ruby)
 > gem is installed, the Ollama path uses the real BERT WordPiece tokenizer
 > (`Woods::Embedding::TokenCounter`) instead of this heuristic. The 4.0
-> divisor below is what the gem falls back to everywhere else.
+> divisor applies to the OpenAI/default path; Ollama falls back to 1.5 when
+> its tokenizer is unavailable.
 
 This is a historical record of the benchmark that picked 4.0 over the
 original 3.5 divisor. It is cited from five places in `lib/` as the evidence
@@ -35,17 +36,21 @@ for that choice, keep the numbers below intact if you edit this doc.
 | 3.8 | 16.2% | 42.5% |
 | **4.0 (shipped)** | **10.6%** | **35.4%** |
 
-Mean chars/token across the corpus was **4.41** (range 3.94–5.42). The
-heuristic always overestimated, never underestimated, across all 19 files,
-which is what makes it safe for token-limit enforcement even at its worst
-case. Code lines and comment/YARD lines had similar ratios (4.38 vs. 4.27
-chars/token), no separate handling needed for either.
+Mean chars/token across the corpus was **4.41** (range 3.94–5.42). These
+aggregate results favor the 4.0 divisor for this sample; they do not establish
+an upper bound on token counts. The recorded range includes values below 4.0,
+so the heuristic can underestimate. Code lines and comment/YARD lines had
+similar ratios (4.38 vs. 4.27 chars/token) in this sample.
+
+The character estimate covers only the text passed to the counter. It does not
+bound the serialized MCP response: text rendering, structured output, provenance,
+and JSON framing can add bytes and tokens beyond that input.
 
 ## What shipped
 
 **The divisor changed from 3.5 to 4.0.** It roughly halves the mean
-overestimate (26.2% → 10.6%) while keeping the conservative
-always-overestimates property, at zero new runtime dependencies. The
+error (26.2% → 10.6%) at zero new runtime dependencies. It remains an estimate,
+not a guarantee that arbitrary input fits a model's token limit. The
 constant lives in one place now (`Woods::TokenUtils::CHARS_PER_TOKEN_BY_PROVIDER`),
 not scattered across call sites, see `lib/woods/token_utils.rb` for the
 current definition and `docs/EMBEDDING_MODELS.md` for the Ollama-side ratio.
@@ -53,8 +58,9 @@ current definition and `docs/EMBEDDING_MODELS.md` for the Ollama-side ratio.
 **tiktoken_ruby was deliberately not added as a runtime dependency.** A 10.6%
 mean error is acceptable for chunking decisions, budget estimates, and
 truncation; a native-extension dependency for marginal accuracy gains wasn't
-worth it. The optional `tokenizers` gem covers the case where exact counts
-matter more (see above).
+worth it. The optional `tokenizers` gem provides counts for its supported BERT
+WordPiece tokenizer, not every model. Strict token-limit enforcement requires
+the tokenizer used by the target model.
 
 ## Reproducing this benchmark