woods 2.0.0.beta2 → 2.0.0.beta4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (233) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +339 -1
  3. data/CONTRIBUTING.md +188 -12
  4. data/README.md +93 -174
  5. data/SECURITY.md +9 -6
  6. data/docs/AGENT_GUIDE.md +109 -8
  7. data/docs/AGENT_SETUP.md +98 -7
  8. data/docs/BACKEND_MATRIX.md +25 -0
  9. data/docs/CLIENT_HOOKS.md +111 -0
  10. data/docs/CONFIGURATION_REFERENCE.md +267 -16
  11. data/docs/CONSOLE_MCP_SETUP.md +80 -7
  12. data/docs/DOCKER_SETUP.md +22 -3
  13. data/docs/EVALUATION.md +464 -1
  14. data/docs/EXTRACTOR_REFERENCE.md +45 -6
  15. data/docs/FAQ.md +11 -12
  16. data/docs/GETTING_STARTED.md +17 -5
  17. data/docs/INCREMENTAL_EXTRACTION.md +147 -7
  18. data/docs/INDEX_LAYOUT.md +382 -0
  19. data/docs/INTERNALS.md +7 -2
  20. data/docs/MCP_SERVERS.md +276 -5
  21. data/docs/MCP_TOOL_COOKBOOK.md +37 -22
  22. data/docs/MCP_WORKTREE_SETUP.md +43 -83
  23. data/docs/NOTION_INTEGRATION.md +13 -0
  24. data/docs/OBSIDIAN_INTEGRATION.md +57 -9
  25. data/docs/PUBLISHED_INDEX.md +72 -0
  26. data/docs/README.md +7 -0
  27. data/docs/RETRIEVAL_GUIDE.md +273 -12
  28. data/docs/RUNTIME_TRACING.md +71 -0
  29. data/docs/SOURCE_FRESHNESS.md +143 -0
  30. data/docs/TROUBLESHOOTING.md +129 -18
  31. data/docs/UNBLOCKED_INTEGRATION.md +25 -0
  32. data/docs/UPGRADING_TO_2.md +48 -22
  33. data/docs/WATCH_DAEMON.md +277 -67
  34. data/exe/woods-agent-config +6 -0
  35. data/exe/woods-extract +5 -0
  36. data/exe/woods-hook-context +6 -0
  37. data/exe/woods-mcp-start +14 -9
  38. data/lib/generators/woods/pgvector_generator.rb +8 -2
  39. data/lib/generators/woods/templates/woods.rb.tt +1 -3
  40. data/lib/tasks/woods.rake +47 -397
  41. data/lib/woods/agent_configuration/applier.rb +135 -0
  42. data/lib/woods/agent_configuration/cli.rb +101 -0
  43. data/lib/woods/agent_configuration/cli_options.rb +29 -0
  44. data/lib/woods/agent_configuration/document.rb +105 -0
  45. data/lib/woods/agent_configuration/error.rb +7 -0
  46. data/lib/woods/agent_configuration/launcher.rb +75 -0
  47. data/lib/woods/agent_configuration/layout.rb +72 -0
  48. data/lib/woods/agent_configuration/managed_section.rb +62 -0
  49. data/lib/woods/agent_configuration/plan.rb +98 -0
  50. data/lib/woods/agent_configuration/plan_diff.rb +38 -0
  51. data/lib/woods/agent_configuration/planned_files.rb +61 -0
  52. data/lib/woods/agent_configuration/planner.rb +63 -0
  53. data/lib/woods/agent_configuration/planner_validation.rb +77 -0
  54. data/lib/woods/agent_configuration/preflight.rb +100 -0
  55. data/lib/woods/agent_configuration/recovery.rb +49 -0
  56. data/lib/woods/ast/node.rb +2 -0
  57. data/lib/woods/ast/parser.rb +38 -5
  58. data/lib/woods/builder.rb +21 -5
  59. data/lib/woods/cache/cache_middleware.rb +28 -7
  60. data/lib/woods/cache/cache_store.rb +4 -5
  61. data/lib/woods/change_set.rb +5 -4
  62. data/lib/woods/console/credential_index.rb +20 -2
  63. data/lib/woods/console/credential_scanner.rb +18 -17
  64. data/lib/woods/console/credential_scanner_registry.rb +36 -0
  65. data/lib/woods/console/dispatch_pipeline.rb +7 -0
  66. data/lib/woods/console/embedded_executor.rb +32 -10
  67. data/lib/woods/console/encrypted_credential_snapshot.rb +16 -0
  68. data/lib/woods/console/rack_middleware.rb +22 -13
  69. data/lib/woods/console/server.rb +18 -16
  70. data/lib/woods/console/sql_noise_stripper.rb +9 -7
  71. data/lib/woods/console/sql_table_scanner.rb +47 -7
  72. data/lib/woods/console/sql_validator.rb +49 -9
  73. data/lib/woods/console/sqlite_read_guard.rb +46 -0
  74. data/lib/woods/coordination/pipeline_lock.rb +3 -2
  75. data/lib/woods/dependency_graph.rb +65 -13
  76. data/lib/woods/embedding/corpus.rb +94 -0
  77. data/lib/woods/embedding/indexer.rb +114 -60
  78. data/lib/woods/embedding/openai.rb +17 -6
  79. data/lib/woods/evaluation/ablation_executor.rb +6 -1
  80. data/lib/woods/evaluation/ablation_timed_executor.rb +22 -4
  81. data/lib/woods/export/typed_reader.rb +56 -0
  82. data/lib/woods/extractor.rb +277 -149
  83. data/lib/woods/extractors/action_cable_extractor.rb +3 -1
  84. data/lib/woods/extractors/behavioral_profile.rb +9 -7
  85. data/lib/woods/extractors/caching_extractor.rb +3 -1
  86. data/lib/woods/extractors/concern_extractor.rb +64 -6
  87. data/lib/woods/extractors/configuration_extractor.rb +7 -3
  88. data/lib/woods/extractors/controller_extractor.rb +13 -4
  89. data/lib/woods/extractors/database_view_extractor.rb +3 -1
  90. data/lib/woods/extractors/declared_parent.rb +55 -0
  91. data/lib/woods/extractors/decorator_extractor.rb +3 -1
  92. data/lib/woods/extractors/engine_extractor.rb +3 -1
  93. data/lib/woods/extractors/event_extractor.rb +4 -2
  94. data/lib/woods/extractors/factory_extractor.rb +3 -1
  95. data/lib/woods/extractors/graphql_extractor.rb +10 -13
  96. data/lib/woods/extractors/i18n_extractor.rb +3 -1
  97. data/lib/woods/extractors/job_extractor.rb +6 -19
  98. data/lib/woods/extractors/lib_extractor.rb +13 -9
  99. data/lib/woods/extractors/mailer_extractor.rb +26 -15
  100. data/lib/woods/extractors/manager_extractor.rb +3 -1
  101. data/lib/woods/extractors/method_parameters.rb +53 -0
  102. data/lib/woods/extractors/middleware_argument.rb +65 -0
  103. data/lib/woods/extractors/middleware_extractor.rb +9 -3
  104. data/lib/woods/extractors/migration_extractor.rb +3 -1
  105. data/lib/woods/extractors/model_extractor.rb +26 -34
  106. data/lib/woods/extractors/package_extractor.rb +24 -4
  107. data/lib/woods/extractors/phlex_extractor.rb +3 -1
  108. data/lib/woods/extractors/policy_extractor.rb +3 -1
  109. data/lib/woods/extractors/poro_extractor.rb +13 -9
  110. data/lib/woods/extractors/pundit_extractor.rb +3 -1
  111. data/lib/woods/extractors/rails_source_extractor.rb +4 -2
  112. data/lib/woods/extractors/rake_task_extractor.rb +4 -2
  113. data/lib/woods/extractors/route_extractor.rb +3 -1
  114. data/lib/woods/extractors/route_helper_resolver.rb +10 -33
  115. data/lib/woods/extractors/scheduled_job_extractor.rb +41 -15
  116. data/lib/woods/extractors/serializer_extractor.rb +4 -2
  117. data/lib/woods/extractors/service_extractor.rb +3 -1
  118. data/lib/woods/extractors/shared_dependency_scanner.rb +2 -2
  119. data/lib/woods/extractors/shared_utility_methods.rb +48 -19
  120. data/lib/woods/extractors/source_nesting.rb +1 -1
  121. data/lib/woods/extractors/state_machine_extractor.rb +3 -1
  122. data/lib/woods/extractors/test_mapping_extractor.rb +3 -1
  123. data/lib/woods/extractors/validator_extractor.rb +3 -1
  124. data/lib/woods/extractors/view_component_extractor.rb +3 -1
  125. data/lib/woods/extractors/view_template_extractor.rb +3 -1
  126. data/lib/woods/gem_mapper.rb +2 -0
  127. data/lib/woods/git_history.rb +116 -0
  128. data/lib/woods/graph_analyzer.rb +35 -6
  129. data/lib/woods/hooks/context_cli.rb +54 -0
  130. data/lib/woods/hooks/context_event.rb +88 -0
  131. data/lib/woods/hooks/context_hint.rb +73 -0
  132. data/lib/woods/hooks/context_impact.rb +77 -0
  133. data/lib/woods/hooks/context_output.rb +47 -0
  134. data/lib/woods/hooks/context_state.rb +102 -0
  135. data/lib/woods/hooks/refresh.rb +79 -0
  136. data/lib/woods/hooks/rule_projection.rb +78 -0
  137. data/lib/woods/input_rules.rb +19 -0
  138. data/lib/woods/mcp/bearer_auth.rb +22 -13
  139. data/lib/woods/mcp/bootstrapper.rb +79 -4
  140. data/lib/woods/mcp/config_resolver.rb +2 -1
  141. data/lib/woods/mcp/index_reader.rb +334 -162
  142. data/lib/woods/mcp/initialization_guidance.rb +27 -0
  143. data/lib/woods/mcp/origin_guard.rb +17 -9
  144. data/lib/woods/mcp/published_lexical_retriever.rb +115 -0
  145. data/lib/woods/mcp/renderers/markdown_renderer.rb +22 -9
  146. data/lib/woods/mcp/renderers/plain_renderer.rb +18 -8
  147. data/lib/woods/mcp/search_results.rb +74 -0
  148. data/lib/woods/mcp/server.rb +178 -63
  149. data/lib/woods/mcp/tool_contract.rb +3 -1
  150. data/lib/woods/mcp/tool_response_renderer.rb +41 -0
  151. data/lib/woods/mcp/traversal_evidence.rb +113 -0
  152. data/lib/woods/mcp/traversal_evidence_index.rb +100 -0
  153. data/lib/woods/mcp/traversal_evidence_page.rb +41 -0
  154. data/lib/woods/mcp/traversal_evidence_text.rb +52 -0
  155. data/lib/woods/mcp/traversal_response.rb +22 -0
  156. data/lib/woods/notion/exporter.rb +56 -17
  157. data/lib/woods/obsidian/destination_plan.rb +98 -0
  158. data/lib/woods/obsidian/name_mapper.rb +19 -3
  159. data/lib/woods/obsidian/note_builder.rb +19 -10
  160. data/lib/woods/obsidian/vault_exporter.rb +88 -32
  161. data/lib/woods/operator/pipeline_guard.rb +18 -13
  162. data/lib/woods/path_dispatcher.rb +13 -6
  163. data/lib/woods/payload_store.rb +27 -26
  164. data/lib/woods/published_index/typed_unit_reader.rb +40 -3
  165. data/lib/woods/published_index.rb +2 -2
  166. data/lib/woods/railtie.rb +3 -3
  167. data/lib/woods/railtie_support.rb +12 -12
  168. data/lib/woods/rake_helpers.rb +382 -0
  169. data/lib/woods/resilience/graph_invariant_validator/membership_checks.rb +71 -0
  170. data/lib/woods/resilience/graph_invariant_validator/node_checks.rb +61 -0
  171. data/lib/woods/resilience/graph_invariant_validator/reverse_relationship_checks.rb +46 -0
  172. data/lib/woods/resilience/graph_invariant_validator.rb +119 -0
  173. data/lib/woods/resilience/index_validator/graph_checks.rb +80 -0
  174. data/lib/woods/resilience/index_validator.rb +112 -23
  175. data/lib/woods/retrieval/context_assembler.rb +50 -15
  176. data/lib/woods/retrieval/lexical_assembler.rb +84 -0
  177. data/lib/woods/retrieval/lexical_index.rb +120 -0
  178. data/lib/woods/retrieval/ranker.rb +4 -2
  179. data/lib/woods/retrieval/scope.rb +108 -0
  180. data/lib/woods/retrieval/scoped_graph_store.rb +32 -0
  181. data/lib/woods/retrieval/scoped_vector_store.rb +55 -0
  182. data/lib/woods/retrieval/search_executor.rb +86 -27
  183. data/lib/woods/retrieval/source_evidence.rb +200 -0
  184. data/lib/woods/retriever.rb +98 -22
  185. data/lib/woods/ruby_analyzer/trace_enricher.rb +77 -38
  186. data/lib/woods/session_tracer/file_store.rb +6 -1
  187. data/lib/woods/session_tracer/middleware.rb +10 -12
  188. data/lib/woods/session_tracer/redis_store.rb +22 -6
  189. data/lib/woods/session_tracer/session_flow_assembler.rb +23 -17
  190. data/lib/woods/session_tracer/solid_cache_coordination.rb +6 -4
  191. data/lib/woods/session_tracer/unit_resolver.rb +63 -0
  192. data/lib/woods/source_inputs/consumer_errors.rb +31 -0
  193. data/lib/woods/source_inputs/handoff.rb +102 -0
  194. data/lib/woods/source_inputs/launcher.rb +157 -0
  195. data/lib/woods/source_inputs/manifest.rb +124 -0
  196. data/lib/woods/source_inputs/private_key.rb +55 -0
  197. data/lib/woods/source_inputs/scanner.rb +171 -0
  198. data/lib/woods/source_inputs/scopes.rb +71 -0
  199. data/lib/woods/source_inputs/session.rb +214 -0
  200. data/lib/woods/source_inputs/status.rb +84 -0
  201. data/lib/woods/source_inputs/verifier.rb +107 -0
  202. data/lib/woods/storage/metadata_store.rb +25 -25
  203. data/lib/woods/storage/pgvector.rb +35 -10
  204. data/lib/woods/storage/qdrant.rb +17 -7
  205. data/lib/woods/storage/vector_store.rb +18 -6
  206. data/lib/woods/tasks.rb +3 -2
  207. data/lib/woods/temporal/json_snapshot_store.rb +58 -9
  208. data/lib/woods/unblocked/exporter.rb +59 -70
  209. data/lib/woods/version.rb +1 -1
  210. data/lib/woods/watch/boot_snapshot.rb +52 -0
  211. data/lib/woods/watch/daemon.rb +154 -32
  212. data/lib/woods/watch/listen_watcher.rb +4 -0
  213. data/lib/woods/watch/polling_watcher.rb +5 -1
  214. data/lib/woods/watch/status.rb +20 -15
  215. data/lib/woods/watch/tree_scan.rb +21 -13
  216. data/lib/woods/watch/watcher.rb +4 -1
  217. data/lib/woods.rb +50 -11
  218. data/plugin/.claude-plugin/plugin.json +1 -1
  219. data/plugin/hooks/adapters/normalize.jq +15 -0
  220. data/plugin/hooks/adapters/normalize.rb +63 -0
  221. data/plugin/hooks/hooks.json +20 -0
  222. data/plugin/hooks/woods-context.sh +50 -0
  223. data/plugin/hooks/woods-input-rules.sh +159 -0
  224. data/plugin/hooks/woods-opencode.mjs +65 -0
  225. data/plugin/hooks/woods-post-edit.sh +2 -225
  226. data/plugin/hooks/woods-refresh.sh +260 -0
  227. data/plugin/hooks/woods-session-start.sh +47 -55
  228. data/plugin/skills/woods-agent-enable/SKILL.md +19 -0
  229. data/plugin/skills/woods-diagnose/SKILL.md +319 -1
  230. data/plugin/skills/woods-investigate/SKILL.md +145 -0
  231. data/plugin/skills/woods-mcp-config/SKILL.md +90 -2
  232. data/plugin/skills/woods-setup/SKILL.md +110 -6
  233. metadata +87 -5
data/docs/MCP_SERVERS.md CHANGED
@@ -27,6 +27,15 @@ bin/rails woods:validate
27
27
  bin/rails woods:stats
28
28
  ```
29
29
 
30
+ For embedding-free ranked retrieval, start Index MCP with
31
+ `WOODS_RETRIEVAL_MODE=lexical`. This opt-in reads the published extraction units;
32
+ it does not probe providers or load vectors. `woods_status.retriever.mode` reports
33
+ `lexical`, and inactive embedding fields are `null`. The default semantic mode
34
+ keeps its existing embedding setup and failure behavior. See
35
+ [retrieval modes](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval) for scoring,
36
+ query limits and measured tradeoffs. Both packaged stdio and HTTP launches honor
37
+ the setting; put it in the MCP process's environment, not just a Rails initializer.
38
+
30
39
  The stdio server can then run outside Rails. Point it at the index root (`tmp/woods/` by default), not at an internal generation or payload directory.
31
40
 
32
41
  ### Configure a stdio client
@@ -47,6 +56,10 @@ Prefer the application's bundle and a project-scoped configuration:
47
56
 
48
57
  `woods-mcp-start` checks that the directory and published manifest exist, then replaces itself with `woods-mcp`. It does not install dependencies or restart a crashed process.
49
58
 
59
+ `woods_status.index.woods_version` identifies the last publisher of the served
60
+ manifest; `server.version` identifies the running MCP reader. Missing writer
61
+ provenance is `null`. See [manifest writer provenance](PUBLISHED_INDEX.md#manifest-writer-provenance).
62
+
50
63
  You can launch the server directly when the client already handles preflight:
51
64
 
52
65
  ```bash
@@ -57,6 +70,12 @@ Keep stdout reserved for MCP protocol messages. Diagnose startup failures from s
57
70
 
58
71
  ### Client configuration locations
59
72
 
73
+ Supporting development builds offer preview/apply/update/remove ownership for
74
+ Claude Code project or explicit user configuration. See
75
+ [managed configuration](AGENT_SETUP.md#managed-claude-code-configuration) for
76
+ `woods-agent-config`, host/Compose preflight, conflict handling, and recovery.
77
+ Manual configuration remains available for older gems and other clients.
78
+
60
79
  MCP clients expose project or user-level server settings in different locations. Use project scope when available, preserve the `command`, `args`, and absolute `cwd` semantics above, and translate only the surrounding client-specific format. Woods is model-independent: compatibility depends on the client supporting MCP stdio or Streamable HTTP, not on whether the connected model is from OpenAI, Anthropic, Google, xAI, or another provider.
61
80
 
62
81
  Client configuration formats can change independently of Woods. If a client rejects otherwise valid JSON, check that client's current MCP documentation.
@@ -99,6 +118,27 @@ Reconnect the client, then call:
99
118
 
100
119
  Prefer a real MCP client's connection flow over a hand-written JSON-RPC pipe. Modern MCP 2026-07-28 requests carry per-request protocol metadata and can use `server/discover` without an initialization handshake; older clients still use `initialize`. A valid raw smoke test must implement one complete flow rather than sending an isolated `tools/list` or `tools/call` request.
101
120
 
121
+ ### Initialization guidance
122
+
123
+ The Index Server supplies a short, client-neutral `instructions` field through
124
+ the SDK's `initialize` response and modern `server/discover`. It describes the
125
+ status → discovery → inspection → bounded traversal → source-verification
126
+ workflow, and lists only the tools actually registered by this server.
127
+ Instructions are stable for unchanged tool registration and bounded to 2,048
128
+ UTF-8 bytes across supported configurations. Building the text does not probe
129
+ providers, extract code, or write configuration.
130
+
131
+ Registration alone does not establish retrieval readiness: check `woods_status`
132
+ before using `codebase_retrieve`, including after reload. The guidance grants
133
+ no extraction, configuration-change, or Console authorization. Detailed usage
134
+ belongs in the [agent guide](AGENT_GUIDE.md).
135
+
136
+ This addition is unreleased after `2.0.0.beta2`. The SDK omits `instructions`
137
+ when negotiating protocol `2024-11-05`; that behavior is preserved. Older gems,
138
+ legacy clients, and clients that do not show server instructions can use the
139
+ agent guide or investigation skill. Leave protocol negotiation enabled rather
140
+ than pinning a newer version solely to obtain guidance.
141
+
102
142
  ### Tools (29 — 14 registered in the packaged default)
103
143
 
104
144
  The Index Server defines 29 schemas across core and conditional capabilities. The normal packaged executable registers the 14 tools below; the remaining schemas require the specialized wiring described afterward.
@@ -108,8 +148,8 @@ The Index Server defines 29 schemas across core and conditional capabilities. Th
108
148
  | `woods_status` | Index health, generation, counts, and retrieval readiness |
109
149
  | `search` | Discover identifiers by regex, prefix, suffix, source, or metadata |
110
150
  | `lookup` | Fetch one exact unit with source, metadata, and relationships |
111
- | `dependencies` | Traverse what a unit depends on (`depth`, `types`, `via` narrow; `limit`, `offset` page) |
112
- | `dependents` | Traverse what depends on a unit (`depth`, `types`, `via` narrow; `limit`, `offset` page) |
151
+ | `dependencies` | Traverse what a unit depends on (`depth`, `types`, `via` narrow; `max_nodes`, `max_edges` budget work; `limit`, `offset` page) |
152
+ | `dependents` | Traverse what depends on a unit (`depth`, `types`, `via` narrow; `max_nodes`, `max_edges` budget work; `limit`, `offset` page) |
113
153
  | `structure` | Summarize structural relationships around a unit |
114
154
  | `trace_flow` | Follow a request, job, mail, or other execution flow |
115
155
  | `framework` | Inspect relevant Rails or installed gem source |
@@ -118,7 +158,29 @@ The Index Server defines 29 schemas across core and conditional capabilities. Th
118
158
  | `domain_clusters` | Discover connected domains in the graph |
119
159
  | `pagerank` | Find structurally central units |
120
160
  | `reload` | Reload a newly published generation without restarting the client |
121
- | `codebase_retrieve` | Natural-language retrieval; returns a configuration error until embeddings exist |
161
+ | `codebase_retrieve` | Natural-language retrieval with embeddings or explicit lexical mode over extraction output |
162
+
163
+ When an identifier appears in multiple extraction types, `framework` reads its
164
+ framework-source bucket and `recent_changes` reads each selected type bucket.
165
+ Their paths and metadata belong to that selected bucket. When session tracing
166
+ is configured, newly recorded requests use the dispatched controller's runtime
167
+ class name; the fallback for requests without an instance respects Rails acronym
168
+ inflections. Existing trace records are unchanged. Controller lookup and root
169
+ outgoing-edge selection preserve the controller type. Downstream references and the shared context pool still use
170
+ bare identifiers. If a dependency encountered within the requested depth has
171
+ multiple published types, `session_trace` returns an `ambiguous_identity` tool
172
+ error naming the identifier and candidate types, with no partial context. This
173
+ also prevents an earlier dependency from occupying a later controller’s context
174
+ key. Unrelated collisions do not block a trace, and a known controller root keeps
175
+ its controller identity. A controller absent from the index remains in the
176
+ timeline without a source reference, so another type cannot fill that reference.
177
+ Candidate discovery and source reads use one pinned
178
+ generation. Corrupt or missing listed artifacts retain the `internal_error`
179
+ failure boundary; they do not prove uniqueness or become `ambiguous_identity` errors.
180
+ Use `depth: 0` for the request timeline, or inspect candidates with typed `lookup`
181
+ calls. Re-extraction does not remove a legitimate cross-type collision. Successful
182
+ traces retain their existing identifiers and response shape; target identity has
183
+ not been migrated globally. These corrections are available in `2.0.0.beta3`.
122
184
 
123
185
  The server also exposes MCP resources and resource templates for indexed units. Tool descriptions returned by MCP are the parameter-level source of truth; [Agent guide](AGENT_GUIDE.md) explains selection strategy.
124
186
 
@@ -130,6 +192,181 @@ error and continues serving the previous aligned generation; it never swaps in a
130
192
  partial or empty replacement. Grant write access for live reloads, or restart the MCP
131
193
  process after publishing a new embedded index.
132
194
 
195
+ ### Graph-analysis pages
196
+
197
+ Unreleased after `2.0.0.beta3`: `graph_analysis` enforces its advertised default
198
+ of 20 rows per section. Pass `limit` and `offset` to page one selected `analysis`
199
+ or each section of `analysis: "all"`. Explicit limits also bound nested hub
200
+ `dependents` lists. Older servers may return every section row when `limit` is
201
+ omitted; pass a limit explicitly when supporting both versions.
202
+
203
+ JSON responses retain `<section>_total`, `<section>_offset` (when positive), and
204
+ `<section>_truncated: true` whenever a page omits rows before or after it. Markdown,
205
+ plain, and Claude responses show the same total and offset on last and empty
206
+ pages. For example, offset 20 with limit 5 over 25 published orphans shows
207
+ `5 of 25 from offset 20`; offset 100 shows `0 of 25 from offset 100`. An empty
208
+ page does not mean the section has no findings. These totals describe the
209
+ published report arrays, which can themselves be bounded during extraction;
210
+ they do not establish complete source-reference coverage.
211
+
212
+ ### Search completeness
213
+
214
+ Search responses retain `query`, `result_count`, and `results`; `result_count`
215
+ is the number returned, not an estimated total. The additive `completeness`
216
+ object describes the requested types, literal filters, and fields in the pinned
217
+ generation. This contract is unreleased after `2.0.0.beta2`.
218
+
219
+ | `reason` | `status` | `has_more` | `total_matches` |
220
+ |---|---|---|---|
221
+ | `exhausted` | `complete` | `false` | Exact count |
222
+ | `result_limit` | `partial` | `true` | `null` (unknown) |
223
+ | `scan_budget` or `regex_timeout` | `partial` | `null` (unknown) | `null` (unknown) |
224
+
225
+ `matched_lower_bound` counts distinct observed `(type, identifier)` matches,
226
+ including at most one lookahead match beyond `limit`. A result-limit response
227
+ therefore establishes another match; an exactly full page can instead be
228
+ complete if the requested domain is exhausted. Deep lookahead shares
229
+ `WOODS_SEARCH_MAX_SCAN` with the initial scan and retains round-robin scanning
230
+ across types. Search does not count the entire omitted tail or offer pagination.
231
+ The existing `types` filter and result labels name directory families:
232
+ `rails_source` includes both Rails and gem source units. Deep reads accept those
233
+ two stored types only in that shared directory; `lookup` and lexical retrieval
234
+ retain the unit's actual `rails_source` or `gem_source` type.
235
+
236
+ All partial responses retain `partial: true` and include a narrowing `hint`.
237
+ JSON exposes these fields; Markdown, plain text, and Claude formats label the
238
+ returned count, stopping reason, known/unknown remainder, and total explicitly.
239
+ Narrow `types`, literal `exact_prefix`/`exact_suffix`, or deep `fields` before
240
+ using discovery as exhaustive evidence. Completeness applies to this index and
241
+ query domain, not to unindexed application code.
242
+
243
+ Detected missing, unreadable, or corrupt artifacts remain `isError: true` with
244
+ `_meta.error_code: "corrupt_artifact"`. Their `_meta.completeness` has
245
+ `status: "unknown"`, `reason: "unreadable_or_corrupt_source"`, and `null` for
246
+ `has_more`, `total_matches`, and `matched_lower_bound`; no successful empty
247
+ result is substituted. Inspect `woods_status` and run `woods:validate`.
248
+
249
+ ### Dependency graph coverage
250
+
251
+ `dependencies` and `dependents` return relationships recorded in the published
252
+ index, not an exhaustive call graph or source-reference index. Extraction combines
253
+ runtime reflection with selective source scanning; arbitrary method-body constant
254
+ references (including references to generic PORO and library classes) may have no
255
+ edge. No dependents, a test-only dependent, or a completed traversal does not prove
256
+ there are no production callers. Verify important absence claims in source.
257
+
258
+ Supporting servers expose the annotated, paginated traversal result in
259
+ `structuredContent.data` for every renderer, including the default packaged
260
+ stdio and HTTP servers. Read `data.total_is_exact`, `data.graph_coverage`, budget
261
+ counters and optional explanation witnesses there; `content[0].text` and
262
+ `structuredContent.text` keep the same human-readable rendering. No `format`
263
+ tool argument is needed or accepted. This additive data payload is unreleased
264
+ after `2.0.0.beta3`; verify the installed response before relying on it. Older
265
+ human-renderer responses can carry only text. The structured nodes and witnesses
266
+ cover the same page, not an additional traversal or an unpaginated graph.
267
+
268
+ Successful responses carry `graph_coverage` with `scope: "published_relationships"`,
269
+ `source_references: "not_exhaustive"`, and a human-readable `notice`. Text formats
270
+ show the same notice, including compact, root-only and empty-page responses.
271
+ This response metadata and the total exactness field below are unreleased after
272
+ Woods `2.0.0.beta3`; older servers need the same conservative interpretation.
273
+
274
+ ### Dependency traversal budgets
275
+
276
+ `dependencies` and `dependents` walk breadth-first in stored graph order. The
277
+ walk defaults to `max_nodes: 1000` (including the root) and `max_edges: 10000`;
278
+ callers can select 1–10,000 nodes and 1–100,000 edge checks. The node budget
279
+ counts distinct nodes admitted after filters. Every candidate edge is charged
280
+ before filtering, including duplicates, cycles, and the forward-edge checks
281
+ needed to match a reverse `via` filter. Thus restrictive filters cannot bypass
282
+ the edge budget. Nodes at the requested `depth` are recorded without reading
283
+ their adjacency lists.
284
+
285
+ When further expansion would exceed a budget, JSON reports `partial: true`,
286
+ `partial_reason: "node_budget"` or `"edge_budget"`, and `traversal_budget` with
287
+ `max_nodes`, `max_edges`, `visited_nodes`, and `visited_edges`. Text renderers
288
+ also identify the partial traversal. Already discovered nodes remain in the
289
+ answer, but an empty `deps` array in a partial answer does not prove a leaf.
290
+ Exact-budget walks that finish all requested work are complete and have no
291
+ `partial` marker.
292
+
293
+ `limit` (default 50) and `offset` only page that discovered result; they never
294
+ change the walk budget or depth. On a partial traversal, `nodes_total`, when
295
+ present for pagination, counts the discovered prefix, **not the full reachable
296
+ graph**. Every successful response includes `total_is_exact`: false for a
297
+ budget cutoff, true when the requested walk finishes, even when its page is
298
+ truncated or empty. It is independent of `limit`/`offset` and is present for
299
+ unpaged answers too. Partial text answers say `Showing N of at least M (total
300
+ unknown: node_budget)` (or `edge_budget`), including when no pagination is needed.
301
+ `M` includes the root and counts the admitted prefix; it is a lower bound for the
302
+ requested root, depth, type/relationship filters and published generation, not a
303
+ count of all application callers. Exactness describes that same recorded-graph
304
+ scope and never implies exhaustive source coverage. Paging beyond that prefix
305
+ stays partial. To explore more, narrow
306
+ `depth`/`types`/`via`, choose another root, or increase the traversal budget within
307
+ its maximum. Keep the root, filters, budgets, and published generation unchanged
308
+ for stable pages. No wall-clock deadline is used, so cutoffs are deterministic.
309
+
310
+ Budgets cover traversal work after per-generation graph loading and cache
311
+ preparation (JSON parsing, typed-edge normalization, node types and database
312
+ metadata). They do not cap that initial load, elapsed time, or total process
313
+ memory. These arguments are unreleased in Woods 2.0.0.beta2; check the connected
314
+ server's tool schema before sending them to an older installation.
315
+
316
+ ### Traversal explanations
317
+
318
+ Supporting development versions accept `explain: true` on `dependencies` and
319
+ `dependents`. Check the connected schema first; this option is unreleased after
320
+ 2.0.0.beta2. Omitted or false keeps the existing compact response.
321
+
322
+ The additive `explanation` object contains:
323
+
324
+ - `direction`: `forward` or `reverse`, plus the requested `root` identity.
325
+ - `edges`: records keyed by response-local IDs such as `e0`. Every record keeps
326
+ the original **source → target** direction, even during reverse traversal.
327
+ `source` contains its recorded `identifier` and `type`; `target` contains its
328
+ identifier and the unique type when the published graph establishes one.
329
+ `via`, `through`, `through_db`, and `disable_joins` preserve recorded values;
330
+ absent legacy attributes are null (shown as unknown in text), including an
331
+ unrecorded `disable_joins` rather than an invented false value.
332
+ - `witnesses`: one shortest breadth-first predecessor per admitted identifier,
333
+ keyed by identifier. Each has `parent`, `edge_id`, `impact` (`root`, `direct`,
334
+ or `transitive`), and `typed_path_complete`. Follow parent references to the
335
+ root to reconstruct one witness; alternative paths are not enumerated.
336
+
337
+ A target name shared by several types has `type: null`,
338
+ `resolution: "ambiguous"`, and sorted `candidate_types`. An unresolved target has
339
+ `resolution: "unresolved"` and an empty candidate list. Forward artifacts do not
340
+ record target types, so the response cannot choose among candidates. A witness
341
+ through an ambiguous or unresolved identity sets `typed_path_complete: false`;
342
+ it describes identifier-level reachability, never a uniquely typed path.
343
+ A true value means only that identities along this witness have unambiguous
344
+ types. It does not establish source-reference coverage or observed execution.
345
+ Text labels this `witness types unambiguous=yes/no`; the JSON key and its meaning
346
+ remain unchanged. The text label change is unreleased after `2.0.0.beta3`.
347
+ `types` filters retain the compact traversal's identifier-level semantics: any
348
+ registered type can qualify a name, while edge evidence keeps its actual source
349
+ owner. Multiple relationship kinds between the same endpoints remain separate.
350
+
351
+ Direct witnesses establish a recorded root relationship; transitive witnesses
352
+ represent inferred downstream reachability through recorded relationships.
353
+ Neither establishes observed execution, confidence, call order, or test coverage.
354
+
355
+ Node pagination retains required ancestor witnesses once, marked `context: true`
356
+ when outside the page; returned rows have `context: false`. Context records do
357
+ not increase the result-row count. Page evidence retains the witness edges and
358
+ other observed relationships among its visible/context endpoints; an empty page
359
+ has empty edge/witness maps. Edge IDs are local to this traversal response.
360
+
361
+ All examined evidence shares the existing edge budget, before `via`/`types`
362
+ filtering. Current `reverse_via` buckets allow direct reverse evidence lookup;
363
+ legacy recovery charges each reverse candidate and every inspected forward edge.
364
+ The shared predecessor forest and emitted records remain bounded by admitted
365
+ nodes and inspected edges. Per-generation JSON loading and the cached
366
+ O(nodes + variants) ownership/type preparation are outside the walk budget;
367
+ explanation mode never flattens all forward edges as per-request preparation.
368
+ Partial traversal and pagination metadata retain the budget contract above.
369
+
133
370
  ### Conditional Index capabilities
134
371
 
135
372
  The Ruby server builder contains 15 additional schemas for sessions, pipeline operations, retrieval feedback, temporal snapshots, and Notion sync. They register only when their required collaborators or configuration are wired.
@@ -140,6 +377,7 @@ The normal packaged executable does not wire pipeline-operator or feedback-store
140
377
 
141
378
  Use HTTP only for a deliberate shared or remote deployment. It expands the network boundary and requires authentication, origin restrictions, and TLS termination. Follow [MCP HTTP transport](MCP_HTTP_TRANSPORT.md); do not translate the stdio example into an unauthenticated public listener.
142
379
 
380
+
143
381
  ## Console Server
144
382
 
145
383
  The Console Server launches a Rails process through direct, Docker, or SSH connection configuration. It reads live data and must be treated as a separate security decision.
@@ -151,11 +389,14 @@ Console MCP is disabled by default because it reads live application data. Enabl
151
389
  ```ruby
152
390
  Woods.configure do |config|
153
391
  config.console_mcp_enabled = true
154
- config.console_mcp_token = ENV["WOODS_CONSOLE_MCP_TOKEN"]
392
+ config.console_mcp_http_enabled = false # stdio-only
155
393
  end
156
394
  ```
157
395
 
158
- The bearer token authenticates HTTP clients and is not sent over stdio. Rails still validates Console configuration while booting: production requires a token of at least 32 characters whenever Console is enabled, including for a stdio-only client. Keep it in the application's secret store, not in the initializer.
396
+ This explicitly disables HTTP Console while retaining stdio access; no HTTP
397
+ token is needed at boot. Existing configurations default to HTTP enabled.
398
+ For HTTP deployment, enable the HTTP flag and configure its token, origins
399
+ and TLS using the [Console setup guide](CONSOLE_MCP_SETUP.md#option-c-http-rack-middleware).
159
400
 
160
401
  Without a console connection file, the executable then launches the Rails task directly from its `cwd`:
161
402
 
@@ -229,3 +470,33 @@ Report vulnerabilities privately through [SECURITY.md](../SECURITY.md).
229
470
  5. Check [Troubleshooting](TROUBLESHOOTING.md) for the exact stderr message.
230
471
 
231
472
  For agent query behavior after connection, continue to [Agent guide](AGENT_GUIDE.md).
473
+
474
+ ### Explicit retrieval and discovery scope
475
+
476
+ On a server whose tool schema advertises them, `packages` and `source_paths` narrow
477
+ `search` and `codebase_retrieve` before candidate limits. These are per-call
478
+ arguments, not configuration settings. Inspect applied scope and completeness;
479
+ a narrow graph query can omit relevant cross-boundary dependencies. See the
480
+ [scope contract](RETRIEVAL_GUIDE.md#explicit-package-and-source-path-scopes) for
481
+ root/nested ownership, path normalization, errors, storage support, and cost.
482
+
483
+ ### Source freshness in status
484
+
485
+ `woods_status` accepts optional `source_check: "quick"` (default, 250ms scan) or
486
+ `"deep"` (five seconds). `index.source_freshness` describes the served generation
487
+ as `current`, `drifted` or `unknown`; missing source/key and incomplete capture
488
+ never count as current. No Rails initialization or provider call is needed.
489
+ See [source freshness](SOURCE_FRESHNESS.md) for scope, private-key handling and
490
+ fresh-process extraction. Existing HEAD/dirty fields remain separate diagnostics.
491
+
492
+ ### Explicit source evidence modes
493
+
494
+ When advertised by the installed schema, `lookup` and `codebase_retrieve` accept
495
+ `evidence: 'compact'` or `'outline'`; omitted/`'full'` preserves existing behavior.
496
+ Retrieval uses its original query. Compact lookup accepts optional `query` and an
497
+ estimated `budget` (default 2000); full lookup remains complete. `lookup` also
498
+ accepts an actual `type` and a `source_sha256` guard for typed, byte-verified
499
+ follow-up from an excerpt. Compact modes cannot be combined with metadata-only
500
+ lookup controls. Structured provenance stays within the existing closed output
501
+ schema's `data` field. Read the [evidence contract](RETRIEVAL_GUIDE.md#compact-published-evidence-and-api-outlines)
502
+ before interpreting published line ranges as physical source locations.
@@ -64,7 +64,7 @@ The Index Server defines **29 schemas**: the packaged executable registers **14*
64
64
  | Snapshot (4) | 4 | Extraction with `enable_snapshots = true` normally creates `woods.sqlite3`, which packaged servers discover. If extraction used the JSON fallback, set `WOODS_SNAPSHOTS=true` on the standalone server. Custom embedded servers pass `snapshot_store:`. Internal SQLite migrations are automatic. Tools: `list_snapshots`, `snapshot_diff`, `unit_history`, `snapshot_detail` |
65
65
  | `notion_sync` | 1 | `notion_api_token` + `notion_database_ids` both set |
66
66
 
67
- `codebase_retrieve` is always registered (no `retrieve` alias exists), but only returns results once an embedding provider is configured and `rake woods:embed` has run.
67
+ `codebase_retrieve` is always registered (no `retrieve` alias exists). Default semantic mode requires an embedding provider and a completed `woods:embed` run. Explicit `WOODS_RETRIEVAL_MODE=lexical` ranks published extraction units without a provider or embeddings; set it in the MCP process environment and restart the server. See [embedding-free lexical retrieval](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval).
68
68
 
69
69
  If an agent reports a missing tool, compare its request with the connected server's registered list and [MCP server boundaries](MCP_SERVERS.md#conditional-index-capabilities). The normal packaged executable does not wire operator or feedback collaborators. **Console Server tools are not all unconditionally registered**: 31 tool schemas exist as an inventory, but only the 9 Tier 1 tools are executable by default, or 11 with `console_embedded_read_tools: true` (adds `console_sql`/`console_query`). Tier 2, Tier 3, and `console_eval` are schema-only in every supported mode; there is no bridge or confirmation flow that unlocks them. See [MCP servers](MCP_SERVERS.md#console-server) for the supported inventory.
70
70
 
@@ -255,13 +255,24 @@ The `metadata.inlined_concerns` array lists which concerns were resolved:
255
255
  }
256
256
  ```
257
257
 
258
- **What you'll get:** A BFS tree of everything that references `User`, controllers, services, jobs, mailers, up to 2 hops out. Set `depth: 1` for direct dependents only.
258
+ **What you'll get:** A BFS traversal of units that reference `User`, such as controllers, services, jobs, and mailers, up to 2 hops out. Set `depth: 1` for direct dependents only.
259
259
 
260
- The answer is bounded to 50 nodes. When it is cut, the response ends with a
260
+ The answer is paged to 50 nodes by default. When it is cut, the response ends with a
261
261
  `Showing N of M (truncated)` line, the same one `graph_analysis` prints. Reach
262
262
  for `depth`, `types` and `via` first: they make the answer smaller. `limit` and
263
263
  `offset` only page what those leave, so a hub read one page at a time still
264
- costs every page.
264
+ repeats the walk. A separate traversal budget can return `partial: true`;
265
+ that marker means the reachable graph is incomplete even after the final page.
266
+ See [traversal budgets](MCP_SERVERS.md#dependency-traversal-budgets) before
267
+ changing `max_nodes` or `max_edges`.
268
+
269
+ To explain why a row is affected, check the connected schema and add
270
+ `"explain": true`. The response preserves source-to-target labels even while
271
+ walking dependents. Its predecessor witnesses distinguish direct relationships
272
+ from transitive inferred reachability, retain ancestor context across pages, and
273
+ mark ambiguous types explicitly. See the
274
+ [explanation contract](MCP_SERVERS.md#traversal-explanations); these witnesses are
275
+ not proof of observed execution.
265
276
 
266
277
  To find only which jobs depend on `User`:
267
278
 
@@ -323,26 +334,30 @@ column.
323
334
  }
324
335
  ```
325
336
 
326
- **Example response:**
337
+ **Example JSON data** (inside the MCP response envelope):
327
338
 
328
339
  ```json
329
- [
330
- {
331
- "identifier": "PaymentsController",
332
- "type": "controller",
333
- "file_path": "app/controllers/payments_controller.rb",
334
- "metadata": {
335
- "actions": ["create", "show", "webhook"],
336
- "routes": [
337
- { "verb": "POST", "path": "/payments", "action": "create" },
338
- { "verb": "POST", "path": "/payments/webhook", "action": "webhook" }
339
- ]
340
- }
340
+ {
341
+ "query": "payment",
342
+ "result_count": 1,
343
+ "results": [
344
+ { "identifier": "PaymentsController", "type": "controller", "match_field": "identifier" }
345
+ ],
346
+ "completeness": {
347
+ "status": "complete",
348
+ "reason": "exhausted",
349
+ "has_more": false,
350
+ "total_matches": 1,
351
+ "matched_lower_bound": 1
341
352
  }
342
- ]
353
+ }
343
354
  ```
344
355
 
345
- Search `source_code` when you want semantic matches, not just naming matches.
356
+ Use `lookup` on the returned identifier for source, actions, and routes. Search
357
+ `source_code` for textual matches beyond names. Supporting versions distinguish
358
+ exact totals from a bounded result prefix; `partial` means this is discovery,
359
+ not an exhaustive list. Completeness metadata is unreleased after `2.0.0.beta2`;
360
+ see the [search contract](MCP_SERVERS.md#search-completeness).
346
361
 
347
362
  ---
348
363
 
@@ -536,7 +551,7 @@ Static tools miss all of these because they only exist after Rails processes the
536
551
  }
537
552
  ```
538
553
 
539
- **What you'll get:** Units with no dependents, nothing in the codebase references them. Good candidates for removal or investigation.
554
+ **What you'll get:** Units with no recorded dependents in the published graph, excluding types treated as natural entry points. These are candidates for investigation, not proof of dead code. Method-body references and dynamic callers may be missing; verify source references, framework entry points, and runtime usage before removing anything. See [dependency graph coverage](MCP_SERVERS.md#dependency-graph-coverage).
540
555
 
541
556
  ---
542
557
 
@@ -796,7 +811,7 @@ Keys without a recognised suffix fall through to ActiveRecord `where(hash)` equa
796
811
 
797
812
  ### "Find code related to subscription billing"
798
813
 
799
- **Tool:** `codebase_retrieve` (Index Server, requires embedding provider)
814
+ **Tool:** `codebase_retrieve` (Index Server, semantic or explicit lexical mode)
800
815
 
801
816
  ```json
802
817
  {
@@ -805,7 +820,7 @@ Keys without a recognised suffix fall through to ActiveRecord `where(hash)` equa
805
820
  }
806
821
  ```
807
822
 
808
- **What you'll get:** A token-budgeted context string of the most semantically relevant units, ranked by hybrid search (semantic + keyword + PageRank). Requires an embedding provider (`embedding_provider: :openai` or `:ollama`) to be configured.
823
+ **What you'll get:** Ranked context within an estimated text-token budget. Default semantic mode uses configured embeddings and hybrid ranking; explicit lexical mode uses field-aware BM25 over published units, without a provider or `woods:embed`. The same query works in either configured mode, though rankings differ. Confirm the active mode with `woods_status.retriever.mode`; see the [retrieval guide](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval) for setup and budget limits.
809
824
 
810
825
  ---
811
826