woods 2.0.0.beta4 → 2.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (95) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +495 -471
  3. data/CONTRIBUTING.md +12 -2
  4. data/README.md +11 -26
  5. data/docs/AGENT_GUIDE.md +31 -12
  6. data/docs/AGENT_SETUP.md +17 -10
  7. data/docs/AUTOMATIC_MAINTENANCE.md +222 -0
  8. data/docs/BACKEND_MATRIX.md +13 -7
  9. data/docs/CLIENT_HOOKS.md +1 -1
  10. data/docs/CONFIGURATION_REFERENCE.md +44 -27
  11. data/docs/CONSOLE_MCP_SETUP.md +95 -16
  12. data/docs/DOCKER_SETUP.md +15 -0
  13. data/docs/EVALUATION.md +10 -4
  14. data/docs/EXTRACTOR_REFERENCE.md +14 -2
  15. data/docs/FAQ.md +14 -3
  16. data/docs/GETTING_STARTED.md +18 -17
  17. data/docs/INCREMENTAL_EXTRACTION.md +8 -3
  18. data/docs/INDEX_LAYOUT.md +2 -2
  19. data/docs/MCP_HTTP_TRANSPORT.md +54 -2
  20. data/docs/MCP_SERVERS.md +28 -11
  21. data/docs/MCP_TOOL_COOKBOOK.md +1 -1
  22. data/docs/MCP_WORKTREE_SETUP.md +13 -1
  23. data/docs/PUBLISHED_INDEX.md +1 -1
  24. data/docs/README.md +2 -1
  25. data/docs/RETRIEVAL_GUIDE.md +57 -8
  26. data/docs/SOURCE_FRESHNESS.md +1 -1
  27. data/docs/TOKEN_BENCHMARK.md +16 -10
  28. data/docs/TROUBLESHOOTING.md +133 -37
  29. data/docs/UPGRADING_TO_2.md +69 -7
  30. data/docs/WATCH_DAEMON.md +172 -17
  31. data/docs/WHY_WOODS.md +9 -5
  32. data/exe/woods-console +13 -11
  33. data/exe/woods-mcp-http +16 -9
  34. data/exe/woods-watch +5 -0
  35. data/lib/generators/woods/watch_generator.rb +53 -0
  36. data/lib/puma/plugin/woods.rb +10 -0
  37. data/lib/tasks/woods.rake +14 -0
  38. data/lib/woods/cache/cache_middleware.rb +18 -11
  39. data/lib/woods/console/adapter_family.rb +39 -0
  40. data/lib/woods/console/credential_index.rb +33 -3
  41. data/lib/woods/console/embedded_executor.rb +401 -43
  42. data/lib/woods/console/model_validator.rb +8 -0
  43. data/lib/woods/console/rack_middleware.rb +39 -10
  44. data/lib/woods/console/redactor.rb +24 -10
  45. data/lib/woods/console/safe_context.rb +44 -7
  46. data/lib/woods/console/sql_noise_stripper.rb +41 -12
  47. data/lib/woods/console/sql_table_scanner.rb +45 -34
  48. data/lib/woods/console/sql_validator.rb +37 -2
  49. data/lib/woods/console/stdio_transport.rb +27 -0
  50. data/lib/woods/extractor.rb +25 -7
  51. data/lib/woods/git_command.rb +6 -7
  52. data/lib/woods/git_provenance.rb +4 -6
  53. data/lib/woods/mcp/bearer_auth.rb +1 -1
  54. data/lib/woods/mcp/bootstrapper.rb +3 -1
  55. data/lib/woods/mcp/initialization_guidance.rb +1 -1
  56. data/lib/woods/mcp/origin_guard.rb +24 -77
  57. data/lib/woods/mcp/origin_policy.rb +124 -0
  58. data/lib/woods/mcp/server.rb +41 -9
  59. data/lib/woods/railtie_support.rb +8 -0
  60. data/lib/woods/retrieval/corpus_status.rb +46 -0
  61. data/lib/woods/retriever.rb +19 -7
  62. data/lib/woods/storage/local_corpus_stats.rb +32 -0
  63. data/lib/woods/storage/metadata_store.rb +20 -0
  64. data/lib/woods/storage/vector_store.rb +10 -0
  65. data/lib/woods/version.rb +1 -1
  66. data/lib/woods/watch/child_environment.rb +30 -0
  67. data/lib/woods/watch/cli.rb +91 -0
  68. data/lib/woods/watch/daemon.rb +55 -7
  69. data/lib/woods/watch/event_stream.rb +70 -0
  70. data/lib/woods/watch/guardian.rb +142 -0
  71. data/lib/woods/watch/installation/layout.rb +70 -0
  72. data/lib/woods/watch/installation/options.rb +128 -0
  73. data/lib/woods/watch/installation/planner.rb +128 -0
  74. data/lib/woods/watch/installation/probe.rb +101 -0
  75. data/lib/woods/watch/installation/receipt.rb +77 -0
  76. data/lib/woods/watch/installation/recovery.rb +64 -0
  77. data/lib/woods/watch/installation/templates.rb +58 -0
  78. data/lib/woods/watch/installation.rb +56 -0
  79. data/lib/woods/watch/lifecycle.rb +182 -0
  80. data/lib/woods/watch/managed_child.rb +113 -0
  81. data/lib/woods/watch/managed_cleanup.rb +48 -0
  82. data/lib/woods/watch/managed_process.rb +144 -0
  83. data/lib/woods/watch/puma_adapter.rb +87 -0
  84. data/lib/woods/watch/puma_child.rb +66 -0
  85. data/lib/woods/watch/supervision_records.rb +95 -0
  86. data/lib/woods/watch/supervision_status.rb +104 -0
  87. data/lib/woods/watch/supervisor.rb +161 -0
  88. data/lib/woods/watch/supervisor_reporting.rb +46 -0
  89. data/plugin/.claude-plugin/plugin.json +1 -1
  90. data/plugin/skills/woods-agent-enable/SKILL.md +1 -1
  91. data/plugin/skills/woods-diagnose/SKILL.md +88 -8
  92. data/plugin/skills/woods-investigate/SKILL.md +6 -6
  93. data/plugin/skills/woods-mcp-config/SKILL.md +43 -1
  94. data/plugin/skills/woods-setup/SKILL.md +66 -4
  95. metadata +37 -5
@@ -23,7 +23,7 @@ This guide covers the most common problems encountered when installing, extracti
23
23
  | `No such container` | Wrong container name | Check with `docker ps --format '{{.Names}}'` |
24
24
  | `JSON parse errors` (MCP) | Rails boot noise on stdout | Remove `puts` calls from initializers |
25
25
  | Query timeout | Large table, no scope | Add scope conditions to narrow results |
26
- | `Extraction failed for …; the previous generation remains active` | A consumer handled a source error during incremental extraction or refresh (unreleased after `2.0.0.beta3`) | Fix the logged source error and retry the [complete batch](INCREMENTAL_EXTRACTION.md#handled-source-errors-and-retry); watch keeps it pending |
26
+ | `Extraction failed for …; the previous generation remains active` | A consumer handled a source error during incremental extraction or refresh (included in Woods `2.0.0`) | Fix the logged source error and retry the [complete batch](INCREMENTAL_EXTRACTION.md#handled-source-errors-and-retry); watch keeps it pending |
27
27
  | Empty extraction output | `eager_load!` failure | Check for `NameError` in boot output |
28
28
  | Git metadata missing | Shallow clone in CI | Use `fetch-depth: 0` for complete history |
29
29
  | Parallel tool calls all fail | MCP client batches calls | Send calls sequentially, validate params first |
@@ -53,13 +53,44 @@ version. See [manifest writer provenance](PUBLISHED_INDEX.md#manifest-writer-pro
53
53
 
54
54
  If a tool call fails with **"Tool not found: … not available in the installed Woods v…"**, the client is asking for a tool a newer gem provides. Run `bundle update woods` and reconnect the MCP server, then retry.
55
55
 
56
+ ### Watcher startup or planned restart fails
57
+
58
+ Managed `woods-watch` startup is **included in Woods `2.0.0`**; record the
59
+ loaded version/path and revision, then verify executable and generator help.
60
+ If changing an initializer stops every Foreman process, replace a bare
61
+ `woods:watch` entry with the [managed setup](WATCH_DAEMON.md#managed-development-startup).
62
+
63
+ Read launcher logs and `woods_status` supervision records separately from daemon
64
+ liveness and index freshness. `retrying` means the last generation remains usable
65
+ while boot is retried. A parked ownership/protocol conflict requires correcting
66
+ the selected owner or installed command and restarting that owner; do not delete
67
+ claim files or kill PIDs taken from status. No index-visible record exists before
68
+ the first boot resolves the application's output directory.
69
+
70
+ Unset `WOODS_WATCH_IDLE_TIMEOUT` in managed modes. If the boot deadline is reached,
71
+ diagnose Bundler/initializer startup before increasing `--boot-timeout`; a valid
72
+ long extraction has a separate readiness state and is not bounded by that clock.
73
+ If setup created a Procfile but normal `bin/dev` still only launches Rails, choose
74
+ Puma or explicitly run the selected Foreman command. The generator never rewrites
75
+ `bin/dev` or starts services during preview.
76
+
77
+ If installation reports a pending transaction, use `woods:watch --operation
78
+ recover` through the Rails generator, initially with `--pretend`; see
79
+ [owned setup recovery](WATCH_DAEMON.md#ownership-updates-and-removal). That Rails
80
+ command boots the application first. For broken initializers use the documented
81
+ direct bundled Ruby helper, which does not boot Rails or require task discovery.
82
+ Both refuse to overwrite intervening edits. A Puma setup refusal for
83
+ `config/puma/development.rb` means the default
84
+ configuration would bypass the generated plugin; select an external/Foreman
85
+ arrangement instead of installing an inactive directive.
86
+
56
87
  ### Semantic graph validation errors
57
88
 
58
89
  In development versions containing #413, `woods:validate` rejects graphs that
59
90
  parse as JSON but disagree with their indexes. Errors name the section and
60
91
  identity, for example `reverse["http_api"]: missing "Order"`, a duplicate typed
61
- variant, or an indexed unit absent from `nodes`. This is unreleased after
62
- `2.0.0.beta2`; check the installed gem before expecting these diagnostics.
92
+ variant, or an indexed unit absent from `nodes`. This is included in Woods
93
+ `2.0.0`; check the installed gem before expecting these diagnostics.
63
94
 
64
95
  Keep the failing generation and report the exact errors. Run a full extraction
65
96
  in a fresh application process with the intended bundle, then validate again.
@@ -88,7 +119,7 @@ Scoped resets leave corrupt state untouched. Valid state keeps any unrelated
88
119
  operation entries, and missing state remains a no-op without creating a file.
89
120
  A permission failure must be corrected before repair can succeed.
90
121
 
91
- This recovery is unreleased after `2.0.0.beta2`; check the installed version.
122
+ This recovery is included in Woods `2.0.0`; check the installed version.
92
123
  Older versions report corrupt state as nothing to repair. Stop pipeline writers,
93
124
  back up the configured guard state's `pipeline_guard.json`, and remove only that
94
125
  file before restarting, or upgrade to a version containing the fix.
@@ -160,7 +191,12 @@ For subsequent runs, use incremental mode instead of full extraction:
160
191
  bundle exec rake woods:incremental
161
192
  ```
162
193
 
163
- Incremental extraction only re-extracts files that changed since the last run. It skips unchanged units and is typically 5-10× faster.
194
+ Incremental extraction dispatches the selected changed paths, including affected
195
+ concern consumers and whole-app extractors whose trigger paths changed. The default
196
+ Git range is `HEAD~1`; pass an explicit range or `CHANGED_FILES` for other batches.
197
+ It can reduce extraction work, but Rails boot, graph rebuilding, and publication
198
+ still contribute to runtime. Measure the improvement in your application; Woods
199
+ does not guarantee a speedup. See the [incremental contract](INCREMENTAL_EXTRACTION.md).
164
200
 
165
201
  ---
166
202
 
@@ -228,7 +264,7 @@ dependents after an incremental run.
228
264
  identities when restoring and updating the graph.
229
265
 
230
266
  **Fix:** Check whether the installed version includes B-193; this fix is
231
- unreleased. After upgrading to a version containing the fix, run
267
+ included in Woods `2.0.0`. After upgrading to a version containing the fix, run
232
268
  `bundle exec rake woods:extract` once to rebuild lost reverse dependencies.
233
269
  Loading an already damaged graph does not restore discarded entries. See the
234
270
  [incremental graph contract](INCREMENTAL_EXTRACTION.md#the-contract).
@@ -239,7 +275,7 @@ Loading an already damaged graph does not restore discarded entries. See the
239
275
  most files as `change_frequency: new` in a shallow CI checkout.
240
276
 
241
277
  **Cause:** A shallow clone truncates HEAD ancestry. The shallow-checkout guard is
242
- unreleased after 2.0.0.beta2: current source omits git enrichment and warns once,
278
+ included in Woods `2.0.0`: Woods omits git enrichment and warns once,
243
279
  rather than treating the truncated history as complete. If repository depth
244
280
  cannot be verified, enrichment is also omitted; check git access and version.
245
281
 
@@ -256,12 +292,33 @@ clone), then run full extraction to replace retained metadata:
256
292
  Two commits can suffice for an incremental diff, but do not establish the full
257
293
  ancestry needed for churn metadata.
258
294
 
295
+ ### Git executable is missing from the extraction environment
296
+
297
+ **Symptom:** Extraction logs `Git history unavailable: git executable was not
298
+ found in PATH`, and newly extracted units have no `metadata.git`. Older builds
299
+ can omit this enrichment silently when the executable is missing.
300
+
301
+ **Fix:** Run `git --version` in the same container and environment that runs
302
+ extraction. Install Git 2.31 or newer there, ensure its executable is on `PATH`,
303
+ then run full `woods:extract` to refresh every unit's history. A working Git
304
+ installation on the host does not provide Git inside an application container.
305
+
306
+ Extraction continues without inventing zero-commit history. The warning appears
307
+ once per extractor instance when the application has a `.git` entry or an
308
+ explicit `WOODS_GIT_DIR`/`GIT_DIR` setting. A source archive with neither remains
309
+ supported and quiet. `GIT_BRANCH`/`GIT_SHA` provenance fallback is unchanged;
310
+ those values identify a build but cannot supply per-file history.
311
+
312
+ This diagnostic is emitted during extraction. `woods_status.ready` and a
313
+ manifest Git SHA do not establish that per-unit history was available, and
314
+ `recent_changes` returning no results does not prove no files changed.
315
+
259
316
  ---
260
317
 
261
318
  ### Git enrichment warns that history could not be read completely
262
319
 
263
- Current source uses an explicit merge-diff mode requiring **Git 2.31 or newer**.
264
- This is unreleased after 2.0.0.beta2: first confirm the installed Woods version.
320
+ Woods 2.0 uses an explicit merge-diff mode requiring **Git 2.31 or newer**.
321
+ First confirm the installed Woods version.
265
322
  Check `git --version` inside the same container/process environment as extraction,
266
323
  and upgrade git if it is older. On a supported version, check that the application's
267
324
  `HEAD` and object store can be read using the same `WOODS_GIT_DIR` setting.
@@ -293,29 +350,62 @@ When it does not, the git keys are omitted from every unit, provenance records
293
350
  `"unknown"`, and one warning names git's own reason. Absent keys mean "not
294
351
  known"; they never mean "brand new".
295
352
 
296
- **Fix:** Point `WOODS_GIT_DIR` at the *canonical* git directory, the one the
297
- worktree's `gitdir:` pointer ultimately leads to, and make sure it is mounted:
353
+ **Fix:** Restore access to both the worktree-specific Git directory and the
354
+ shared objects and refs using the mount layouts below, then run a full
355
+ `woods:extract` to replace retained metadata.
356
+
357
+ ### Git directory mounts for linked worktrees
358
+
359
+ `WOODS_GIT_DIR` is passed directly to Git's `--git-dir`. It selects that
360
+ directory's `HEAD` for manifest provenance, per-file history, and incremental
361
+ diff ranges. **For a linked worktree, pointing it at the shared `.git` root
362
+ selects the primary checkout's HEAD.** A successful Git command alone does
363
+ not prove Woods is reading the intended branch.
364
+
365
+ First inspect Git metadata on the host, from the intended worktree:
298
366
 
299
367
  ```bash
300
- # docker-compose.yml, mounting the parent repository's git directory
301
- # volumes:
302
- # - /path/to/repo/.git:/canonical-git:ro
303
- WOODS_GIT_DIR=/canonical-git bundle exec rake woods:extract
368
+ git -C /path/to/worktree rev-parse --absolute-git-dir
369
+ # Example: /path/to/repo/.git/worktrees/wt
370
+ git -C /path/to/worktree rev-parse --path-format=absolute --git-common-dir
371
+ # Example: /path/to/repo/.git
372
+ git -C /path/to/worktree rev-parse --abbrev-ref HEAD
373
+ git -C /path/to/worktree rev-parse HEAD
304
374
  ```
305
375
 
306
- `WOODS_GIT_DIR` wins over whatever the worktree pointer says, and applies to
307
- every git call Woods makes: per-unit enrichment, `manifest.json` provenance,
308
- and the diff range `woods:incremental` resolves. All three run through
309
- `Woods::GitCommand.argv`, so the override cannot reach two of them and miss the
310
- third.
376
+ The worktree ID in this example is `wt`. Use the ID returned by Git metadata;
377
+ it need not match the branch name. Choose one of these layouts:
378
+
379
+ - **Same-path mount:** mount the complete shared directory read-only at its
380
+ original absolute path (`/path/to/repo/.git:/path/to/repo/.git:ro`). With the
381
+ application's existing `.git` pointer resolvable, leave `WOODS_GIT_DIR`
382
+ unset and remove conflicting Git-directory overrides from the environment.
383
+ - **Relocated mount:** mount that complete directory read-only at a new path
384
+ (`/path/to/repo/.git:/mounted-common:ro`), including `objects`, `refs`, and
385
+ `worktrees`. Select the worktree-specific directory inside it:
386
+
387
+ ```bash
388
+ WOODS_GIT_DIR=/mounted-common/worktrees/wt bundle exec rake woods:extract
389
+ ```
390
+
391
+ Mounting only the private worktree directory can leave its `commondir` pointer
392
+ without access to shared objects and refs. Git's own environment variables are
393
+ inherited by the subprocess; check any existing `GIT_DIR` and `GIT_COMMON_DIR`
394
+ settings when diagnosing the effective layout. The complete layouts above
395
+ preserve both worktree identity and shared storage.
396
+
397
+ In the extraction container, verify the relocated selection against the host
398
+ branch and exact SHA before extracting (replace `/app` and `wt` as needed):
399
+
400
+ ```bash
401
+ git --git-dir=/mounted-common/worktrees/wt --work-tree=/app -C /app rev-parse --abbrev-ref HEAD
402
+ git --git-dir=/mounted-common/worktrees/wt --work-tree=/app -C /app rev-parse HEAD
403
+ ```
311
404
 
312
- **`GIT_DIR` alone is not enough for a linked worktree.** Woods honors git's own
313
- `GIT_DIR` and `GIT_COMMON_DIR` because git does, but setting `GIT_DIR` to a
314
- worktree's private git directory only moves the failure: the `commondir`
315
- pointer inside it is relative, so it still resolves to a path that is not
316
- mounted, and `GIT_COMMON_DIR` does not override it. Either mount the canonical
317
- git directory at the same absolute path the pointer names, or use
318
- `WOODS_GIT_DIR`.
405
+ After fixing the selection, run full `woods:extract` and verify the published
406
+ manifest. Incremental extraction can retain older per-file Git metadata.
407
+ A commit alone does not necessarily trigger the source-file watcher; run a
408
+ full extraction when current history and provenance are required.
319
409
 
320
410
  ---
321
411
 
@@ -375,17 +465,23 @@ retries after a later filesystem event.
375
465
 
376
466
  ### `manifest.json` shows the wrong branch (or `git_branch: "unknown"`) in a worktree
377
467
 
378
- **Symptom:** `git_branch` / `git_sha` in `manifest.json` name a different branch than the worktree is actually on, or report `"unknown"`. The extracted units themselves are correct, only the provenance metadata is off.
468
+ **Symptom:** `git_branch` / `git_sha` in `manifest.json` name a different
469
+ branch or SHA than the intended worktree, or report `"unknown"`.
379
470
 
380
- **Cause:** In a linked git worktree, `.git` is a *file* containing a `gitdir:` pointer to the real git directory, often an absolute host path. When extraction runs where that path can't be resolved (e.g. inside a container where the host path isn't mounted), git can't read the ref. Woods now reports `"unknown"` in that case rather than emitting a stale, misleading value (previously it fell back to a baked `GIT_BRANCH`/`GIT_SHA` build arg).
471
+ **Cause:** An unreachable `.git` file's `gitdir:` pointer prevents Git from
472
+ resolving the worktree's HEAD. An override selecting the shared `.git` root
473
+ instead resolves the primary checkout's HEAD successfully. That wrong selection
474
+ also affects per-file history and HEAD-based incremental ranges.
381
475
 
382
- **Fix:** Make the worktree's git directory reachable from the extraction environment, for example, mount the parent repository (the directory the `gitdir:` pointer references) into the container, or run extraction from a normal (non-worktree) checkout. With the real git directory reachable, `git_branch`/`git_sha` resolve correctly. If the checkout legitimately ships without a `.git` at all (a source tarball, or a Docker `COPY` that excludes it), set `GIT_BRANCH` / `GIT_SHA` explicitly. Woods honors these when there is no `.git` at the root (or no git binary), but suppresses them when a `.git` *is* present but unresolvable (so a stale build arg can't mask a worktree).
476
+ **Fix:** Follow [Git directory mounts for linked worktrees](#git-directory-mounts-for-linked-worktrees),
477
+ compare the selected branch and exact SHA in the extraction environment, then
478
+ run a full extraction. Compare the newly published manifest, not a retained
479
+ generation. A commit without a source edit may leave the watcher idle.
383
480
 
384
- When the canonical git directory is mounted but not at the path the pointer
385
- names, set `WOODS_GIT_DIR` to where it actually is. It wins over the pointer for
386
- provenance and for unit-level git metadata both. Setting git's own `GIT_DIR` to
387
- the worktree's private git directory does not work: its `commondir` pointer is
388
- relative and resolves outside the mount.
481
+ For a checkout legitimately shipped without `.git` (such as a source tarball),
482
+ `GIT_BRANCH` / `GIT_SHA` can supply provenance. They are fallbacks only when
483
+ `.git` is absent or Git is unavailable; a present but unresolvable `.git`
484
+ reports `"unknown"` instead of substituting stale build arguments.
389
485
 
390
486
  ---
391
487
 
@@ -395,9 +491,9 @@ relative and resolves outside the mount.
395
491
 
396
492
  ### Index cannot be resolved at startup
397
493
 
398
- **Symptom:** An Index MCP executable exits with `Could not resolve a published Woods index in: /path/to/...` even though extraction completed. This headline is unreleased after `2.0.0.beta3`; older versions say `No manifest.json found`. Both mean the selected index could not resolve its manifest, not that an atomic index needs a root manifest.
494
+ **Symptom:** An Index MCP executable exits with `Could not resolve a published Woods index in: /path/to/...` even though extraction completed. This headline is included in Woods `2.0.0`; older versions say `No manifest.json found`. Both mean the selected index could not resolve its manifest, not that an atomic index needs a root manifest.
399
495
 
400
- Embedded Index MCP startup through `IndexReader` also raises an `ArgumentError` with the selected directory and layout guidance when the marker cannot resolve a manifest, including malformed marker shapes such as `[]` or a numeric `payload` (unreleased after `2.0.0.beta3`). Earlier builds may expose a raw `TypeError` or `NoMethodError` for those shapes. Inspect the marker and preserve the failing index before attempting recovery.
496
+ Embedded Index MCP startup through `IndexReader` also raises an `ArgumentError` with the selected directory and layout guidance when the marker cannot resolve a manifest, including malformed marker shapes such as `[]` or a numeric `payload` (included in Woods `2.0.0`). Earlier builds may expose a raw `TypeError` or `NoMethodError` for those shapes. Inspect the marker and preserve the failing index before attempting recovery.
401
497
 
402
498
  **Cause:** The selected directory is not the published index root, the published generation cannot be resolved, or the path is not visible to the MCP process. A container path is appropriate for a container process; a host process needs the host-visible path.
403
499
 
@@ -2,14 +2,41 @@
2
2
 
3
3
  Woods 2.0 changes observable index identifiers, publication layout, vector-store reconciliation, and the supported MCP surface. Plan a clean re-index. Do not upgrade a shared or durable index in place without a backup and a rollback window.
4
4
 
5
- This guide assumes the last v1 release, 1.6.1, and targets 2.0.0.
5
+ This guide covers the supported 1.6.x line and targets 2.0.0. Use the latest
6
+ published 1.6.x security patch as the rollback version.
6
7
 
7
8
  <!-- release-state:upgrade-availability -->
8
- > This tree declares 2.0.0.beta4 as a prerelease. After RubyGems lists it, pin it with
9
- > `gem "woods", "2.0.0.beta4"`; `~> 2.0` resolves only once
10
- > 2.0.0 is published.
11
9
  <!-- release-state:end -->
12
10
 
11
+ ## 2.0.1 security maintenance update
12
+
13
+ This maintenance line adds Console request/output policy corrections and isolates
14
+ retrieval contexts between retriever instances. It does not add the graph or
15
+ extraction features under development for 2.1. Confirm the installed package
16
+ version and use its matching tag documentation.
17
+
18
+ No index or database schema migration is required. Restart Console/MCP processes
19
+ after upgrading. Explicit malformed HTTP origin entries now fail at boot with the
20
+ offending entry named; fix the entry rather than weakening authentication.
21
+ Automatic and manual Console mounts raise `Woods::ConfigurationError` for these
22
+ settings, including invalidly encoded entries. With
23
+ no allowlist configured, the existing loopback defaults remain unchanged. See
24
+ [HTTP origin matching](MCP_HTTP_TRANSPORT.md#browser-origins-dns-rebinding-defense).
25
+
26
+ Raw `console_sql` now refuses ambiguous protected results and genuinely unknown
27
+ adapter families. PostgreSQL-subclass adapters retain PostgreSQL handling;
28
+ structured Console tools remain available with other adapters. Prefer explicit
29
+ unaliased scalar projections or structured reads when a query is refused.
30
+ [Console setup](CONSOLE_MCP_SETUP.md#maintenance-policy-corrections) describes the
31
+ compatibility boundary. Context caches refill after restart; retired entries
32
+ follow the configured TTL or backend eviction. See
33
+ [retrieval cache options](CONFIGURATION_REFERENCE.md#retrieval-cache-options).
34
+ Rolling back restores the affected behavior.
35
+
36
+ Polymorphic `belongs_to` association counts remain unsupported in 2.0.1 and
37
+ 1.6.4 and return a generic execution error; the 2.1 functional correction is
38
+ not backported.
39
+
13
40
  ## Upgrade outcome
14
41
 
15
42
  After this runbook you will have:
@@ -21,6 +48,37 @@ After this runbook you will have:
21
48
  - an MCP client connected to the v2 packaged tool surface;
22
49
  - a documented way back to v1 if verification fails.
23
50
 
51
+ ### Console read compatibility
52
+
53
+ For supporting security-patch revisions, any nonempty column or EAV redaction
54
+ policy makes `console_sql` refuse relation and CTE column alias lists, including
55
+ lists on base tables, derived tables, parenthesized `VALUES` sources and table
56
+ functions, regardless of the selected names. Use explicit, unaliased protected
57
+ columns or structured tools.
58
+ Typed EAV lookup includes all registered models with a case-insensitive matching
59
+ final table name, including across schemas. It can therefore mask extra values;
60
+ qualification does not narrow this conservative type set. Sensitive key values
61
+ still require their exact stored or cast spelling.
62
+
63
+ The 1.6.x function denylist becomes a read-only function allowlist in 2.x.
64
+ Supply one `console_query` expression per `select` array entry: 2.x refuses
65
+ comma-combined entries that 1.6.4 splits. Raw SQL requires a recognized adapter
66
+ family; structured tools remain available on other adapters. Configure binary
67
+ secret columns explicitly for redaction. See the canonical
68
+ [read policy compatibility](CONSOLE_MCP_SETUP.md#read-policy-compatibility) and
69
+ [statement timeouts](CONSOLE_MCP_SETUP.md#statement-timeout) for limits.
70
+
71
+ Supporting Console HTTP revisions deliberately permit allowlisted non-loopback
72
+ Hosts through the SDK where 2.0.0 could refuse them. Bearer authentication remains
73
+ required. An explicit origin list replaces browser-origin defaults; wildcards
74
+ are not supported. Ruby-configured 2.x origins reject surrounding whitespace
75
+ that 1.6.4 trims; the HTTP executable trims comma-separated environment entries
76
+ on both lines. Review [HTTP origin configuration](MCP_HTTP_TRANSPORT.md#origin-configuration-compatibility).
77
+
78
+ Context-cache namespace rotation retires entries for normal TTL or backend
79
+ eviction; disabling both can retain them indefinitely. See
80
+ [retrieval cache options](CONFIGURATION_REFERENCE.md#retrieval-cache-options).
81
+
24
82
  ## What changes
25
83
 
26
84
  | v2 change | What can break | Required response |
@@ -95,7 +153,11 @@ Also back up managed Obsidian/Unblocked destinations before allowing a mass stal
95
153
 
96
154
  ### 3. Choose a rollback point
97
155
 
98
- Keep the v1 Gemfile/lockfile commit and all durable-store backups until v2 extraction, MCP calls, retrieval, and exports are verified. Downgrading the gem does not translate v2 identifiers back to v1.
156
+ Record and test a Gemfile/lockfile selecting the latest published 1.6.x security
157
+ patch as the rollback bundle. If the current installation is older, verify that
158
+ patched v1 bundle before beginning the v2 migration. Keep its commit and all
159
+ durable-store backups until v2 extraction, MCP calls, retrieval, and exports are
160
+ verified. Downgrading the gem does not translate v2 identifiers back to v1.
99
161
 
100
162
  ## Upgrade the application
101
163
 
@@ -142,7 +204,7 @@ bin/rails woods:validate
142
204
  bin/rails woods:stats
143
205
  ```
144
206
 
145
- Unreleased after `2.0.0.beta3`: `woods:clean` removes index artifacts but keeps
207
+ Included in Woods `2.0.0`: `woods:clean` removes index artifacts but keeps
146
208
  the output directory and its hidden extraction guard. This stable guard lets
147
209
  concurrent writers coordinate safely; its presence does not mean an index remains.
148
210
 
@@ -331,7 +393,7 @@ Complete every applicable check:
331
393
  If verification fails:
332
394
 
333
395
  1. stop v2 MCP, watcher, embedding, and exporter processes;
334
- 2. restore the v1 Gemfile and lockfile or deploy the recorded v1 commit;
396
+ 2. restore the tested, patched v1 Gemfile and lockfile or deploy its recorded commit;
335
397
  3. run the v1 `woods:clean` before restoring anything under the configured output directory;
336
398
  4. either restore the complete pre-upgrade v1 output-directory backup, or run a fresh v1 extraction and then restore its v1 `dumps/` and configuration artifacts;
337
399
  5. restore external vector-store and managed export backups when v2 modified them;
data/docs/WATCH_DAEMON.md CHANGED
@@ -34,15 +34,160 @@ Ctrl-C to stop.
34
34
  | `WOODS_WATCH_CATCH_UP` | `1` | `0` skips the startup reconciliation |
35
35
  | `WOODS_WATCH_TRUST_FOREIGN_HOST` | unset | `1` lets a reader trust a fresh foreign-host heartbeat without checking its pid locally; see [cross-host liveness](#cross-host-liveness) |
36
36
 
37
- Run it under a supervisor. When boot-captured configuration changes the daemon
38
- exits `75` (`EX_TEMPFAIL`) on purpose, see [Restart triggers](#restart-triggers).
37
+ The raw task needs a supervisor that restarts it after exit `75` (`EX_TEMPFAIL`);
38
+ see [Restart triggers](#restart-triggers). Docker restart policies can supply
39
+ this. A bare task in Foreman cannot: Foreman shuts down its process group when a
40
+ child exits. Use the managed launcher below for native Procfile development.
39
41
 
40
- ```yaml
41
- # Procfile.dev
42
- web: bin/rails server
43
- woods: bundle exec rake woods:watch
42
+ ### Managed development startup
43
+
44
+ **Included in Woods `2.0.0` ([#538](https://github.com/lost-in-the/woods/issues/538)).**
45
+ Record the installed gem path and Git revision as well as VERSION; a checkout
46
+ can have new behavior with the last beta's version. Verify both commands before
47
+ using the new setup:
48
+
49
+ ```bash
50
+ bundle exec woods-watch --help
51
+ bin/rails generate woods:watch --help
52
+ ```
53
+
54
+ Choose one lifecycle owner per application index:
55
+
56
+ | Existing development workflow | Setup |
57
+ |---|---|
58
+ | Rails/Puma, including a `bin/dev` that just starts Rails | Opt-in `puma` mode; the development Puma master starts one separate extraction process, regardless of worker count. |
59
+ | An existing Foreman workflow | `procfile` mode with the exact existing Procfile and selected normal Foreman command. |
60
+ | Docker Compose, Grove, or another supervisor | `external` mode returns the raw task command; configure its lifecycle through the existing supervisor. |
61
+
62
+ For Puma, preview before applying:
63
+
64
+ ```bash
65
+ bin/rails generate woods:watch --mode puma --pretend
66
+ bin/rails generate woods:watch --mode puma
67
+ ```
68
+
69
+ This adds an executable `bin/woods-watch` and an owned directive to
70
+ `config/puma.rb`. The directive loads the plugin only when the active Woods gem
71
+ contains it. An absent gem or an older version without the plugin leaves Puma
72
+ running without a watcher; this also works for Git/path bundles. The adapter
73
+ separately enforces Puma's finalized development environment.
74
+ Starting a console, task, or production
75
+ Puma does not start a watcher. The watcher still uses another Rails process and its associated memory.
76
+ Puma installation checks the application's installed Puma version (supported
77
+ majors: 6, 7, and 8 on native Unix Ruby) and selects the normal `bin/rails server`
78
+ path using `config/puma.rb`. If `config/puma/development.rb` exists, setup refuses
79
+ because Puma would choose it first. Custom `-C` configurations need an explicit
80
+ external/Foreman arrangement; this generator does not discover or rewrite them.
81
+
82
+ For an existing Foreman arrangement, select the command you will actually use:
83
+
84
+ ```bash
85
+ bin/rails generate woods:watch --mode procfile --procfile Procfile.dev \
86
+ --manager-command 'foreman start -f Procfile.dev' --pretend
87
+ ```
88
+
89
+ Repeat without `--pretend` to apply. Preflight checks task discovery and Foreman
90
+ availability with bounded commands; it does not start services. Installation
91
+ preserves `bin/dev` and every unowned service. If `bin/dev` only runs Rails, it
92
+ will still only run Rails: choose Puma mode or explicitly use the selected
93
+ Foreman command. No process-manager gem is installed automatically.
94
+
95
+ Run setup with the same application environment as normal startup. Preflight
96
+ preserves environment-based Bundler configuration, including `BUNDLE_PATH`,
97
+ `BUNDLE_APP_CONFIG`, and excluded groups, while resetting inherited activation
98
+ state and requesting frozen resolution. Earlier Git builds of this
99
+ installer dropped those settings ([#540](https://github.com/lost-in-the/woods/issues/540)).
100
+ If normal task discovery works but installer preflight cannot find Rails or an
101
+ installed dependency, check the loaded revision and bundle environment before
102
+ reinstalling gems or adding a persistent `.bundle/config` workaround.
103
+
104
+ Installer refusals return a nonzero exit status with their diagnostic (#544,
105
+ included in Woods `2.0.0`). Earlier Git builds printed the refusal but exited
106
+ successfully, so automation using those builds must also check the diagnostic and
107
+ whether installation actually applied.
108
+
109
+ Use `--child-command 'bundle exec rails woods:watch'` when that is the
110
+ application's actual Rails entrypoint. Command strings become explicit argument
111
+ vectors; shell pipelines, variable expansion, and shell setup scripts are not
112
+ supported. Do not prepend an `environment` task or an already booted Rails runner.
113
+
114
+ ### Ownership, updates, and removal
115
+
116
+ The generator records relative owned paths and fingerprints in `.woods-watch.json`.
117
+ Commit it with the generated configuration so a new clone/worktree can update or
118
+ remove that setup. Runtime transaction state stays under `tmp/woods-watch-install/`;
119
+ keep that directory ignored. Changes to owned content or executable permissions
120
+ cause a conflict rather than an overwrite. Review the conflict and restore or
121
+ adapt the owned setup explicitly; do not delete the receipt to force an overwrite.
122
+
123
+ ```bash
124
+ # Change modes or update owned setup (include the selected mode's options).
125
+ bin/rails generate woods:watch --operation update --mode puma --pretend
126
+ # Remove owned startup fragments and wrapper; preserve application edits.
127
+ bin/rails generate woods:watch --operation remove --pretend
44
128
  ```
45
129
 
130
+ Repeat without `--pretend` to apply. Stop an existing watcher through its current
131
+ manager before changing owners; the generator does not stop running processes.
132
+ Explicit updates refresh owned directives in place, preserving surrounding text
133
+ and the block's newline style. Repeating setup preserves the existing directive.
134
+ Earlier Git builds used a Puma guard that checked only whether Woods was loaded;
135
+ use the update command above with a supporting bundle to install the capability
136
+ guard ([#542](https://github.com/lost-in-the/woods/issues/542), included in Woods
137
+ `2.0.0`). Do not hand-edit the owned block or receipt.
138
+
139
+ For a permanent downgrade, remove the startup configuration while the supporting
140
+ gem is still installed. The updated Puma guard lets a branch with an older gem
141
+ boot without indexing; it does not make `bin/woods-watch` or a Foreman entry
142
+ compatible with that gem.
143
+ For Docker/Grove, see [automatic maintenance](AUTOMATIC_MAINTENANCE.md).
144
+
145
+ An interrupted apply retains its local transaction journal. When Rails boots,
146
+ preview recovery with `bin/rails generate woods:watch --operation recover
147
+ --pretend`, then repeat without `--pretend`. The Rails command boots the application
148
+ before invoking the generator; its `--pretend` prevents installation writes,
149
+ not application initialization.
150
+
151
+ If an initializer prevents Rails boot, run the boot-free recovery helper directly
152
+ from the application root using its bundle:
153
+
154
+ ```bash
155
+ bundle exec ruby -rwoods/watch/installation -e \
156
+ 'puts Woods::Watch::Installation.new(root: Dir.pwd).recover(pretend: true)'
157
+ ```
158
+
159
+ Repeat with `pretend: false` to restore the recorded transaction. Recovery checks
160
+ snapshots and changes only owned files; concurrent edits keep the journal for
161
+ manual resolution. The helper does not load Rails or run task probes. Do not
162
+ delete a pending journal to force setup.
163
+
164
+ ### Managed restart and failure behavior
165
+
166
+ `woods-watch -- bin/rails woods:watch` stays outside Rails and starts a fresh child
167
+ for each planned restart. It absorbs exit 75 so Foreman/Puma remain running.
168
+ Managed launching requires POSIX process groups and `fork` (Linux/macOS Ruby).
169
+ Other platforms should use the raw task with an external supervisor.
170
+ Boot failures retry with bounded backoff; a failed extraction leaves the last
171
+ good generation readable. `--boot-timeout SECONDS` defaults to 300 seconds and
172
+ bounds boot/handshake, not a valid extraction or writer-lock wait.
173
+
174
+ Managed mode requires `WOODS_WATCH_IDLE_TIMEOUT` to be unset: automatic maintenance
175
+ needs a resident child. An unexpected clean stop, incompatible child, or another
176
+ owner parks the launcher with a diagnostic until its normal owner restarts it.
177
+ There is no automatic ownership takeover. Managed duplicate prevention is scoped
178
+ to one host/process namespace; do not mix raw and managed writers across different
179
+ containers sharing an index. Use one external owner for that arrangement.
180
+
181
+ Daemon liveness, completed startup reconciliation, and source freshness remain
182
+ different facts. Managed supervision records under `watch_supervisors/` expose
183
+ starting/retrying/parked state separately in `woods_status`; an alive supervisor
184
+ without a maintaining child does not provide daemon coverage. If the first Rails
185
+ boot fails before resolving the configured index path, diagnostics are logs-only.
186
+
187
+ Verify startup catch-up, an edit, and an initializer restart through the existing
188
+ MCP connection before declaring automatic maintenance active. With worktree
189
+ management, also verify source and index mounts switch together.
190
+
46
191
  ## One cycle
47
192
 
48
193
  ```
@@ -169,7 +314,7 @@ is alive, so *alive has to mean covered*.
169
314
  The standalone `woods:watch` task snapshots reload/restart inputs before invoking
170
315
  Rails' `environment` task. Inputs unchanged across that boundary, including
171
316
  carried paths that remain deleted, may be reconciled by a full extraction.
172
- Unreleased after `2.0.0.beta3`: registered restart inputs deleted while the
317
+ Included in Woods `2.0.0`: registered restart inputs deleted while the
173
318
  daemon was stopped also trigger a full extraction after a fresh environment
174
319
  boot. Nominal framework paths still use the bounded deletion sweep.
175
320
  Changes during environment initialization still require restart. Lock contention,
@@ -283,6 +428,14 @@ Ignored by default: `.git`, `node_modules`, `tmp`, `log`, `coverage`,
283
428
  `vendor/bundle`, `public/assets`, `public/packs`, `storage`. That ignore list is
284
429
  what keeps a polling scan bounded.
285
430
 
431
+ A commit, ref update, or other Git-only change does not necessarily trigger a
432
+ source-file event. The watcher can therefore keep source current while the
433
+ published Git history or provenance still describes an earlier commit. Run full
434
+ `woods:extract` when current Git metadata is required; incremental extraction
435
+ refreshes history only for rewritten units. For containerized linked worktrees,
436
+ verify [Git directory selection](TROUBLESHOOTING.md#git-directory-mounts-for-linked-worktrees)
437
+ as well as source and index mounts.
438
+
286
439
  Polling and startup catch-up preserve each logical path when multiple directory
287
440
  symlinks point to the same source tree. For example, `a_shared/user.rb` and
288
441
  `app/models/user.rb` both remain visible; an earlier alias must not hide the path
@@ -293,15 +446,15 @@ aliases in large watched trees. Ignored logical paths are still pruned.
293
446
 
294
447
  ## Placement
295
448
 
296
- The spike asked for three placements to be compared and one chosen. Every
297
- collaborator on `Woods::Watch::Daemon` is injected, so all three are reachable
298
- from the same class, but the default is **(b), a dedicated daemon per
299
- worktree**:
449
+ Run one dedicated extraction process per active worktree/index. The managed
450
+ Puma adapter and native launcher above retain that separation. Requiring Woods
451
+ or loading its Railtie does not start indexing by itself. Custom integrations
452
+ can use the injected daemon collaborators deliberately:
300
453
 
301
454
  | Option | Verdict |
302
455
  |---|---|
303
456
  | **(b) Dedicated daemon**: *recommended default* | One extra booted app per worktree. Isolated: a crash, a restart, or a storm affects only the index. Lifecycle is drivable from worktree hooks. |
304
- | **(a) Embedded in the dev server** via the Railtie | Marginal memory cost ~0 where a booted app already exists, but couples index freshness to the dev server running and puts extraction on its threads. `Daemon#process` is public precisely so a host can do this deliberately. |
457
+ | **(a) Custom in-process integration** | A host can call public `Daemon#process` deliberately, but must own threading, reloading, and lifecycle. This is not an automatic Railtie feature or the managed Puma adapter. |
305
458
  | **(c) Host watcher + in-container session** | Solves bind-mount event unreliability, but with the most moving parts. Forcing the polling backend solves the same problem with none. |
306
459
 
307
460
  Measured on the fixture app (Ruby 3.3, Rails 8.0):
@@ -652,7 +805,7 @@ hooks (`plugin/hooks/hooks.json`), both shipped disabled:
652
805
  Both read `cwd` from the hook payload, so a linked worktree uses its own index.
653
806
  Both require an existing `generation.json` and `WOODS_HOOKS_ENABLED=1`;
654
807
  `WOODS_HOOKS_DISABLED=1` overrides enablement. The broader refresh task is
655
- **unreleased after Woods 2.0.0.beta2**. Check the installed gem's task list
808
+ **included in Woods `2.0.0`**. Check the installed gem's task list
656
809
  (`bundle exec rake -T woods:hook_refresh`, through the application container
657
810
  when appropriate) before enabling this plugin version. An older gem's unknown
658
811
  task error leaves queued events in place; installing the plugin does not upgrade
@@ -750,8 +903,10 @@ per-process language-server index does not.
750
903
 
751
904
  N resident daemons is N booted apps, and most slots are dormant most of the
752
905
  time. `idle_timeout` (off by default) stops a daemon after that many seconds
753
- without a file event, so a slot nobody is working in stops holding ~65 MB. A
754
- worktree hook or session start revives it.
906
+ without a file event. Revival requires a separately configured supervisor or
907
+ startup hook; the shipped SessionStart hook only checks freshness. Managed
908
+ `woods-watch` rejects this setting because an idle child shutdown would leave
909
+ automatic maintenance inactive until the owner restarts it.
755
910
 
756
911
  ```ruby
757
912
  Woods::Watch::Daemon.new(output_dir: …, idle_timeout: 900).run # 15 minutes
@@ -813,8 +968,8 @@ supplies batches to it.
813
968
 
814
969
  ### Optional bounded context hints
815
970
 
816
- Context hints are a separate Claude Code opt-in, **unreleased after Woods
817
- 2.0.0.beta2**. Verify `bundle exec woods-hook-context --help` in the installed
971
+ Context hints are a separate Claude Code opt-in, **included in Woods `2.0.0`**.
972
+ Verify `bundle exec woods-hook-context --help` in the installed
818
973
  application bundle before enabling `WOODS_HOOK_CONTEXT_ENABLED=1`. The plugin
819
974
  version alone does not establish gem support. `WOODS_HOOKS_DISABLED=1` disables
820
975
  both context and refresh; `WOODS_HOOKS_ENABLED` controls only the existing