woods 2.0.0.beta3 → 2.0.0.beta4

data/docs/MCP_WORKTREE_SETUP.md

@@ -1,127 +1,87 @@
 # MCP Registration in Git Worktrees
 
-> **Claude Code specific.** This page covers Claude Code's MCP registration model (`/mcp`, `~/.claude/plugins/`); other MCP clients manage per-directory registration their own way.
+> **Claude Code specific.** Other MCP clients manage registration and subagent tool access differently. Use the client's current documentation alongside the Woods [MCP setup guide](MCP_SERVERS.md).
 
-When you work in a git worktree, a separate directory checked out from the same repository, your MCP tools may not be available to subagents running in that directory. This page explains why, how to fix it, and how to confirm the registration took effect.
+A git worktree is a separate checkout. For Woods, verify both which servers a session can access and which checkout's index those servers serve.
 
-## Why Worktree Subagents May Not See Woods Tools
+## Separate sessions and inherited subagents
 
-MCP server registration in Claude Code is controlled by `.mcp.json` files. Claude Code discovers these files by walking up the directory tree from the working directory. It stops at the first `.mcp.json` it finds (or at `~/.claude/settings.json` for global registrations).
+A **new Claude Code session launched in a worktree** can have different MCP registrations from a session in the main checkout:
 
-A git worktree has its own root directory separate from the main repository checkout. When a subagent starts inside the worktree root, it walks up from that path, not from the main repository root. Unless a `.mcp.json` exists inside the worktree directory tree or in an ancestor shared with both checkouts, the subagent sees no MCP servers.
+| Registration scope | Location | Availability |
+|---|---|---|
+| Local | Project entry in `~/.claude.json` | The project where the server was registered |
+| Project | `.mcp.json` in the project root | That project, subject to client approval/settings |
+| User | `mcpServers` in `~/.claude.json` | Across the user's projects |
 
-Example directory layout:
+A tracked `.mcp.json` may already be present in another worktree; a local, untracked file will not be copied by Git. User-level registration can make a server available across checkouts, but its configured paths still determine which index it serves.
 
-```
-~/work/my-app/            ← main checkout, has .mcp.json here
-~/work/my-app-feature/    ← worktree, no .mcp.json, so MCP tools are missing
-```
+An **inherited subagent** receives the parent conversation's MCP tools, subject to tool restrictions. Changing the subagent's working directory does not by itself launch another Woods server or switch the served index. Distinguish this from starting an independent client process in that directory.
+
+See Claude Code's [MCP installation scopes](https://code.claude.com/docs/en/mcp#mcp-installation-scopes) and [subagent tool access](https://code.claude.com/docs/en/sub-agents#available-tools). Do not diagnose registration by assuming the client searches ancestor directories for the nearest `.mcp.json`.
 
-A subagent spawned in `~/work/my-app-feature/` will not find the `.mcp.json` from `~/work/my-app/`.
+## Configure a server for the intended checkout
 
-## Fix: Add a `.mcp.json` to the Worktree Root
+First check `/mcp` in the session that needs Woods. If a suitable server is already connected, verify its index before adding a duplicate registration.
 
-Create a `.mcp.json` in the worktree's root directory with the same woods server entries you use in the main checkout:
+For a host with the application's Ruby bundle installed, a project-root `.mcp.json` can select the bundle and index explicitly:
 
 ```json
 {
   "mcpServers": {
     "woods": {
-      "command": "woods-mcp-start",
-      "args": ["./tmp/woods"]
-    },
-    "woods-console": {
-      "command": "docker",
-      "args": [
-        "compose", "exec", "-T", "app",
-        "bundle", "exec", "rake", "woods:console"
-      ],
-      "cwd": "/absolute/host/path/to/worktree"
+      "command": "bundle",
+      "args": ["exec", "woods-mcp-start", "/absolute/path/to/worktree/tmp/woods"],
+      "env": {
+        "BUNDLE_GEMFILE": "/absolute/path/to/worktree/Gemfile"
+      }
     }
   }
 }
 ```
 
-Adjust the paths and arguments to match your project's setup. In particular:
-
-- `./tmp/woods` is a relative path, it resolves against the worktree root, which is correct if extraction output is written into each worktree separately.
-- If you share a single extraction output directory between checkouts, use the absolute path to the shared output: `"/absolute/path/to/main-checkout/tmp/woods"`.
-- For Docker projects, the `docker compose exec` command works the same from any host path.
-
-## How Plugin Discovery Works
-
-Claude Code also discovers MCP servers registered in plugin manifests. A plugin at `~/.claude/plugins/<plugin-name>/admin-tools/.mcp.json` is loaded globally, its servers are available in any session regardless of working directory.
+Replace both paths with the intended Rails worktree. Preserve other server entries. These absolute paths are machine-specific; do not commit them as shared team defaults. Follow the client's approval/reconnection steps, then verify the connection below.
 
-If your team distributes the woods MCP registration through a shared plugin, the worktree problem does not apply. Check whether woods is already registered this way:
+For Docker, use the [container-first setup](DOCKER_SETUP.md#default-run-it-through-the-application-container). Select the intended Compose project and service, and verify that its mounted application directory is the worktree you mean to index. `docker compose exec` targets an existing container; running the command from a different checkout does not change that container's mounts. The index path must be visible inside that container.
 
-```bash
-ls ~/.claude/plugins/
-# look for a directory containing admin-tools/.mcp.json
-cat ~/.claude/plugins/<plugin-name>/admin-tools/.mcp.json
-```
-
-If you find the woods servers registered there, subagents in any worktree will have access automatically, you do not need a per-worktree `.mcp.json`.
-
-## Verifying MCP Registration for a Subagent
+Keep a separate output directory for each checkout when you need checkout-specific answers. If you intentionally point at another checkout's index, Woods serves that published content; registration alone does not make it describe the current worktree.
 
-### Option 1: List tools from a claude session in the worktree
+## Plugin-provided servers
 
-Open a new Claude Code session with the worktree as the working directory and run:
+An enabled plugin can supply MCP server definitions, but installing a plugin does not establish that a Woods server is connected to the intended checkout. Availability depends on the plugin's configuration, enabled scope, and the client's tool restrictions. Check `/mcp` rather than assuming a fixed path under `~/.claude/plugins/` or global availability.
 
-```
-/mcp
-```
+The Woods setup/configuration skills help configure a server; their presence alone is not verification that the Index Server is running. See [agent setup](AGENT_SETUP.md).
 
-This lists all connected MCP servers and their tools. If `woods` and/or `woods-console` appear, registration is working.
+## Verify registration and the served index
 
-### Option 2: Check via the woods status tool
+1. Open `/mcp` in the relevant Claude Code session and inspect the Woods connection and available tools.
+2. Call the Index Server's `woods_status`. Check `index_dir`, generation, and available freshness/provenance information against the intended extraction. A successful connection to the wrong index is still the wrong setup.
+3. Use `search` and typed `lookup` for a known unit from that checkout. When checking a subagent, verify that it can call the inherited tools and is using the same intended index.
 
-Ask Claude to call the status tool:
-
-```
-Use woods-console console_status to check what models are available.
-```
-
-If the tool runs successfully, MCP is registered and the console server is reachable.
-
-### Option 3: Inspect the MCP config that Claude Code loaded
-
-From the worktree directory, run:
-
-```bash
-cat .mcp.json
-```
-
-If the file exists and contains the woods entries, Claude Code will use it. If the file is missing, check parent directories up to your home directory for any `.mcp.json` that would be discovered.
+The optional Console Server is a separate connection to a booted Rails application. If deliberately enabled, check it with `console_status` and verify its application/container separately. Console access is not required to verify the Index Server.
 
 ## Extraction Provenance in Worktrees (`git_branch` / `git_sha`)
 
-`manifest.json` records the `git_branch` and `git_sha` the extraction ran against. In a linked worktree, `.git` is a **file** containing a `gitdir:` pointer to the real git directory, frequently an absolute host path, rather than a `.git` directory.
+The published payload's `manifest.json` records the extraction's `git_branch` and `git_sha`. In a linked worktree, `.git` is a file containing a `gitdir:` pointer, often to an absolute host path. That private worktree directory also refers to the parent repository's shared Git data.
 
-Woods resolves provenance with worktree-aware git plumbing (`git -C <root> rev-parse`), so an ordinary worktree reports the correct branch and SHA. When a `.git` is present but the pointed-to git directory **cannot be resolved**: most commonly a worktree extracted inside a container where the host path isn't mounted. Woods records `git_branch: "unknown"` / `git_sha: "unknown"` rather than a stale, misleading value: a baked `GIT_BRANCH`/`GIT_SHA` build arg is **not** trusted here (it could be stale). The env vars are honored only when there is no `.git` at the root at all (a non-repo checkout, e.g. a Docker `COPY` that excludes `.git`) or git is unavailable.
+Woods uses worktree-aware Git commands. If a present `.git` cannot be resolved, provenance is `"unknown"`; stale `GIT_BRANCH`/`GIT_SHA` values are not substituted. Those environment variables are fallbacks only when the root has no `.git` or Git is unavailable. Temporal snapshots skip an unknown SHA.
 
-To get correct provenance from a containerized worktree, mount the directory the `gitdir:` pointer references (the parent repository's `.git`) into the container so git can resolve it. Temporal snapshots skip an `"unknown"` SHA, so misleading provenance never keys a snapshot.
+For extraction in a container, make the canonical Git directory and the worktree's pointer resolvable there. Mounting only the private worktree Git directory can leave its shared object store unreachable. Follow the [Git provenance troubleshooting guide](TROUBLESHOOTING.md) for mount and `WOODS_GIT_DIR` guidance, and the [published index layout](INDEX_LAYOUT.md) when locating the manifest.
 
 ## Troubleshooting
 
-**"Unknown tool" or "no MCP server named woods"**
-
-The woods MCP server is not registered for this session. Add a `.mcp.json` to the worktree root as shown above, then restart the session.
+**Woods or an expected tool is absent**
 
-**Console server starts but returns "unsupported: Not yet implemented in embedded mode" for console_sql / console_query**
+Check `/mcp` for connection errors, enabled registration scopes, and project approvals. For a subagent, also check its tool restrictions. Compare the requested tool with the [supported tool surface](MCP_SERVERS.md#conditional-index-capabilities); some tools require optional collaborators and are not registered by the packaged server. Do not add duplicate registrations before establishing which case applies.
 
-The console server is registered and reachable, but `embedded_read_tools` is disabled (the default). To enable console_sql and console_query in embedded mode, see the [Console MCP Setup guide](CONSOLE_MCP_SETUP.md), specifically the `embedded_read_tools: true` option for the Rack middleware.
+**The server connects but shows another checkout**
 
-**Extraction output is empty or stale in the worktree**
+Inspect its configured bundle, index path, Compose project/service, and container mounts. Re-extract from the intended Rails checkout into its own output directory, then reconnect to that index and repeat `woods_status`. See [Docker setup](DOCKER_SETUP.md).
 
-Extraction writes to `tmp/woods/` relative to the Rails application root (inside the container). If the worktree's volume mount points to a different host path than the main checkout, run extraction again from the worktree:
-
-```bash
-docker compose exec app bundle exec rake woods:extract
-```
+**Two registrations use the same server name**
 
-See [DOCKER_SETUP.md](DOCKER_SETUP.md) for the full Docker workflow.
+Check the client's [scope precedence](https://code.claude.com/docs/en/mcp#scope-hierarchy-and-precedence) and the effective connection shown by `/mcp`. Do not assume the two definitions merge or that a project file overrides every other scope.
 
-**Worktree `.mcp.json` conflicts with main checkout `.mcp.json`**
+**Console SQL/query tools are absent or unsupported**
 
-Each file is independent. Claude Code loads the one closest to the working directory. There is no inheritance or merging between them. Keep both files in sync manually, or move the shared configuration into a global plugin manifest.
+Console read-tool availability is configured separately from registration. Follow [Console MCP setup](CONSOLE_MCP_SETUP.md) for the supported 9-tool default and optional 11-tool mode.

data/docs/PUBLISHED_INDEX.md

@@ -109,6 +109,23 @@ The reader wraps `Woods::MCP::IndexReader` with `auto_refresh: false`; the unit
 
 `Woods::MCP::IndexReader#find_unit` keys its identifier map on identifier alone. If two type directories both list the same identifier (a model and a service both named `Foo`, for example), whichever type sorts last in `Woods::MCP::IndexReader::TYPE_DIRS` silently wins, and `unit(identifier)` returns that one. Pass `type:` to read a specific type's unit file directly and skip the collision entirely; `#table_database_map` always does this internally (`type: 'model'`), so a same-named non-model unit can never shadow a model's `table_name`/`database`.
 
+### Actual unit types and directory families
+
+Unreleased after `2.0.0.beta3`: `unit` and `units` accept actual published
+`graphql_type`, `graphql_mutation`, `graphql_resolver`, `graphql_query`, and
+`gem_source` types. Enumeration preserves each unit's actual type rather than
+labeling every GraphQL member `graphql` or every gem source `rails_source`.
+Check the loaded Git revision when testing a development checkout.
+
+The existing `graphql` and `rails_source` filters remain directory-family
+aliases: `graphql` selects all four GraphQL types; `rails_source` selects both
+Rails and gem sources. Returned records keep their actual type. Use
+`units(type: 'gem_source')` for gem sources only; to select Rails sources only,
+filter `units(type: 'rails_source')` on each entry's `"type" == "rails_source"`.
+Unknown types return no records, and an explicit GraphQL subtype never matches
+another subtype. Enumeration continues to return index-entry fields, not full
+unit source bodies.
+
 ### `available_generations`: published means published
 
 A generation is listed only when both hold:

data/docs/RETRIEVAL_GUIDE.md

@@ -63,7 +63,7 @@ Keyword results are scored by how many distinct fields matched (identifier, sour
 
 ## Configuring Retrieval
 
-Retrieval requires an embedding provider and a vector store. Set these in `config/initializers/woods.rb`.
+Semantic retrieval requires an embedding provider and a vector store. Configure these in `config/initializers/woods.rb` before embedding. For provider-free ranked retrieval over extraction output, use [explicit lexical mode](#embedding-free-lexical-retrieval) in the MCP process environment instead.
 
 ### Presets (recommended)
 
@@ -180,7 +180,12 @@ WOODS_RETRIEVAL_MODE=lexical bundle exec woods-mcp-start ./tmp/woods
 For a Ruby-built retriever, set `config.retrieval_mode = :lexical` and supply a
 populated metadata store to `Builder#build_retriever`. The packaged MCP server
 loads published unit JSON itself; it does not boot Rails or read current source
-files. Neither path constructs an embedding provider or vector adapter. The
+files. Neither path constructs an embedding provider or vector adapter. If a
+no-provider error suggests only OpenAI, Ollama or `search`, explicit lexical
+mode is still available from `2.0.0.beta3`: set the variable in the MCP client
+configuration and restart that server. Confirm `woods_status.retriever.mode`
+is `lexical`; setting it only in a Rails initializer does not configure a
+separate MCP process. The
 existing `:semantic` mode remains the default; provider failures never switch
 modes automatically. A Rails initializer is not loaded by the standalone MCP
 process, so set the environment variable in that process's client configuration.
@@ -191,15 +196,25 @@ and validations). Exact full identifiers rank first, with ambiguous typed owners
 retained. Other ties are deterministic. Responses name the lexical mode and
 matching fields/terms; runtime-field hits include the selected published runtime
 values. Lexical Ruby results leave the semantic-only `type_rank_context` table
-`nil`; they do not report a global vector rank or vector fallback. The top 20 eligible positive matches are considered for the
-output budget; this is ranked discovery, not an exhaustive match listing. Explicit
+`nil`; they do not report a global vector rank or vector fallback. The top 20
+eligible positive matches form the candidate shortlist for the output budget.
+The lexical header reports `sources included`, `candidates considered`, and
+`candidate limit: 20`. Included sources count the actual returned entries;
+considered candidates count the shortlist after filtering and the limit, not
+all matches or all documents examined. This is ranked discovery, not an
+exhaustive match listing. Explicit
 `types` filters override default exclusions, as in semantic retrieval, and apply
 before that limit. A query with no lexical evidence returns no matches; unrelated
 graph hubs are never added. Query-seeded graph ranking is evaluation-only.
 
 The budget covers headers, matching explanations and truncation notices using a
 labelled character-based estimate, not an exact provider tokenizer. Full source
-remains available through `lookup`. Very small budgets can omit all sources. The
+remains available through `lookup`. Count text is charged before source selection;
+final counts do not trigger a second selection pass. Very small budgets can omit
+all sources or clip the header itself. Zero candidates means no lexical matches;
+positive candidates with zero included sources means no source entry fit the
+available budget. The same counts apply to full, compact, outline and scoped
+retrieval, agreeing with returned source attribution. The
 reader pins one published generation for building and querying its immutable
 lexical snapshot, rebuilding after publication. Corrupt units fail explicitly;
 they cannot quietly become a successful partial index. Older flat indexes rebuild
@@ -353,6 +368,10 @@ bundle exec rake woods:embed
 | `text-embedding-3-small` (default) | 1536 |
 | `text-embedding-3-large` | 3072 |
 
+Woods' pgvector HNSW adapter supports at most 2,000 dimensions. For the large
+model, request a supported output width explicitly or choose another backend;
+see [pgvector configuration](CONFIGURATION_REFERENCE.md#pgvector-postgresql).
+
 **Ollama default model:** `nomic-embed-text`. Dimensions are detected dynamically on first embed.
 
 ---

data/docs/TROUBLESHOOTING.md

@@ -8,7 +8,7 @@ This guide covers the most common problems encountered when installing, extracti
 
 | Error message | Cause | Fix |
 |---------------|-------|-----|
-| `No manifest.json found` | Wrong index path or no published generation | Use the path visible to the server process; run `woods:validate` |
+| `Could not resolve a published Woods index` (older versions: `No manifest.json found`) | Wrong index path or unresolved published generation | Select the existing index using a path visible to the server process; see [startup diagnostics](#index-cannot-be-resolved-at-startup) |
 | `uninitialized constant Rails` | Not running inside Rails app | Run via `bundle exec rake` in Rails root |
 | `type "vector" does not exist` | pgvector not installed | `CREATE EXTENSION vector` in PostgreSQL |
 | `Connection refused (localhost:11434)` | Ollama not running | `ollama serve` |
@@ -23,6 +23,7 @@ This guide covers the most common problems encountered when installing, extracti
 | `No such container` | Wrong container name | Check with `docker ps --format '{{.Names}}'` |
 | `JSON parse errors` (MCP) | Rails boot noise on stdout | Remove `puts` calls from initializers |
 | Query timeout | Large table, no scope | Add scope conditions to narrow results |
+| `Extraction failed for …; the previous generation remains active` | A consumer handled a source error during incremental extraction or refresh (unreleased after `2.0.0.beta3`) | Fix the logged source error and retry the [complete batch](INCREMENTAL_EXTRACTION.md#handled-source-errors-and-retry); watch keeps it pending |
 | Empty extraction output | `eager_load!` failure | Check for `NameError` in boot output |
 | Git metadata missing | Shallow clone in CI | Use `fetch-depth: 0` for complete history |
 | Parallel tool calls all fail | MCP client batches calls | Send calls sequentially, validate params first |
@@ -390,13 +391,17 @@ relative and resolves outside the mount.
 
 ## MCP Server Problems
 
-### "No manifest.json" error when starting the Index Server
+<a id="no-manifestjson-error-when-starting-the-index-server"></a>
 
-**Symptom:** `woods-mcp-start` exits with an error like `No manifest.json found at /path/to/...` even though extraction completed.
+### Index cannot be resolved at startup
 
-**Cause:** The Index Server is using the container-internal path rather than the host-side path to the volume-mounted output. The server runs on the host and cannot access container filesystem paths.
+**Symptom:** An Index MCP executable exits with `Could not resolve a published Woods index in: /path/to/...` even though extraction completed. This headline is unreleased after `2.0.0.beta3`; older versions say `No manifest.json found`. Both mean the selected index could not resolve its manifest, not that an atomic index needs a root manifest.
 
-**Fix:** Use the host path in your `.mcp.json`:
+Embedded Index MCP startup through `IndexReader` also raises an `ArgumentError` with the selected directory and layout guidance when the marker cannot resolve a manifest, including malformed marker shapes such as `[]` or a numeric `payload` (unreleased after `2.0.0.beta3`). Earlier builds may expose a raw `TypeError` or `NoMethodError` for those shapes. Inspect the marker and preserve the failing index before attempting recovery.
+
+**Cause:** The selected directory is not the published index root, the published generation cannot be resolved, or the path is not visible to the MCP process. A container path is appropriate for a container process; a host process needs the host-visible path.
+
+**Fix:** Point at the existing index before extracting again. Check the examined directory in the error and the [MCP path precedence](CONFIGURATION_REFERENCE.md#environment-variables). For a host-side launch whose working directory contains `tmp/woods`, for example:
 
 ```json
 {
@@ -409,13 +414,7 @@ relative and resolves outside the mount.
 }
 ```
 
-Verify the output is accessible from the host:
-
-```bash
-ls ./tmp/woods/manifest.json
-```
-
-**Since Woods 2.0, a healthy index may not have `manifest.json` at the output root at all.** Extraction publishes each generation into an immutable `payloads/gen-<N>/` directory and points to it from `generation.json`. If the flat path is missing, check the payload path instead before assuming extraction failed:
+**Since Woods 2.0, a healthy index may not have `manifest.json` at the output root at all.** Extraction publishes each generation into an immutable `payloads/gen-<N>/` directory and points to it from `generation.json`. Inspect the marker and its payload in the MCP process's filesystem before assuming extraction failed (use the generation named by your marker):
 
 ```bash
 cat ./tmp/woods/generation.json                    # {"number": 42, "payload": "payloads/gen-42", ...}
@@ -427,7 +426,7 @@ Update their gate using the [filesystem layout contract](INDEX_LAYOUT.md), which
 includes Bash/jq and Python readers. An upload must pin and copy one complete
 payload before publishing its captured pointer; keep a failed copy unpublished.
 
-`woods-mcp-start` and `IndexReader` already resolve this automatically, this is only for manual inspection. If neither path has a manifest, your Docker volume mount is not configured correctly. See [DOCKER_SETUP.md](DOCKER_SETUP.md).
+`woods-mcp-start` and `IndexReader` resolve this automatically; these commands are for manual inspection. Legacy flat indexes use a root `manifest.json`. If neither layout resolves, check the selected path, pointer, payload and any volume mount. See [DOCKER_SETUP.md](DOCKER_SETUP.md) for container launches.
 
 ---
 

data/docs/UPGRADING_TO_2.md

@@ -5,8 +5,8 @@ Woods 2.0 changes observable index identifiers, publication layout, vector-store
 This guide assumes the last v1 release, 1.6.1, and targets 2.0.0.
 
 <!-- release-state:upgrade-availability -->
-> RubyGems lists 2.0.0.beta3 as a prerelease. Pin it explicitly with
-> `gem "woods", "2.0.0.beta3"`; `~> 2.0` resolves only once
+> This tree declares 2.0.0.beta4 as a prerelease. After RubyGems lists it, pin it with
+> `gem "woods", "2.0.0.beta4"`; `~> 2.0` resolves only once
 > 2.0.0 is published.
 <!-- release-state:end -->
 
@@ -142,6 +142,10 @@ bin/rails woods:validate
 bin/rails woods:stats
 ```
 
+Unreleased after `2.0.0.beta3`: `woods:clean` removes index artifacts but keeps
+the output directory and its hidden extraction guard. This stable guard lets
+concurrent writers coordinate safely; its presence does not mean an index remains.
+
 The clean extract is required for corrected identifier shapes. Do not use an incremental run as the first v2 extraction: after `woods:clean` there is no baseline, and v2 `woods:incremental` refuses that state rather than publishing a near-empty index as the application's complete truth.
 
 An interrupted extraction leaves readers on the last complete generation because Woods publishes `generation.json` only after the payload is complete. Re-run the task; do not delete a partial directory speculatively. A run that completes its payload but cannot publish the marker now fails loudly instead of reporting success, so treat a non-zero exit as work to redo rather than as a partial success.

data/docs/WATCH_DAEMON.md

@@ -96,7 +96,7 @@ partial write:
 
 | Failure | What happens |
 |---|---|
-| Reload raises (`SyntaxError`, `NameError`) | Degraded status naming the reason; index intact at generation N; retried on the next event |
+| Reload raises (`SyntaxError`, `NameError`) | Degraded status naming the reason; index intact at generation N; pending paths retried on the next file event or heartbeat |
 | Extraction raises | Degraded status; generation not advanced |
 | Payload directory can't be opened, over a payload-born index | Degraded status; generation not advanced. An incremental run only writes the units it touched, so there is no complete flat index it could fall back to publishing, see [Payload publishing](#payload-publishing) |
 | Index written but the generation bump failed | Degraded status; paths carried forward. The extractor deliberately does not fail an otherwise-good extraction over an unwritable marker, but the marker *is* what readers refresh on, so the daemon cross-checks that the number moved rather than reporting `running` over an index nothing can see |
@@ -119,17 +119,23 @@ at a known generation, reason attached), `stopped` (nothing is maintaining this
 index). A stale answer is only dangerous when nothing says so.
 
 The file is written world-readable (0644) by design: host-side hooks read it
-through a bind mount. Every other artifact Woods writes stays at 0600.
+through a bind mount. Writes through `Woods::AtomicFile` default to owner-only
+0600 unless the caller supplies another mode. This is not a guarantee for every
+Woods artifact: the SQLite metadata store does not enforce 0600, and a newly
+created database uses 0644 under umask 022. Restrict access to the output
+directory according to the source and metadata it contains.
 
 Note that `SyntaxError` is a `ScriptError`, not a `StandardError`. Rescuing
 only the latter would let a half-typed file kill the daemon.
 
 A cycle that fails to land its work never loses its paths. Lock contention, a
 failed reload, and a raising extraction all carry the batch into `@pending`, and
-the next cycle folds it back in, the files really did change, and no later
-event will mention them again. The retry is not a tight loop: a degraded cycle
-ends the drain and waits for the next event, because the cause needs an edit to
-clear.
+the next cycle folds it back in even if no new event mentions those files.
+A degraded cycle ends the current drain to avoid a tight retry loop. Pending
+paths are retried on the next file event or [heartbeat](#the-heartbeat), so a
+finished contending writer does not require another edit to trigger recovery.
+Heartbeat retries use a separate worker so status updates and lock refresh
+continue while extraction runs.
 
 ### The heartbeat
 
@@ -163,6 +169,9 @@ is alive, so *alive has to mean covered*.
 The standalone `woods:watch` task snapshots reload/restart inputs before invoking
 Rails' `environment` task. Inputs unchanged across that boundary, including
 carried paths that remain deleted, may be reconciled by a full extraction.
+Unreleased after `2.0.0.beta3`: registered restart inputs deleted while the
+daemon was stopped also trigger a full extraction after a fresh environment
+boot. Nominal framework paths still use the bounded deletion sweep.
 Changes during environment initialization still require restart. Lock contention,
 extraction failure, and publication failure retain the full-reconciliation
 obligation for retry; a successful publish clears it.
@@ -202,8 +211,9 @@ already tolerate the duplicate paths this produces against whatever catch-up
 finds on its own via the tree scan.
 
 Deletions need one extra step, because a deleted file leaves no mtime to scan:
-if any path the index attributes a unit to is gone from disk, the daemon runs
-one cycle with an *empty* change set, which reaches the ghost units through the
+registered restart inputs follow the full-reconciliation rule above. For other
+registered paths gone from disk, a deletion-only startup runs one cycle with an
+*empty* change set, which reaches the ghost units through the
 extractor's bounded deletion sweep. Deliberately empty, naming the paths would
 make the deletions authoritative for every unit type, and some registered paths
 are nominal (on Rails < 7.1, `ActiveRecord::SchemaMigration` registers a

data/exe/woods-mcp-start

@@ -7,7 +7,7 @@
 # same as when launched directly. RubyGems loads gem executables as Ruby, so
 # this wrapper must remain a Ruby program when packaged.
 
-index_dir = ARGV[0] || ENV.fetch('WOODS_DIR', nil)
+index_dir = ARGV[0] || ENV.fetch('WOODS_DIR', nil) || ENV.fetch('WOODS_OUTPUT', nil)
 
 if index_dir.nil? || index_dir.empty?
   warn 'Error: No index directory specified.'
@@ -15,9 +15,13 @@ if index_dir.nil? || index_dir.empty?
   exit 1
 end
 
+index_dir = File.expand_path(index_dir)
+path_remedy = 'Point at the existing index with an explicit path, WOODS_DIR, or WOODS_OUTPUT. ' \
+              'If no index exists, run `bundle exec rake woods:extract` in your Rails app.'
+
 unless File.directory?(index_dir)
   warn "Error: Index directory does not exist: #{index_dir}"
-  warn 'Run extraction first: bundle exec rake woods:extract'
+  warn path_remedy
   exit 1
 end
 
@@ -27,7 +31,7 @@ end
 # whole library just to check one file is wasted work on every boot. A
 # payload-born index has no manifest.json at the root — it lives under the
 # directory generation.json's `payload` pointer names — so the pointer is
-# followed here too, with the same escape guard, before concluding the
+# followed here too, with the same realpath containment check, before concluding the
 # directory holds no index. woods-mcp re-checks this properly through
 # Bootstrapper regardless; this is just an early, friendlier exit.
 def manifest_present?(index_dir)
@@ -39,20 +43,21 @@ def manifest_present?(index_dir)
   require 'json'
   require_relative '../lib/woods/atomic_file'
   payload_name = JSON.parse(Woods::AtomicFile.read(generation_path))['payload']
-  return false if payload_name.nil? || payload_name.empty?
+  return false unless payload_name.is_a?(String) && !payload_name.empty?
 
-  root = File.expand_path(index_dir)
-  candidate = File.expand_path(File.join(root, payload_name))
+  root = File.realpath(index_dir)
+  candidate = File.realpath(payload_name, root)
   return false unless candidate.start_with?("#{root}#{File::SEPARATOR}")
 
   File.file?(File.join(candidate, 'manifest.json'))
-rescue JSON::ParserError, SystemCallError
+rescue JSON::ParserError, SystemCallError, TypeError, NoMethodError
   false
 end
 
 unless manifest_present?(index_dir)
-  warn "Error: No manifest.json in: #{index_dir}"
-  warn 'Run extraction first: bundle exec rake woods:extract'
+  warn "Error: Could not resolve a published Woods index in: #{index_dir}"
+  warn 'Expected generation.json pointing to a payload manifest.json, or a legacy flat manifest.json.'
+  warn path_remedy
   exit 1
 end
 

data/lib/generators/woods/pgvector_generator.rb

@@ -2,6 +2,7 @@
 
 require 'rails/generators'
 require 'rails/generators/active_record'
+require 'woods/storage/pgvector'
 
 module Woods
   module Generators
@@ -15,7 +16,7 @@ module Woods
     #
     # Usage:
     #   rails generate woods:pgvector
-    #   rails generate woods:pgvector --dimensions 3072
+    #   rails generate woods:pgvector --dimensions 768
     #
     class PgvectorGenerator < Rails::Generators::Base
       include ActiveRecord::Generators::Migration
@@ -25,11 +26,16 @@ module Woods
       desc 'Creates the woods_vectors table (pgvector column + HNSW index) used by the Woods vector store'
 
       class_option :dimensions, type: :numeric, default: 1536,
-                                desc: 'Vector dimensions (1536 for text-embedding-3-small, 3072 for large)'
+                                desc: 'Vector dimensions (1-2000; default 1536 for text-embedding-3-small)'
 
       # @return [void]
       def create_migration_file
         @dimensions = options[:dimensions]
+        maximum = Woods::Storage::VectorStore::Pgvector::MAX_HNSW_DIMENSIONS
+        unless @dimensions.is_a?(Integer) && @dimensions.between?(1, maximum)
+          raise ArgumentError, "dimensions must be a positive Integer no greater than #{maximum} for pgvector HNSW"
+        end
+
         migration_template(
           'add_pgvector_to_woods.rb.erb',
           'db/migrate/add_pgvector_to_woods.rb'

data/lib/woods/agent_configuration/applier.rb

@@ -50,8 +50,10 @@ module Woods
         "#{@layout.receipt_path}.pending"
       end
 
-      def with_lock
-        path = "#{@layout.receipt_path}.lock"
+      def with_lock(paths = @layout.lock_paths, &operation)
+        return operation.call if paths.empty?
+
+        path, *remaining = paths
         Document.validate_path!(path)
         FileUtils.mkdir_p(File.dirname(path), mode: 0o700)
         File.open(path, File::RDWR | File::CREAT | File::NOFOLLOW | File::NONBLOCK, 0o600) do |lock|
@@ -61,7 +63,7 @@ module Woods
                   'Another Woods configuration operation is active; retry after it finishes'
           end
 
-          yield
+          with_lock(remaining, &operation)
         ensure
           lock&.flock(File::LOCK_UN)
         end

data/lib/woods/agent_configuration/cli.rb

@@ -78,12 +78,12 @@ module Woods
         write_plan(path, plan, layout)
         PlanDiff.show(plan, @stderr) if options[:diff]
         plan.summary.merge('plan_file' => File.expand_path(path),
-                           'runtime_files' => ["#{layout.receipt_path}.lock", "#{layout.receipt_path}.pending"])
+                           'runtime_files' => layout.runtime_paths)
       end
 
       def write_plan(path, plan, layout)
         target = File.expand_path(path)
-        protected_paths = layout.allowed_paths + ["#{layout.receipt_path}.lock", "#{layout.receipt_path}.pending"]
+        protected_paths = layout.allowed_paths + layout.runtime_paths
         raise Conflict, 'Plan output must differ from every managed/runtime target' if protected_paths.include?(target)
 
         Document.validate_path!(target)

data/lib/woods/agent_configuration/layout.rb

@@ -50,6 +50,19 @@ module Woods
         end
       end
 
+      # Lock actual managed targets, since user-scoped paths can be shared
+      # by different applications. Keep the receipt lock name for existing
+      # same-application callers; receipts and recovery journals stay separate.
+      def lock_paths
+        allowed_paths.map do |path|
+          path == receipt_path ? "#{path}.lock" : "#{path}.woods.lock"
+        end.sort
+      end
+
+      def runtime_paths
+        lock_paths + ["#{receipt_path}.pending"]
+      end
+
       def identity
         { 'client' => 'claude', 'scope' => scope, 'root' => root, 'config_dir' => config_dir,
           'config_path' => config_path, 'receipt_path' => receipt_path }

data/lib/woods/console/credential_scanner.rb

@@ -145,9 +145,9 @@ module Woods
 
       # Scan a value (String, Hash, Array, or any other object) for credentials.
       #
-      # Strings are gsub'd against every active pattern. Hash values and Array
-      # elements are walked recursively; keys and non-string scalars
-      # (Integer, Float, true/false, nil) pass through untouched.
+      # Strings and Symbols are scanned against every active pattern. Hash
+      # keys/values and Array elements are walked recursively; numeric values,
+      # booleans, and nil pass through untouched. Symbols retain their type.
       #
       # @param value [Object]
       # @return [Array(Object, Hash{Symbol=>Integer})] two-tuple of the scanned
@@ -164,6 +164,7 @@ module Woods
       def walk(value, counts, index)
         case value
         when String then scan_string(value, counts, index)
+        when Symbol then scan_string(value.to_s, counts, index).to_sym
         when Hash   then walk_hash(value, counts, index)
         when Array  then value.map { |item| walk(item, counts, index) }
         else value

data/lib/woods/console/dispatch_pipeline.rb

@@ -103,7 +103,14 @@ module Woods
         response = @conn_mgr.send_request(request)
         return error_from_response(response, request) unless response['ok']
 
+        # Materialize the wire representation before policy checks. Otherwise
+        # Symbols and custom serializers can introduce unscanned strings later.
+        # Redact first so protected serializers never run; redact again after
+        # normalization to cover fields introduced by a custom serializer.
+        # Both renderers consume this same JSON-compatible data tree.
         result = @ctx.redact(response['result'])
+        result = JSON.parse(JSON.generate(result))
+        result = @ctx.redact(result)
         result = scan_for_credentials(result, request)
         text = @renderer ? @renderer.render_default(result) : JSON.pretty_generate(result)
         success_response(text)