woods 2.0.0.beta2 → 2.0.0.beta4

data/docs/TROUBLESHOOTING.md

@@ -8,7 +8,7 @@ This guide covers the most common problems encountered when installing, extracti
 
 | Error message | Cause | Fix |
 |---------------|-------|-----|
-| `No manifest.json found` | Wrong index path or no published generation | Use the path visible to the server process; run `woods:validate` |
+| `Could not resolve a published Woods index` (older versions: `No manifest.json found`) | Wrong index path or unresolved published generation | Select the existing index using a path visible to the server process; see [startup diagnostics](#index-cannot-be-resolved-at-startup) |
 | `uninitialized constant Rails` | Not running inside Rails app | Run via `bundle exec rake` in Rails root |
 | `type "vector" does not exist` | pgvector not installed | `CREATE EXTENSION vector` in PostgreSQL |
 | `Connection refused (localhost:11434)` | Ollama not running | `ollama serve` |
@@ -23,8 +23,9 @@ This guide covers the most common problems encountered when installing, extracti
 | `No such container` | Wrong container name | Check with `docker ps --format '{{.Names}}'` |
 | `JSON parse errors` (MCP) | Rails boot noise on stdout | Remove `puts` calls from initializers |
 | Query timeout | Large table, no scope | Add scope conditions to narrow results |
+| `Extraction failed for …; the previous generation remains active` | A consumer handled a source error during incremental extraction or refresh (unreleased after `2.0.0.beta3`) | Fix the logged source error and retry the [complete batch](INCREMENTAL_EXTRACTION.md#handled-source-errors-and-retry); watch keeps it pending |
 | Empty extraction output | `eager_load!` failure | Check for `NameError` in boot output |
-| Git metadata missing | Shallow clone in CI | Use `fetch-depth: 2` or higher |
+| Git metadata missing | Shallow clone in CI | Use `fetch-depth: 0` for complete history |
 | Parallel tool calls all fail | MCP client batches calls | Send calls sequentially, validate params first |
 | HTTP transport refuses to start on `0.0.0.0` | Missing bearer token | Set `WOODS_MCP_HTTP_TOKEN=…` or bind loopback only |
 | HTTP transport returns `403 Origin not allowed` | Origin header not in allow-list | Set `WOODS_MCP_HTTP_ALLOWED_ORIGINS="https://example.com"` (comma-separated; default is loopback-only) |
@@ -43,8 +44,55 @@ For a single-call health snapshot, call the Index Server's `woods_status` tool.
 
 Agents cold-connecting to a server should call `woods_status` before any other tool, it eliminates most "why is this empty?" guesswork.
 
+If `woods:validate` warns that the manifest writer and reader major versions
+differ, run full extraction using the intended gem and follow the upgrade guide.
+An invalid `woods_version` warns without failing structural validation; a missing
+or null value is normal for older indexes. Compare `index.woods_version` with
+`server.version` in MCP status; never infer an unknown writer from the reader's
+version. See [manifest writer provenance](PUBLISHED_INDEX.md#manifest-writer-provenance).
+
 If a tool call fails with **"Tool not found: … not available in the installed Woods v…"**, the client is asking for a tool a newer gem provides. Run `bundle update woods` and reconnect the MCP server, then retry.
 
+### Semantic graph validation errors
+
+In development versions containing #413, `woods:validate` rejects graphs that
+parse as JSON but disagree with their indexes. Errors name the section and
+identity, for example `reverse["http_api"]: missing "Order"`, a duplicate typed
+variant, or an indexed unit absent from `nodes`. This is unreleased after
+`2.0.0.beta2`; check the installed gem before expecting these diagnostics.
+
+Keep the failing generation and report the exact errors. Run a full extraction
+in a fresh application process with the intended bundle, then validate again.
+Do not edit derived reverse/file/type indexes to silence the check. If a fresh
+full run still fails, report the invariant and source units as an extraction bug.
+The checker never repairs or republishes the index itself.
+
+An unresolved target is legal; missing both a node and its unit cannot be
+classified as external versus accidentally omitted from current metadata alone.
+A green report also does not prove that an indexed relationship executes at
+runtime. See [the checked invariants and limitations](INDEX_LAYOUT.md#semantic-graph-validation).
+
+### Corrupt pipeline cooldown state
+
+In a custom Index MCP server configured with an `operator` and
+`pipeline_guard`, a corrupt `pipeline_guard.json` denies full pipeline runs
+until repaired. The packaged `woods-mcp` executable does not expose pipeline
+operations; confirm the connected server's tools before using this recovery.
+
+For versions containing B-159, call `pipeline_repair` with
+`{"action":"reset_cooldowns"}` to replace malformed JSON, non-object JSON, or an
+empty guard file with an empty state object under the guard's file lock. This
+explicit action clears the extraction and embedding cooldowns; subsequent runs
+can start immediately. The direct Ruby equivalent is `guard.reset!(:all)`.
+Scoped resets leave corrupt state untouched. Valid state keeps any unrelated
+operation entries, and missing state remains a no-op without creating a file.
+A permission failure must be corrected before repair can succeed.
+
+This recovery is unreleased after `2.0.0.beta2`; check the installed version.
+Older versions report corrupt state as nothing to repair. Stop pipeline writers,
+back up the configured guard state's `pipeline_guard.json`, and remove only that
+file before restarting, or upgrade to a version containing the fix.
+
 ## Extraction Problems
 
 ### Extraction produces empty or incomplete output
@@ -171,21 +219,60 @@ bundle exec rake woods:extract
 
 ---
 
+### External dependency targets lose dependents after incremental extraction
+
+**Symptom:** An external target such as `http_api` loses previously indexed
+dependents after an incremental run.
+
+**Cause:** Versions affected by B-193 can split symbolic and string target
+identities when restoring and updating the graph.
+
+**Fix:** Check whether the installed version includes B-193; this fix is
+unreleased. After upgrading to a version containing the fix, run
+`bundle exec rake woods:extract` once to rebuild lost reverse dependencies.
+Loading an already damaged graph does not restore discarded entries. See the
+[incremental graph contract](INCREMENTAL_EXTRACTION.md#the-contract).
+
 ### Git metadata is missing or shows zeros
 
-**Symptom:** Units have `last_modified_at: null` or `change_frequency: 0` in the JSON output.
+**Symptom:** Per-unit `metadata.git` is absent, or an older Woods version reports
+most files as `change_frequency: new` in a shallow CI checkout.
 
-**Cause:** The git repository is a shallow clone (common in CI with `fetch-depth: 1`). Woods uses `git log` to compute change frequency, a shallow clone has no history to analyze.
+**Cause:** A shallow clone truncates HEAD ancestry. The shallow-checkout guard is
+unreleased after 2.0.0.beta2: current source omits git enrichment and warns once,
+rather than treating the truncated history as complete. If repository depth
+cannot be verified, enrichment is also omitted; check git access and version.
 
-**Fix:** Fetch at least two commits:
+**Fix:** Fetch complete history (`git fetch --unshallow` for an existing shallow
+clone), then run full extraction to replace retained metadata:
 
 ```yaml
 # .github/workflows/index.yml
 - uses: actions/checkout@v4
   with:
-    fetch-depth: 2    # minimum for incremental; use 0 for full history
+    fetch-depth: 0
 ```
 
+Two commits can suffice for an incremental diff, but do not establish the full
+ancestry needed for churn metadata.
+
+---
+
+### Git enrichment warns that history could not be read completely
+
+Current source uses an explicit merge-diff mode requiring **Git 2.31 or newer**.
+This is unreleased after 2.0.0.beta2: first confirm the installed Woods version.
+Check `git --version` inside the same container/process environment as extraction,
+and upgrade git if it is older. On a supported version, check that the application's
+`HEAD` and object store can be read using the same `WOODS_GIT_DIR` setting.
+
+A failed or incomplete history stream is discarded as a whole; extraction continues
+without that enrichment, rather than publishing partial or zero-count history.
+Previously retained incremental units can still carry older metadata. After fixing
+git, run a full extraction to refresh every unit. See the
+[history contract](CONFIGURATION_REFERENCE.md#git-enrichment-history) for merge
+counting and upgrade compatibility.
+
 ---
 
 ### Every unit reports `commit_count: 0` and `change_frequency: "new"`
@@ -304,13 +391,17 @@ relative and resolves outside the mount.
 
 ## MCP Server Problems
 
-### "No manifest.json" error when starting the Index Server
+<a id="no-manifestjson-error-when-starting-the-index-server"></a>
 
-**Symptom:** `woods-mcp-start` exits with an error like `No manifest.json found at /path/to/...` even though extraction completed.
+### Index cannot be resolved at startup
 
-**Cause:** The Index Server is using the container-internal path rather than the host-side path to the volume-mounted output. The server runs on the host and cannot access container filesystem paths.
+**Symptom:** An Index MCP executable exits with `Could not resolve a published Woods index in: /path/to/...` even though extraction completed. This headline is unreleased after `2.0.0.beta3`; older versions say `No manifest.json found`. Both mean the selected index could not resolve its manifest, not that an atomic index needs a root manifest.
 
-**Fix:** Use the host path in your `.mcp.json`:
+Embedded Index MCP startup through `IndexReader` also raises an `ArgumentError` with the selected directory and layout guidance when the marker cannot resolve a manifest, including malformed marker shapes such as `[]` or a numeric `payload` (unreleased after `2.0.0.beta3`). Earlier builds may expose a raw `TypeError` or `NoMethodError` for those shapes. Inspect the marker and preserve the failing index before attempting recovery.
+
+**Cause:** The selected directory is not the published index root, the published generation cannot be resolved, or the path is not visible to the MCP process. A container path is appropriate for a container process; a host process needs the host-visible path.
+
+**Fix:** Point at the existing index before extracting again. Check the examined directory in the error and the [MCP path precedence](CONFIGURATION_REFERENCE.md#environment-variables). For a host-side launch whose working directory contains `tmp/woods`, for example:
 
 ```json
 {
@@ -323,20 +414,19 @@ relative and resolves outside the mount.
 }
 ```
 
-Verify the output is accessible from the host:
-
-```bash
-ls ./tmp/woods/manifest.json
-```
-
-**Since Woods 2.0, a healthy index may not have `manifest.json` at the output root at all.** Extraction publishes each generation into an immutable `payloads/gen-<N>/` directory and points to it from `generation.json`. If the flat path is missing, check the payload path instead before assuming extraction failed:
+**Since Woods 2.0, a healthy index may not have `manifest.json` at the output root at all.** Extraction publishes each generation into an immutable `payloads/gen-<N>/` directory and points to it from `generation.json`. Inspect the marker and its payload in the MCP process's filesystem before assuming extraction failed (use the generation named by your marker):
 
 ```bash
 cat ./tmp/woods/generation.json                    # {"number": 42, "payload": "payloads/gen-42", ...}
 ls ./tmp/woods/payloads/gen-42/manifest.json
 ```
 
-`woods-mcp-start` and `IndexReader` already resolve this automatically, this is only for manual inspection. If neither path has a manifest, your Docker volume mount is not configured correctly. See [DOCKER_SETUP.md](DOCKER_SETUP.md).
+Custom scripts that require a root `dependency_graph.json` have the same failure.
+Update their gate using the [filesystem layout contract](INDEX_LAYOUT.md), which
+includes Bash/jq and Python readers. An upload must pin and copy one complete
+payload before publishing its captured pointer; keep a failed copy unpublished.
+
+`woods-mcp-start` and `IndexReader` resolve this automatically; these commands are for manual inspection. Legacy flat indexes use a root `manifest.json`. If neither layout resolves, check the selected path, pointer, payload and any volume mount. See [DOCKER_SETUP.md](DOCKER_SETUP.md) for container launches.
 
 ---
 
@@ -839,3 +929,24 @@ view sharing the same name. Current writers distinguish typed storage identities
 public names remain unchanged. Snapshot migration 007 runs automatically and keeps
 existing rows, but cannot recover variants lost by older writers. See the
 [upgrade guide](UPGRADING_TO_2.md) for storage, flow rebuild, and rollback details.
+
+## Watch exits 75 repeatedly at startup
+
+Check the installed Woods version and its [watch daemon guide](WATCH_DAEMON.md).
+Older releases, including `2.0.0.beta2`, can repeatedly request restart when a
+boot-captured file is newer than the last index generation. Stop the supervisor,
+run one successful `bundle exec rake woods:extract` in the application environment,
+then start the standalone `bundle exec rake woods:watch` process again.
+
+With startup snapshot support, a fresh environment boot performs the full
+reconciliation automatically. Changes during environment initialization or live
+watching still require restart. Do not prepend `environment` to the watch command
+or start it inside an already initialized process when relying on this recovery.
+
+## Source freshness is unknown or drifted
+
+Read `woods_status.index.source_freshness.reasons`. Old indexes, an inaccessible
+source root/private key, a quick scan limit and an unverified boot boundary are
+different causes. Try `source_check: "deep"` for a budget limit; use the fresh
+launcher for a new verified baseline. Do not delete pending hook events or alter
+key permissions just to suppress a warning. See [source freshness](SOURCE_FRESHNESS.md).

data/docs/UNBLOCKED_INTEGRATION.md

@@ -139,6 +139,20 @@ https://github.com/your-org/your-repo/blob/main/app/models/order.rb
 
 This means Unblocked citations link directly to the relevant source code.
 
+Different identifiers defined in one synced file retain the existing URI
+rule: the lexically first identifier keeps the bare URI; siblings use
+`?unit=<encoded identifier>`. Original identifiers and unambiguous URIs do
+not change when two extracted types share a name but have different files.
+
+Two **different types with the same identifier and source file** cannot be
+represented by this URI rule. Woods reports `ambiguous export URI` and skips
+all documents sharing that base URI. It also disables stale-document deletion
+for that run, including with `UNBLOCKED_FORCE_PURGE=1`. Existing remote
+documents remain untouched. This is a deliberate refusal pending an explicit
+remote URI migration; renaming public extraction identifiers is not required.
+Collision detection includes all published types, including excluded types
+and units outside a partial sync's top-N selection.
+
 ## Rate Limits
 
 The Unblocked API allows 1,000 calls per day (resets at midnight PST). A typical
@@ -236,6 +250,17 @@ state an unchanged codebase costs ~0 calls.
 Pair with `woods:incremental` to re-extract only changed files; the sync then
 pushes only the documents whose content actually changed.
 
+Before any API mutation, Woods reads and validates the complete published unit
+set under one generation pin. Full and partial selection use actual unit types;
+the GraphQL family includes its four published subtypes once. Standalone
+`sync_type` and `sync_type_partial` calls perform the same preflight. Missing,
+unreadable, or mismatched identities refuse before uploads, deletion, or
+manifest writes. Preserve the error, validate and regenerate the index, then
+retry. Force flags cannot bypass this check. Preflight reads all published
+unit bodies, so local read cost and temporary memory scale with the index even
+for a single-type sync. Custom readers must provide complete published
+enumeration or complete per-bucket listings plus strict typed lookup.
+
 ### Escape hatches
 
 - `UNBLOCKED_FORCE_FULL_SYNC=1`: re-push every document, ignoring the unchanged

data/docs/UPGRADING_TO_2.md

@@ -5,8 +5,8 @@ Woods 2.0 changes observable index identifiers, publication layout, vector-store
 This guide assumes the last v1 release, 1.6.1, and targets 2.0.0.
 
 <!-- release-state:upgrade-availability -->
-> RubyGems lists 2.0.0.beta2 as a prerelease. Pin it explicitly with
-> `gem "woods", "2.0.0.beta2"`; `~> 2.0` resolves only once
+> This tree declares 2.0.0.beta4 as a prerelease. After RubyGems lists it, pin it with
+> `gem "woods", "2.0.0.beta4"`; `~> 2.0` resolves only once
 > 2.0.0 is published.
 <!-- release-state:end -->
 
@@ -47,6 +47,25 @@ After this runbook you will have:
 
 ## Before changing the bundle
 
+### Check the loader for wrapper-nested classes
+
+File-path-governed naming of classes inside class namespaces requires **Zeitwerk
+mode with Zeitwerk 2.6.9 or later** (`cpath_expected_at`, introduced in
+[Zeitwerk 2.6.9](https://github.com/fxn/zeitwerk/blob/main/CHANGELOG.md#269-25-july-2023)). This is a requirement
+of that naming capability, not a higher Rails minimum. Older Zeitwerk and
+classic-mode applications can still extract ordinary declarations, but Woods
+cannot use the loader to distinguish a file's class from its enclosing class
+wrappers. Two sibling files may then derive the same wrapper identifier and
+extraction will abort rather than silently discard one.
+
+For a `same-type identifier collision`, inspect both named files and the
+application's loader mode/version before rewriting valid namespace wrappers.
+On an older-loader host, move to a compatible Zeitwerk version and Zeitwerk mode,
+verify that the application boots and eager-loads, then run a fresh full
+extraction. Rebuild embeddings and exports if identifiers change. A genuine
+duplicate under a supported loader still needs distinct constants or one source
+file. Woods does not provide a classic-mode naming fallback for this case.
+
 ### 1. Record the current installation
 
 Run in the same environment that boots Rails:
@@ -106,7 +125,7 @@ Pay particular attention to:
 - `output_dir` and environment overrides;
 - storage and embedding provider settings;
 - the configured embedding model/dimension;
-- `console_mcp_enabled`, the `console_mcp_token` secret source, allowed origins, path, and embedded read-tool flags;
+- `console_mcp_enabled`, `console_mcp_http_enabled`, the HTTP `console_mcp_token` secret source, allowed origins, path, and embedded read-tool flags;
 - snapshot, session, Notion, Obsidian, and Unblocked settings;
 - old `config.extractors` or `config.add_gem` calls, which are not implemented selectors.
 
@@ -123,6 +142,10 @@ bin/rails woods:validate
 bin/rails woods:stats
 ```
 
+Unreleased after `2.0.0.beta3`: `woods:clean` removes index artifacts but keeps
+the output directory and its hidden extraction guard. This stable guard lets
+concurrent writers coordinate safely; its presence does not mean an index remains.
+
 The clean extract is required for corrected identifier shapes. Do not use an incremental run as the first v2 extraction: after `woods:clean` there is no baseline, and v2 `woods:incremental` refuses that state rather than publishing a near-empty index as the application's complete truth.
 
 An interrupted extraction leaves readers on the last complete generation because Woods publishes `generation.json` only after the payload is complete. Re-run the task; do not delete a partial directory speculatively. A run that completes its payload but cannot publish the marker now fails loudly instead of reporting success, so treat a non-zero exit as work to redo rather than as a partial success.
@@ -198,6 +221,15 @@ tmp/woods/
 
 Woods tasks, readers, exporters, and MCP servers resolve this automatically. Custom tooling must read `generation.json`, resolve its `payload` relative to the index root, reject paths that escape that root, and then read the payload files. A missing payload key represents the legacy flat layout.
 
+Use the [filesystem layout contract](INDEX_LAYOUT.md) for Bash/jq and Python
+examples. Multi-file reads and uploads must keep the selected payload pinned
+against retention for the complete read/copy; pointer resolution alone does not
+protect a directory from being pruned.
+
+The optional manifest `woods_version` records its last publisher. A matching
+major version after an incremental run does not establish that older units were
+migrated; keep the full re-extraction requirement. See [writer provenance](PUBLISHED_INDEX.md#manifest-writer-provenance).
+
 Graph consumers must also tolerate multiple typed variants for the same textual identifier. Do not collapse nodes by identifier alone when type is part of identity.
 
 The bundle requires patched MessagePack >=1.8.2 and JSON >=2.19.9, <3.
@@ -254,28 +286,22 @@ Structural reads still work from a read-only index mount, but the `reload` tool
 
 ### Console users: preserve or configure the HTTP token
 
-If Console MCP is disabled, no Console token action is required. If it is enabled, configure a token of at least 32 characters through a secret manager or environment variable rather than committing it to the initializer:
-
-```bash
-WOODS_CONSOLE_MCP_TOKEN="$(openssl rand -hex 32)"
-export WOODS_CONSOLE_MCP_TOKEN
-```
-
-```ruby
-config.console_mcp_token = ENV.fetch("WOODS_CONSOLE_MCP_TOKEN")
-```
-
-Persist the generated value in the application's normal secret store before opening a new shell or deploying. Never print, log, or commit the token during an agent-operated upgrade.
-
-Rails-mounted Console HTTP clients must send the token as `Authorization: Bearer <token>`. With Console enabled, a missing token has these outcomes:
+If HTTP Console is enabled, preserve or configure a secret token of at least
+32 characters. Missing tokens warn outside production and HTTP requests fail
+closed with 401; production boot refuses them. Configured short tokens raise
+while HTTP Console is enabled. Keep token values in the application's normal
+secret store, never in committed configuration.
 
-- outside production, Rails warns and the Console HTTP endpoint returns 401;
-- in production, Rails refuses to boot;
-- in every environment, a configured token shorter than 32 characters raises a configuration error.
+Stdio does not use a bearer token. On versions supporting
+`console_mcp_http_enabled`, set it to `false` for stdio-only use without HTTP
+boot validation, while keeping the master `console_mcp_enabled` flag on.
+The HTTP flag defaults to `true` to preserve existing deployments; choosing a
+stdio client alone does not turn HTTP off. Older versions without this flag
+still require a token at production boot whenever Console is enabled.
 
-The stdio Console transport does not send or authenticate with the bearer token. Outside production it can run without one, although Rails still warns because enabling Console also activates the guarded Rack endpoint. In production, Rails boot validation still requires the token even when stdio is the intended transport. The safest default is to configure the token whenever Console is enabled, or leave Console disabled.
+Follow [Console MCP setup](CONSOLE_MCP_SETUP.md) for transport-specific setup
+and the [Configuration reference](CONFIGURATION_REFERENCE.md) for defaults.
 
-See [Console MCP setup](CONSOLE_MCP_SETUP.md) for client examples and [Configuration reference](CONFIGURATION_REFERENCE.md) for the complete security settings.
 
 ## Verify before rollout