woods 2.0.1 → 2.1.0

data/docs/MCP_TOOL_COOKBOOK.md

@@ -58,11 +58,11 @@ The Index Server defines **29 schemas**: the packaged executable registers **14*
 | Tool group | Count | Wiring condition |
 |------------|-------|------------------|
 | Always-on | 14 | Always registered, `lookup`, `search`, `dependencies`, `dependents`, `structure`, `graph_analysis`, `domain_clusters`, `pagerank`, `framework`, `recent_changes`, `reload`, `codebase_retrieve`, `trace_flow`, `woods_status` |
-| `session_trace` | 1 | `Woods.configuration.session_store` set and session tracer enabled |
+| `session_trace` | 1 | Custom/embedded Index process has a configured `session_store` supporting `read` and `sessions`; the packaged executable does not load Rails initializer configuration |
 | Operator (5) | 5 | Custom embedded server wires an operator: `pipeline_extract`, `pipeline_embed`, `pipeline_status`, `pipeline_diagnose`, `pipeline_repair` |
 | Feedback (4) | 4 | Custom embedded server wires a feedback store: `retrieval_rate`, `retrieval_report_gap`, `retrieval_explain`, `retrieval_suggest` |
-| Snapshot (4) | 4 | Extraction with `enable_snapshots = true` normally creates `woods.sqlite3`, which packaged servers discover. If extraction used the JSON fallback, set `WOODS_SNAPSHOTS=true` on the standalone server. Custom embedded servers pass `snapshot_store:`. Internal SQLite migrations are automatic. Tools: `list_snapshots`, `snapshot_diff`, `unit_history`, `snapshot_detail` |
-| `notion_sync` | 1 | `notion_api_token` + `notion_database_ids` both set |
+| Snapshot (4) | 4 | Extraction with `enable_snapshots = true` normally creates `woods.sqlite3`, which packaged servers discover. `WOODS_SNAPSHOTS=true` enables construction, preferring SQLite; it does not force JSON or import JSON history. Custom builders can pass an explicit JSON `snapshot_store:`; see [store selection](MCP_SERVERS.md#conditional-index-capabilities). Internal SQLite migrations are automatic. Tools: `list_snapshots`, `snapshot_diff`, `unit_history`, `snapshot_detail` |
+| `notion_sync` | 1 | Token and `notion_database_ids` configured inside a custom/embedded Index process; use `bin/rails woods:notion_sync` for ordinary application export |
 
 `codebase_retrieve` is always registered (no `retrieve` alias exists). Default semantic mode requires an embedding provider and a completed `woods:embed` run. Explicit `WOODS_RETRIEVAL_MODE=lexical` ranks published extraction units without a provider or embeddings; set it in the MCP process environment and restart the server. See [embedding-free lexical retrieval](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval).
 
@@ -887,6 +887,12 @@ Trigger extraction, then reload the server's in-memory data:
 
 **What you'll get:** Lists of added, modified, and deleted units between the two git SHAs. Use `list_snapshots` first to find valid SHA values.
 
+**Included in Woods 2.1 (#593):** use the exact stored SHAs returned by
+`list_snapshots`; prefixes are not expanded automatically. `snapshot_diff`
+returns `not_found` if either snapshot is absent and `invalid_params` for a
+malformed SHA. Two available snapshots with no differences still return a
+successful empty diff. These checks apply to both JSON and SQLite stores.
+
 ---
 
 ### "How has the User model evolved?"
@@ -908,58 +914,10 @@ Trigger extraction, then reload the server's in-memory data:
 
 ### GitHub Actions for Incremental Extraction
 
-Run incremental extraction on every push, cache the index between runs:
-
-```yaml
-# .github/workflows/woods.yml
-name: Update Codebase Index
-
-on:
-  push:
-    branches: [main]
-  pull_request:
-
-jobs:
-  index:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 2   # needed for incremental diff
-
-      - name: Set up Ruby
-        uses: ruby/setup-ruby@v1
-        with:
-          bundler-cache: true
-
-      - name: Restore index cache
-        uses: actions/cache@v4
-        with:
-          path: tmp/woods
-          key: woods-${{ github.ref }}-${{ github.sha }}
-          restore-keys: |
-            woods-${{ github.ref }}-
-            woods-
-
-      - name: Run database migrations
-        run: bundle exec rails db:migrate RAILS_ENV=test
-
-      - name: Update codebase index
-        run: bundle exec rake woods:incremental
-        env:
-          RAILS_ENV: test
-          GITHUB_BASE_REF: ${{ github.base_ref }}
-
-      - name: Validate index
-        run: bundle exec rake woods:validate
-```
-
-For Docker-based CI:
-
-```yaml
-      - name: Update codebase index
-        run: docker compose exec -T app bundle exec rake woods:incremental
-```
+Use the [canonical incremental CI recipe](INCREMENTAL_EXTRACTION.md#github-actions-with-an-exact-baseline).
+It fetches the actual base ref, restores an index for the exact baseline commit,
+and runs full extraction when that cache is absent. The same guide covers
+nested Rails applications and forwarding the selected range into Docker.
 
 ---
 

data/docs/NOTION_INTEGRATION.md

@@ -171,7 +171,13 @@ steps:
 
 ### MCP Server
 
-If using the MCP Index Server, the `notion_sync` tool is available:
+The packaged Index Server does not load Rails initializers and does not register
+`notion_sync` merely because the application has Notion configured. Use
+`bin/rails woods:notion_sync` in the Rails application for the standard workflow.
+
+A custom/embedded Index Server can register the tool when the token and database
+IDs are configured in that server process. Confirm it appears in `tools/list`
+before calling it:
 
 ```json
 {

data/docs/PUBLISHED_INDEX.md

@@ -249,6 +249,12 @@ Retention is `WOODS_PAYLOAD_RETENTION` generations (default 3). Raise it on a CI
 
 `woods:check:moved_messages` opens two retained generations through `Woods::PublishedIndex` (block form, so both readers' retention locks always release) and reports every public method name that looks like it moved from one unit to another while a `:test_coverage` edge did not follow.
 
+The task does not boot Rails. Pass `WOODS_OUTPUT` for a custom index; an explicit
+relative value is relative to the command's working directory. Without it, the
+Woods 2.1 #591 fix selects `tmp/woods` beside the loaded Rakefile, including
+`rake -f /app/Rakefile` from another directory. Earlier builds used the invoking
+directory instead; pass an absolute output path on those builds.
+
 Each row is a **candidate move into a unit without mapped tests**, never a proven coverage loss: the check matches on method name and kind alone, so two unrelated methods that happen to share both look identical to a real move. Reported only when the source unit was covered before and the destination is not covered after; a method that was never covered is not a regression this check owns. `WOODS_CHECK_STRICT=1` exits 1 on any finding, but the check stays heuristic either way, strict mode changes the exit code, not the confidence of a row.
 
 ```bash

data/docs/README.md

@@ -69,7 +69,12 @@ and stdio or Streamable HTTP endpoints directly.
 
 ## Maintainer material
 
-Historical build-phase documents are not user guides. Source checkouts also contain the [MCP protocol decision record](https://github.com/lost-in-the/woods/blob/main/docs/design/MCP_2026_STRATEGY.md), [generated self-analysis diagrams](https://github.com/lost-in-the/woods/tree/main/docs/self-analysis), and the maintainer work ledger in `backlog.json`; these maintainer-only paths are not packaged with the gem.
+Historical build-phase documents are not user guides. Source checkouts also contain the [MCP protocol decision record](https://github.com/lost-in-the/woods/blob/main/docs/design/MCP_2026_STRATEGY.md) and [generated self-analysis diagrams](https://github.com/lost-in-the/woods/tree/main/docs/self-analysis); these maintainer-only paths are not packaged with the gem.
+
+[GitHub issues](https://github.com/lost-in-the/woods/issues) track active work
+and pending decisions. The old local backlog is retired; historical records,
+approved decisions and issue migration links are archived under `.Codex/`.
+See [work tracking](../CONTRIBUTING.md#work-tracking) for ownership and release labels.
 
 Contract records that read as maintainer reference rather than guides:
 

data/docs/RETRIEVAL_GUIDE.md

@@ -167,6 +167,23 @@ unit still prepares no text before accepting a matching no-vector checkpoint.
 Empty-vector reconciliation waits until all batches succeed, so a later provider
 failure does not retire those old vectors.
 
+### Prepared-input checkpoints
+
+**Included in Woods 2.1:** checkpoint schema 2 records preparation policy and
+ordered complete input fingerprints, including chunk/storage IDs. A change to a
+prepared prefix or chunks re-embeds even if the source hash is unchanged; metadata
+that does not affect prepared text does not force an embedding. Old checkpoints
+invalidate once, so budget for one re-embed after upgrading. Complete input
+bounds and lossless splitting are described in [Embedding Models](EMBEDDING_MODELS.md).
+
+Successful units advance their checkpoint only after all their inputs are
+embedded, stored and obsolete chunks reconciled. A provider failure can leave
+earlier successful batches committed; it does not mark the failed unit complete.
+Built-in durable stores refuse changed replacements when existing IDs cannot be
+enumerated. Custom stores without enumeration retain adapter-owned cleanup and
+cannot claim exact obsolete-chunk reconciliation. Keep backups for rollback;
+older readers do not understand the new checkpoint policy.
+
 ---
 
 ## Embedding-free lexical retrieval

data/docs/SOURCE_FRESHNESS.md

@@ -14,6 +14,7 @@ before using it; upgrading the plugin alone does not upgrade Woods.
 |---|---|---|
 | `current` | All covered inputs match their consumer baselines, with a verified fresh boot boundary and complete checks. A dirty checkout can be current. | Use the indexed facts within the coverage below. |
 | `drifted` | At least one captured input differs, was removed, or a relevant input was added. | Inspect the changed paths; choose a full run or a justified targeted refresh. |
+| `unavailable` | **Included in Woods 2.1:** source evidence exceeded its serialized-size limit; the code index was published successfully. | Inspect `unavailable.size_bytes` and `unavailable.limit_bytes`; another full extraction will not reduce this size. |
 | `unknown` | Evidence is incomplete: for example an old index, missing source/key, scan limit, opaque symlink directory, or unproved boot/consumer boundary. | Inspect `reasons`; use a deep check or fresh full capture as appropriate. |
 
 Drift can coexist with incomplete coverage. `reasons` reports both; absence of a
@@ -22,6 +23,13 @@ capture/traversal completion, while boot and consumer qualifications remain in
 `reasons`. `counts` contains total observed added/changed/removed paths; each
 `changes` list contains at most 30 paths and `truncated` marks longer lists.
 
+**Included in Woods 2.1:** `comparison_complete` also identifies whether the
+previous consumer baseline can establish additions. A missing or incompatible
+baseline stays unknown; it does not make every visible file an addition. Files
+not reached by an incomplete scan are not reported as deleted. Matching paths
+with different captured identities still prove drift. An unreadable entry leaves
+coverage incomplete while scanning continues through accessible siblings.
+
 ```json
 {"source_check":"deep"}
 ```
@@ -37,6 +45,19 @@ The result is tied to the **served** generation, including a reader holding an
 older generation during a concurrent publication. It is recomputed on each call;
 a source edit does not require an index generation change to become visible.
 
+**Included in Woods 2.1:** `recommendations` separates recovery actions:
+
+- `deep_check`: the quick reader reached its time limit; request one deep check.
+- `inspect_source_scan`: inspect `verification_reasons`, permissions, source
+  mapping and scan limits; a deeper scan cannot repair unavailable inputs.
+- `inspect_source_limits`: source freshness is unavailable because evidence exceeds its size limit; the code index remains usable.
+- `fresh_capture`: the captured baseline, boot or consumer evidence is incomplete;
+  fix the reported cause and run the launcher in a fresh process.
+
+Several recommendations can apply together. `verification_reasons` describes the
+current reader scan; `reasons` also retains capture failures. A quick reader limit
+alone does not require rebuilding the index.
+
 ## Establish a fresh baseline
 
 Run the launcher through the application's installed bundle:
@@ -56,12 +77,37 @@ commas and newlines. Refresh accepts known extractor names. Invalid arguments
 fail before extraction; the launcher propagates the child's failure or daemon
 stand-down exit 75. Split oversized incremental batches or choose full.
 
+For an application with a custom `config.output_dir`, **always pass the matching
+`--output PATH` or `WOODS_OUTPUT`**. The launcher cannot evaluate Rails configuration
+before capturing its boot inputs. In Woods 2.0.0, its implicit `tmp/woods` default
+overrides custom configuration and can create a second index.
+
+**Included in Woods 2.1 ([#591](https://github.com/lost-in-the/woods/issues/591)):**
+an implicit default no longer injects `WOODS_OUTPUT` into Rails boot. The task
+compares finalized configuration with that preboot default and refuses a mismatch
+before extraction, naming the configured path and explicit output remedy. An
+initializer using `ENV.fetch('WOODS_OUTPUT', custom_path)` therefore retains its
+configured default for this check. Explicit CLI/environment output still overrides
+configuration and binds capture, key and publication to the same directory.
+The refused launch may already have created a private key under `tmp/woods`, but
+publishes no generation there and preserves the configured index's generation.
+
 The launcher captures source before a **fresh child** evaluates its Gemfile,
 Rakefile, Rails boot and eager loading. A private one-use handoff binds the capture
 to root, output, action, rules, nonce and parent process. It waits for the child
 and removes the handoff when the child exits. Source is checked again before
-publication; edits during boot/extraction retain the earlier identity and are
-reported, rather than being silently adopted as a current baseline.
+publication. In Woods 2.0.0, edits during boot/extraction retain the earlier
+identity and are reported in the new generation, rather than being silently
+adopted as a current baseline.
+
+**Included in Woods 2.1:** the
+[constant-reference writer](EXTRACTOR_REFERENCE.md#constant-source-references)
+requires verified source for its graph evidence. Source changes after capture,
+during boot or extraction, refuse publication while reference enrichment
+participates. The final verification also checks covered non-Ruby inputs. The
+preceding generation remains active instead of publishing a new drifted
+generation. Retry against stable source in a fresh process. The existing
+generation can still report drift through `woods_status`.
 
 Existing Rake tasks, direct `Extractor` calls and the watch daemon remain usable.
 Their post-boot capture is marked `unverified_boot_boundary`; current bytes alone
@@ -80,22 +126,71 @@ scanner's exclusions. Explicit source roots override generic exclusions; the
 index output is always excluded. Contained file symlinks are checked for stable
 resolution; directory symlinks and escaping/unreadable inputs leave uncertainty.
 
+**Included in Woods 2.1 (#653):** supporting writers scan
+directory links under their logical application paths, preserving independent
+aliases. Static files with no input consumer do not prevent publication merely
+because a parent directory is a symlink. Entries in an external linked directory
+may be enumerated to determine scope, but an input file must resolve inside the
+application root before Woods opens its contents. Package files are inputs even
+outside `app/` and `lib/`; an external package file still refuses verification.
+Directory cycles, links that change during scanning, unreadable entries and scan
+limits keep evidence incomplete. Output-directory aliases are pruned. Run a fresh
+full extraction after upgrading to establish the updated capture rules; do not
+replace valid links or disable source verification to work around an older writer.
+
+**Included in Woods 2.1:** source paths use UTF-8 bytes even
+when the process locale is `C`. A filename with invalid UTF-8 bytes produces
+`undecodable_source_path` and incomplete coverage; directory entries with those
+names are pruned. Diagnostic labels escape the original bytes and are bounded.
+The watcher skips these entries, and reference verification retains the preceding
+generation. **Included in Woods 2.1:** the publication refusal names the reason
+and up to three escaped, bounded path labels, including entries outside source
+consumer scopes; it never includes file contents. Rename the affected entries to
+valid UTF-8 names before a fresh capture. Explicit root/output paths with invalid UTF-8 bytes are rejected as
+configuration errors.
+
 Every consuming scope keeps its own identities. An events scan can reread a
 service file while its service unit remains untouched; refreshing events does
 not certify the retained service unit. Successful file/whole-extractor work
 updates only its scopes, including negative results and confirmed deletion.
-Unchanged scopes keep their earlier baseline. Boot inputs advance only on a full
-run. Partial runtime changes retain an explicit `runtime_consumption` uncertainty
+Unchanged scopes keep their earlier baseline. A file can therefore remain
+`drifted` after an incremental task or hook re-extracts it: its file-extractor
+scope may be current while retained `runtime` evidence still refers to the old
+source. Known differences take precedence over the accompanying uncertainty;
+`unverified_boot_boundary` does not hide them. Inspect the reported scopes and
+use a fresh verified full launcher run when all reflected facts need a new
+baseline. An ordinary unverified full run cannot establish verified freshness.
+Boot inputs advance only on a full run. Partial runtime changes retain an explicit `runtime_consumption` uncertainty
 when Woods cannot prove every retained reflected fact was re-serialized. Named
 framework refreshes do not certify unrelated application inputs. A handled
 extractor error retains an explicit `extractor:<name>` uncertainty even if the
 extractor returns an empty result. Successful consumers keep their own evidence;
 a later full run without that failure can replace the uncertainty.
+In the Woods 2.1 reference writer, successfully returned models
+also retain their individual source evidence when another model fails. An
+optional framework model with a missing table therefore does not prevent
+unrelated incremental work; the model-extractor uncertainty remains visible.
+If an older writer already published a baseline without that individual evidence,
+run one full extraction after upgrading to rebuild it.
+
+With the **Woods 2.1 reference writer**, an edited Ruby service
+retained by an events-only refresh instead prevents publication: its current
+source cannot certify its older runtime facts or cached reference resolution.
+See [reference baseline recovery](INCREMENTAL_EXTRACTION.md#source-reference-baseline-and-upgrades).
+Omitted non-reference inputs retain their earlier consumer identity. The watcher
+keeps failed work for retry; it does not expand an incomplete batch or rebuild a
+missing reference baseline automatically.
 
 Custom loader source outside captured roots, missing eager-load coverage and
 uncaptured application-owned unit paths remain unknown. This is application
 source evidence: it does not certify external database schemas/data, remote
 configuration, installed gem bytes, provider state or live runtime services.
+**Included in Woods 2.1:** RubyGems installation metadata can establish that
+loaded/unit source inside the application directory belongs to an installed gem,
+including gems installed under `vendor/bundle`. Such files do not create false
+application-coverage errors. A vendor-shaped directory alone is insufficient;
+local path gems and custom loaders still need captured source roots. Explicit
+`--source-root` declarations retain coverage even for installed gem directories.
 
 ## Containers and hooks
 
@@ -107,6 +202,16 @@ same verifier without Rails initialization or provider work. Its optional
 Base64-encoded JSON transport supports `output`, `root` (an explicit reader-side
 source mapping), and `mode` (`quick` or `deep`).
 
+**Included in Woods 2.1:** results expose `recorded_root` (writer location),
+`checked_root` (the directory actually scanned), and `root_source` (`recorded`,
+`explicit`, or `working_directory`). `current` applies only to that checked root.
+The MCP IndexReader keeps using the recorded root; it does not infer a checkout
+from the index's location. For a copied index, a current result about the original
+root says nothing about edits in the copy. Run `woods:source_status` in the copy,
+or provide an explicit `root` mapping. The task defaults to its process working
+directory, including inside containers; launch it from the application root, not
+an unrelated directory or a monorepo parent. Missing source remains unknown.
+
 The opt-in SessionStart hook uses `WOODS_HOOK_RAKE` and `WOODS_OUTPUT`, including a
 Docker command prefix without requiring a host application bundle. It prints
 an actionable drift or unknown warning and stays quiet for current evidence.
@@ -114,6 +219,10 @@ Its ten-second process deadline includes command startup; the scan uses quick
 mode. Missing older tasks and failed/timed-out commands report unknown. Cancelling
 Docker exec does not itself prove the process inside the container stopped.
 A quiet session hook does not acknowledge deferred PostToolUse queue entries.
+**Included in Woods 2.1:** unknown warnings use the returned recommendations,
+including every applicable recovery action. A quick reader timeout advises a deep
+check without requiring a rebuild. Older tasks without recommendations retain
+the conservative generic unknown warning.
 
 ## Artifact and cost
 
@@ -128,11 +237,53 @@ unknown; Woods does not silently repair permissions or rotate keys. Independent
 outputs have different identities and must be compared using their own keys and
 consumer semantics, not raw manifest equality.
 
+### Identity-key recovery
+
+Verified launching refuses an unavailable, insecure or malformed
+`<output>/.source-inputs.key`; it cannot establish verified capture without that
+key. In Woods 2.1 diagnostics, the error names the path and safe
+file requirements while reader reason codes remain unchanged. No key bytes are
+printed and Woods does not chmod, chown, replace or rotate the file automatically.
+
+Inspect the file and mount from the application environment. It must be a regular
+file, not a symlink/FIFO, contain exactly 32 bytes, belong to the process's UID,
+and grant no group/other permissions (normally mode `0600`). A host/container UID
+mismatch requires correcting the selected runtime user or deliberately repairing
+ownership of the known original key. Confirm the file belongs to this index before
+changing permissions. Do not expose the key in logs or copy it into payloads.
+
+If the original key is lost, replaced or cannot be trusted, retain the previous
+index and establish a fresh full baseline in a new empty output directory with
+the intended application user. Configure the writer and readers together.
+Do not replace a key and assume the old generation now has valid source evidence.
+
 Failed/no-op extraction does not advance the artifact's published generation.
 Flat fallback and older indexes lack verified atomic source evidence.
 `WOODS_PROFILE=1` reports `source capture` and `source verification` separately.
 Capture/recheck each allow up to ten seconds with the same file/byte caps.
 
+**Included in Woods 2.1:** the published manifest and private launcher handoff
+have a 16 MiB serialized-size limit. If size alone exceeds that limit, extraction
+continues and publishes the code index successfully with a loud warning and
+bounded evidence: `state: "unavailable"`, reason `source_manifest_too_large`, and
+`unavailable.size_bytes` / `unavailable.limit_bytes`. A launcher handoff above the
+limit continues in a fresh child without claiming verified preboot capture.
+Status, validation and hooks retain this limitation; they do not recommend an
+identical full rebuild. Source freshness remains unavailable on subsequent
+incremental runs until a full capture fits the limit.
+
+Normal changed-file and Git-based incremental work continues. Unchanged
+source-reference facts require both unchanged keyed source identities and exact
+typed ownership in the validated, hash-bound reference cache from the same
+published generation. Missing, corrupt or mismatched proof still refuses reuse.
+The watcher retains a compact captured-tree fingerprint for catch-up only; that
+fingerprint never establishes runtime freshness. Invalid evidence, failed writes
+and source instability retain their existing publication safeguards.
+
+Version-1 manifests remain readable. The optional `comparison_complete` field
+supplements existing coverage errors; an unavailable manifest never claims
+complete comparison coverage.
+
 September 2026 fixture measurements: a pinned Writebook source tree (456 visited
 files, 223 hashed, 233KB) completed quick scans in median 36ms native / 45ms on a
 Linux container bind mount. Discourse (26,133 visited, 6,209 hashed, 56.4MB) needed
@@ -140,4 +291,5 @@ about 1.6–2 seconds; quick scans returned unknown, while five-second scans com
 traversal and still reported an opaque directory symlink. Native Ruby 4.0 and
 container Ruby 3.4 differed, so these are a budget envelope, not a filesystem
 speed comparison or a macOS virtiofs benchmark. Measure your own application;
-a large or slow source tree may need a full extraction rather than a longer read.
+a large or slow source tree may need a deep check. Rebuilding a verified baseline
+does not remove the reader's time, file or byte limits.

data/docs/TOKEN_BENCHMARK.md

@@ -1,20 +1,12 @@
 # Token Estimation Benchmark
 
-> **Single source of truth:** `Woods::TokenUtils.chars_per_token_for(provider)`
-> in `lib/woods/token_utils.rb`. Production code uses **4.0 chars/token** for
-> the OpenAI path and **1.5 chars/token** for the Ollama / WordPiece path.
-> Both are applied consistently by `Woods::Builder#chars_per_token_for`,
-> `ContextAssembler`, `TextPreparer`, and `ExtractedUnit#estimated_tokens`.
-> The cost-model layer (`lib/woods/cost_model/`) is the one exception, it
-> uses its own pre-aggregated `TOKENS_PER_CHUNK` constant (450) rather than
-> a per-string ratio, since it measures a different thing (per-chunk average
-> vs. per-string chars/token).
->
-> When the optional [`tokenizers`](https://github.com/ankane/tokenizers-ruby)
-> gem is installed, the Ollama path uses the real BERT WordPiece tokenizer
-> (`Woods::Embedding::TokenCounter`) instead of this heuristic. The 4.0
-> divisor applies to the OpenAI/default path; Ollama falls back to 1.5 when
-> its tokenizer is unavailable.
+> **Historical sizing evidence, not an input-limit guarantee.**
+> `Woods::TokenUtils.chars_per_token_for(provider)` supplies character estimates
+> for retrieval assembly and initial sizing. Woods 2.1 uses
+> a conservative UTF-8 byte bound for known OpenAI embedding models and honest
+> estimates for Ollama/custom models. Prefixes count toward admission; oversized
+> inputs are split without dropping source. The optional tokenizer gem no longer
+> downloads or implicitly selects BERT. See [Embedding Models](EMBEDDING_MODELS.md).
 
 This is a historical record of the benchmark that picked 4.0 over the
 original 3.5 divisor. It is cited from five places in `lib/` as the evidence
@@ -58,9 +50,9 @@ current definition and `docs/EMBEDDING_MODELS.md` for the Ollama-side ratio.
 **tiktoken_ruby was deliberately not added as a runtime dependency.** A 10.6%
 mean error is acceptable for chunking decisions, budget estimates, and
 truncation; a native-extension dependency for marginal accuracy gains wasn't
-worth it. The optional `tokenizers` gem provides counts for its supported BERT
-WordPiece tokenizer, not every model. Strict token-limit enforcement requires
-the tokenizer used by the target model.
+worth it. An explicitly injected local tokenizer must match the target model. A conservative
+upper bound can also establish admission for a known encoding; a character
+average cannot.
 
 ## Reproducing this benchmark
 

data/docs/TROUBLESHOOTING.md

@@ -32,6 +32,17 @@ This guide covers the most common problems encountered when installing, extracti
 | Tool returns `error_code: :not_configured` | Feature flag or credential not set | Check `config_key` in `_meta` and the linked `doc_link` |
 | Tool returns `error_code: :rate_limited` | `PipelineGuard` 5-min cooldown hit | Wait `retry_after_seconds` from `_meta`, then retry |
 
+### Source-reference baseline needs a full extraction
+
+**Included in Woods 2.1.** The related initial diagnostic is
+`Source-reference baseline is missing or incompatible`. Both mean the writer
+cannot safely reuse its reference cache or verify the source consumed by retained
+units. It can follow an older-index upgrade, missing cache artifacts, or source
+changes omitted from the refresh batch. Confirm the loaded gem revision and
+output directory, then follow the [full baseline rebuild](INCREMENTAL_EXTRACTION.md#source-reference-baseline-and-upgrades).
+The failed operation leaves the published generation unchanged. Preserve pending
+watcher work; increasing traversal budgets cannot repair extraction coverage.
+
 ### First-Pass Diagnostics
 
 For a single-call health snapshot, call the Index Server's `woods_status` tool. It reports:
@@ -67,6 +78,12 @@ the selected owner or installed command and restarting that owner; do not delete
 claim files or kill PIDs taken from status. No index-visible record exists before
 the first boot resolves the application's output directory.
 
+For an abandoned foreign-container claim, follow
+[ownership-verified claim recovery](WATCH_DAEMON.md#recovering-an-abandoned-managed-claim).
+The Woods 2.1 command requires the exact token and a free lifetime
+lease; old claims require the documented legacy recovery. Age is not proof of
+abandonment. `woods:clean` preserves ownership sidecars and does not reset them.
+
 Unset `WOODS_WATCH_IDLE_TIMEOUT` in managed modes. If the boot deadline is reached,
 diagnose Bundler/initializer startup before increasing `--boot-timeout`; a valid
 long extraction has a separate readiness state and is not bounded by that clock.
@@ -74,6 +91,10 @@ If setup created a Procfile but normal `bin/dev` still only launches Rails, choo
 Puma or explicitly run the selected Foreman command. The generator never rewrites
 `bin/dev` or starts services during preview.
 
+An empty idle-timeout variable crashes older tasks even though managed validation
+accepts it. Unset it on those builds; the Woods 2.1 #591 fix consistently treats
+empty or whitespace-only values as unset.
+
 If installation reports a pending transaction, use `woods:watch --operation
 recover` through the Rails generator, initially with `--pretend`; see
 [owned setup recovery](WATCH_DAEMON.md#ownership-updates-and-removal). That Rails
@@ -415,7 +436,7 @@ full extraction when current history and provenance are required.
 
 **Cause:** The range git was asked to diff does not resolve in the checkout: a GitLab `CI_COMMIT_BEFORE_SHA` of all zeros (new branch), a GitHub base ref that was never fetched, a shallow clone with no `HEAD~1`, or a typo'd revision. This used to read as "no relevant files changed" and the task exited 0 while the sync never ran; it now fails closed, because a green job hiding a skipped sync lets index drift grow unbounded. The one stand-down: a *running* watch daemon maintaining the same index exits 0 with a printed reason, since its start-up catch-up covers the changes. A degraded daemon covers nothing and still exits 1.
 
-**Fix:** Repair or provide the range (fetch the base ref, e.g. `fetch-depth: 2` or more, or correct the CI environment variables), set `CHANGED_FILES` explicitly to bypass range resolution, or run a full extraction:
+**Fix:** Fetch the actual base ref and enough history to resolve both endpoints (for example `fetch-depth: 0` with an explicit base-ref fetch in GitHub Actions), or correct the CI environment variables. A depth of two does not establish an arbitrary CI base. Set `CHANGED_FILES` explicitly to bypass range resolution, or run a full extraction:
 
 ```bash
 bundle exec rake woods:extract
@@ -461,6 +482,31 @@ the unreachable payload. A resident `woods:watch` process handles the same
 failure differently: it reports `degraded`, carries the changed paths, and
 retries after a later filesystem event.
 
+For the optional embedded `pipeline_extract` tool, a client using the Tasks
+extension sees the task become `failed` for this publication refusal on a
+revision containing #584 (included in Woods 2.1). A background-start response
+alone does not mean extraction completed. The packaged Index Server does not
+register this tool.
+
+### Incremental extraction or refresh reports "Extraction failed for ..."
+
+A selected whole-app extractor raised before returning a complete result, or
+its initialization failed. In Woods 2.1, a successful sibling extractor cannot
+turn that failed batch into a successful publication (#584). Readers retain the prior generation. Inspect the
+earlier log line naming the failed extractor, fix its cause, then retry the
+complete changed-file list or refresh selection. Preserve the published index;
+deleting it does not repair the failing extractor.
+
+### Incremental extraction reports "Restart-sensitive inputs"
+
+On revisions containing #588 (included in Woods 2.1), the direct incremental
+API and optional embedded pipeline refuse schema or boot-configuration inputs.
+Apply required migrations, then run `bin/rails woods:extract` in a fresh Rails
+process. Reusing an embedded server's old Rails configuration or schema cache
+cannot establish current runtime facts. The fresh one-shot `woods:incremental`
+task selects a full run automatically for these inputs. See the
+[runtime input contract](INCREMENTAL_EXTRACTION.md#what-a-change-actually-requires-reload-restart-or-neither).
+
 ---
 
 ### `manifest.json` shows the wrong branch (or `git_branch: "unknown"`) in a worktree
@@ -737,6 +783,15 @@ Woods detects the dimension mismatch and raises `Woods::MCP::DimensionMismatch`
 
 **A dimension mismatch is never silently tolerated.** If you are getting poor results without seeing this error, the cause is something else.
 
+For an unsupported `dimensions` request or a wrong-width cached vector, compare
+the installed embedding and reader revisions as well as the model, endpoint,
+and explicit width configuration. The request/cache consistency fix (#586) is
+included in Woods 2.1: it separates stored widths from requested reductions,
+keeps fixed-width ada requests compatible, and separates embedding cache entries
+by provider configuration. See [embedding options](CONFIGURATION_REFERENCE.md#embedding-options)
+and [cache identity](CONFIGURATION_REFERENCE.md#retrieval-cache-options). Do not
+remove a width guard to accept mismatched vectors.
+
 ---
 
 ### OpenAI API errors during embedding
@@ -786,20 +841,16 @@ config.embedding_options = { host: 'http://localhost:11434' }
 
 **Symptom:** `rake woods:embed` fails with `Ollama API error: 400 {"error":"the input length exceeds the context length"}`. Individual chunks may look smaller than the configured `num_ctx`.
 
-**Cause:** Ollama's `/api/embed` endpoint enforces the model's **native** `context_length`, not the `options.num_ctx` override (see [ollama/ollama#14186](https://github.com/ollama/ollama/issues/14186)). For `nomic-embed-text` that's 2048 tokens, regardless of what `num_ctx` is set to. Separately, without the `tokenizers` gem, Woods estimates token counts from character length, which under-counts dense Ruby source, so chunks that look safe by char count still trip the 2048-token ceiling.
-
-**Fix:** Use Woods 2.0 and install the `tokenizers` gem:
-
-```ruby
-# Gemfile
-gem 'woods', '~> 2.0'
-gem 'tokenizers', '~> 0.5'   # exact BERT WordPiece token counting
-```
+**Cause:** the server enforces the selected model's actual context window.
+Character estimates can undercount dense Ruby or multilingual source; a larger
+`num_ctx` setting does not establish that the model accepts a larger input.
 
-Woods now:
-
-1. Advertises the native context ceiling per model (2048 for `nomic-embed-text`, 8192 for `bge-m3`/`snowflake-arctic-embed2`, etc.) so the chunker sizes inputs correctly.
-2. Uses the real BERT tokenizer to verify every chunk, catching the 10–20% gap between char-based estimates and Ollama's internal count.
+**Diagnosis:** record the loaded Woods revision, model and configured context.
+In supporting builds after 2.0.0, Woods counts the full metadata prefix plus
+source, splits without truncating source, and requests Ollama `truncate: false`.
+An overflow is a visible refusal; read the unit/model/limit diagnostic rather
+than enabling truncation or installing an unrelated BERT tokenizer. See the
+[input counting contract](EMBEDDING_MODELS.md#why-num_ctx-isnt-enough).
 
 If you want fewer chunks per unit and have the disk space, switch to a larger-context model:
 
@@ -1046,3 +1097,8 @@ source root/private key, a quick scan limit and an unverified boot boundary are
 different causes. Try `source_check: "deep"` for a budget limit; use the fresh
 launcher for a new verified baseline. Do not delete pending hook events or alter
 key permissions just to suppress a warning. See [source freshness](SOURCE_FRESHNESS.md).
+
+If `woods-extract` refuses an identity key, use
+[identity-key recovery](SOURCE_FRESHNESS.md#identity-key-recovery). If it reports a
+configured-output mismatch, rerun with an explicit matching `--output` or
+`WOODS_OUTPUT`; do not bypass the check or move a capture/key to another index.