woods 2.0.0.beta2 → 2.0.0.beta4

data/docs/EXTRACTOR_REFERENCE.md

@@ -2,7 +2,7 @@
 
 Woods ships **35 extractor classes** producing **39 distinct unit types**: one for each meaningful category of Rails code. This doc covers what each extractor captures, how to configure them, and the shape of the data they produce.
 
-> **Counts explained.** `lib/woods/extractors/` contains 42 files: 35 extractor classes (each ending in `_extractor.rb`) plus 7 supporting utilities (`shared_utility_methods`, `shared_dependency_scanner`, `callback_analyzer`, `behavioral_profile`, `route_helper_resolver`, `ast_source_extraction`, `source_nesting`). The 39 unit types comes from some extractors emitting multiple categories, `GraphQLExtractor` alone produces four (`graphql_type`, `graphql_mutation`, `graphql_resolver`, `graphql_query`), and `RailsSourceExtractor` produces both `rails_source` and `gem_source`. Supporting utilities enrich existing extractors (callback side-effects, behavioral config, AST-based source slicing, nested-namespace resolution) but are not themselves extractors and do not appear in the unit type enumeration. The authoritative mapping is `Woods::Extractor::TYPE_TO_EXTRACTOR_KEY` in `lib/woods/extractor.rb`.
+> **Counts explained.** `lib/woods/extractors/` contains 35 extractor classes (each ending in `_extractor.rb`) plus supporting utilities such as `shared_utility_methods`, `shared_dependency_scanner`, `callback_analyzer`, `behavioral_profile`, `route_helper_resolver`, `ast_source_extraction`, `source_nesting`, and `declared_parent`. The 39 unit types comes from some extractors emitting multiple categories, `GraphQLExtractor` alone produces four (`graphql_type`, `graphql_mutation`, `graphql_resolver`, `graphql_query`), and `RailsSourceExtractor` produces both `rails_source` and `gem_source`. Supporting utilities enrich existing extractors (callback side-effects, behavioral config, AST-based source slicing, nested-namespace resolution) but are not themselves extractors and do not appear in the unit type enumeration. The authoritative mapping is `Woods::Extractor::TYPE_TO_EXTRACTOR_KEY` in `lib/woods/extractor.rb`.
 
 ---
 
@@ -60,9 +60,14 @@ Every extractor returns `Array<ExtractedUnit>`. An `ExtractedUnit` is a self-con
 
 **Key details:**
 - Uses `ActiveRecord::Base.descendants` for discovery (runtime introspection, not static parsing)
+- Named, source-defined app model mixins (for example `Card::Pinnable` in `app/models/card/pinnable.rb`) resolve through runtime source locations, with conventional concern paths as fallbacks. Included mixins also receive `:concern` units, so their actual files map to the includer through dependency edges. Gem-owned modules stay outside this discovery. Conventional concern files retain their existing identity even when nested helpers share the same file. Outside those directories, each included runtime mixin receives its own concern identity even when several share a source file; editing that file refreshes every includer.
 - Inlines concerns: all `include FooConcern` references are resolved and the concern source is appended to `source_code`. Inlined concern names are recorded in `metadata[:inlined_concerns]`
-- Extracts all 19 callback types: `before_validation`, `after_validation`, `before_save`, `after_save`, `around_save`, `before_create`, `after_create`, `around_create`, `before_update`, `after_update`, `around_update`, `before_destroy`, `after_destroy`, `around_destroy`, `after_commit`, `after_rollback`, `after_initialize`, `after_find`, `after_touch`
+- Reads Rails' per-event callback chains (`_save_callbacks`, `_create_callbacks`, and the other lifecycle events), preserving each chain's order and each entry's `kind`, filter and conditions. The public `type` combines kind and event, such as `before_save` or `after_create`; Rails' separate `before_commit` event is reported as `before_commit`, not `before_before_commit`. `callback_count` equals the emitted callback list's length. The list includes framework-registered callbacks; it is runtime metadata, not an application-only filter or a cross-event execution trace. Regenerate the index after upgrading to pick up corrected callback metadata.
+- Proc/lambda filters, including Rails-generated association callbacks, use stable source-site labels in both metadata and callback chunks: `#<Proc app/models/post.rb:12>` (or `lambda`). App paths are relative to `Rails.root`; external paths are retained and native procs use `native`. Rails 6's numeric filter identity is resolved through `raw_filter`. These labels describe location and callable kind, not captured closure state; callbacks are never executed. Model condition labels retain their existing format.
+- Default callback-object representations omit process addresses: an instance becomes `#<CleanupCallback>`, an anonymous class becomes `#<Class>`, and its instance becomes `#<#<Class>>`. Anonymous namespace prefixes are normalized too (for example, `#<Module>::CleanupCallback`). Named classes and custom `to_s` labels retain their text. These labels do not distinguish arbitrary object state; separate registered callbacks remain separate entries even when their descriptive labels match. Controller object-filter formatting is unchanged.
+- Direct Proc/lambda validation option values (for example inclusion/exclusion membership or message callables) use the same stable kind/source-site labels without execution. Validation order and duplicates, condition formats (`if`, `unless`, `on`), and non-Proc values are unchanged. Nested arrays/hashes are not recursively normalized, and labels do not serialize captured closure state. Run a full extraction after upgrading to refresh retained validation metadata; the stored schema is unchanged.
 - Callback side-effects are analyzed via `CallbackAnalyzer`: detects columns written (`self.col =`), jobs enqueued (`perform_later`), and services called
+- Reflects model class and instance methods after reading the schema, so Rails schema-loading optimizations produce the same method metadata in cold and warmed runs. Application-defined constructors remain visible; Rails versions that install an optimized singleton `new` during schema loading consistently include it in `class_methods`.
 - Automatically skips HABTM join models and anonymous classes
 - Chunks every model into semantic sections: `:summary`, `:associations`, `:callbacks`, `:validations`, `:scopes`, `:methods`
 - **Runtime-generated method detection:** Because extraction runs inside a booted Rails process, `instance_methods(false)` captures every method Rails generates dynamically, enum predicates (`status_active?`, `status_pending?`), association builders (`build_profile`, `create_line_item!`), attribute accessors, and dynamically registered scopes. Static analysis tools cannot see these methods because they only exist after Rails processes the DSL declarations at boot time
@@ -158,6 +163,8 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 - Route context is inlined in `source_code` as a comment header, not just in metadata
 - Chunks per-action: each action becomes a `:action` chunk with its applicable filters and route
 - Metadata includes permitted params (strong parameters), response formats, and applied filters per action
+- Inline callbacks use stable source-site labels in filter metadata, controller annotations and action chunks: `#<Proc app/controllers/posts_controller.rb:12>` (or `lambda`). The controller filter metadata and annotations use the same labels for `if`/`unless` procs. App paths are relative to `Rails.root`; external paths are retained and native procs use `native`. Labels describe the callable location and kind, not captured closure state, and never execute callbacks.
+- Route helper resolution accepts every live named controller/action route, including `file_path`, `image_url`, `download_path`, and `root_path`. Unknown filesystem/asset helpers produce no edge; matching names are conservative source references, not proof a call executes.
 - Extracts `redirect_to` navigation edges: named route helpers (`posts_path`, `users_url`) are resolved to controller targets via `RouteHelperResolver`, producing `:redirect_to` dependency edges (gated by `extract_navigation_edges` config)
 
 **Edge cases:**
@@ -194,6 +201,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 - Scans: `app/services`, `app/interactors`, `app/operations`, `app/commands`, `app/use_cases`
 - Extracts public entry points (`call`, `perform`, `execute`, `run`), custom error classes, and dependency references
 - File-based discovery (not class introspection), so it catches services with non-standard superclasses
+- `initialize_params` describes declared names, default presence and keyword status from Ruby syntax. Nested/comma-bearing defaults are not evaluated or treated as parameters; named rest, keyword-rest and block parameters retain their names. Anonymous forwarding has no name to report; malformed source produces an empty parameter list.
 
 **Example output (abbreviated):**
 
@@ -218,6 +226,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **Key details:**
 - Scans: `app/jobs`, `app/workers`, `app/sidekiq`
 - Extracts queue name, retry configuration, concurrency options, perform method arguments, and callbacks
+- `perform_params` uses the same syntax-aware signature parsing as service initializers and preserves its `name`, `splat` (`single`/`double`/null), and `has_default` fields. Keyword defaults do not invent additional argument names.
 - Records what triggers this job (reverse lookup via dependency graph after extraction)
 - Supports both ActiveJob and Sidekiq native workers
 
@@ -243,9 +252,14 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **What it captures:** ActionMailer classes with their mailer actions, defaults, template paths, callbacks, and helper usage.
 
 **Key details:**
-- Discovers via class introspection (`ActionMailer::Base.descendants`)
+- Discovers `ApplicationMailer.descendants` when that class exists, otherwise `ActionMailer::Base.descendants`; an app without ActionMailer contributes no mailer units.
+- Discovery and direct extraction accept only mailers backed by an existing app-owned source file, excluding dependency mailers and fabricated convention paths.
 - Each mailer action corresponds to an email template, template paths are recorded in metadata
 - Extracts `default from:`, `layout`, and per-action subject patterns
+- Action names are sorted consistently in metadata, the generated header, template discovery and action chunks. Callback chain order and duplicate registrations are preserved.
+- Direct Proc-valued defaults and Proc callback filters use source-location/kind labels without executing them; application paths are relative to `Rails.root`. Default containers, literal strings and non-Proc values retain their existing types. This does not serialize closure captures or recursively normalize arbitrary nested objects.
+- Object callback filters use descriptive labels with addresses removed only from Ruby's default representation, including anonymous classes and namespaces. Custom labels and literal hexadecimal text are preserved. Labels do not serialize callback object state, and extraction never invokes callbacks.
+- After upgrading, run a full extraction to refresh retained mailer units. Stabilized headers and callable labels can cause a one-time source-hash change; stored index schemas are unchanged.
 
 ---
 
@@ -294,7 +308,8 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 
 **Key details:**
 - Extracts the entire stack as one unit (not one per middleware)
-- Records middleware class names, insertion order, and any initialization arguments
+- Records middleware class names, insertion order, and initialization arguments as readable strings
+- Argument rendering preserves literal strings and nested array/hash configuration. Procs use source locations; anonymous classes (including Ruby temporary names used by Rails executors/reloaders) use parent names and method source locations. Opaque objects using Ruby's default `to_s` are represented by class, without walking private runtime state. Custom `to_s` output is preserved, so application-defined nondeterministic renderers can still vary. Closure captures and opaque object internals are not serialized.
 - No per-file mapping, so incremental re-extraction re-runs `MiddlewareExtractor` wholesale when `config/application.rb`, `Gemfile.lock`, or a file under `config/initializers`/`config/environments` changes
 
 ---
@@ -331,6 +346,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **Key details:**
 - File-based scanning, no Rails boot needed for the actual file reading
 - Records which partials a template renders and which instance variables it expects
+- Loads the runtime route collection before caching named helpers, including Rails lazy route sets. Fresh-process incremental view extraction resolves the same navigation targets as full extraction.
 - Extracts navigation dependencies: `link_to` and `form_with`/`form_for` calls using `_path`/`_url` route helpers are resolved to controller targets via `RouteHelperResolver`
 - Navigation edges use `:link_to` and `:form_action` via types in the dependency array
 - Gated by `extract_navigation_edges` config (default: true)
@@ -371,6 +387,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 - Scans `app/models` for files that don't define an `ActiveRecord::Base` descendant
 - Common examples: value objects, form objects placed in `app/models`, domain structs
 - Excludes concerns (those go to ConcernExtractor)
+- `parent_class` and the generated Parent annotation describe the selected unit declaration only. Nested or sibling classes cannot supply its parent. An implicit `Object` parent, a dynamic superclass expression, or unparseable source produces `nil`; explicit constant-path parents retain their source names.
 
 ---
 
@@ -410,6 +427,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 - Produces unit types: `graphql_type`, `graphql_mutation`, `graphql_resolver`, `graphql_query`
 - Extracts field metadata (types, descriptions, complexity, arguments), authorization patterns (Pundit, CanCan, `authorized?`), and dependencies on models/services
 - Since all GraphQL units come from one extractor, incremental re-extraction handles them via `extract_graphql_file`
+- `parent_class` and summary chunks describe the selected declaration's explicit constant-path superclass, preserving its written qualification. Nested or sibling declarations and literal text cannot supply a parent. Implicit Object, module interfaces, dynamic superclass expressions, unavailable source, and invalid source have no declared parent (`null` metadata; `unknown` in summaries). This is source declaration metadata, not resolved runtime ancestry.
 
 **Example output (abbreviated):**
 
@@ -469,6 +487,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **Key details:**
 - Identifier is the package directory relative to `Rails.root` (`.` for the root package), the same name Packwerk uses
 - Honors `packwerk.yml` `package_paths` and `exclude`; without one, `**/` with the Packwerk default excludes (`bin`, `node_modules`, `script`, `tmp`, `vendor`)
+- With those exact defaults, excluded top-level directories are pruned before discovery descends into them, so an index or snapshots under `tmp/` do not add package-scan work. Custom patterns or exclusions retain their configured glob behavior. Hidden directories and symlink directories are not recursively followed by default.
 - `metadata`: `name`, `dependencies` (sorted), `enforce_dependencies` (`true`, `false`, or `"strict"`), `enforce_privacy`, `layer` (pks), `public_path`, `owner`
 - Each declared dependency becomes a `{ type: :package, target: <name>, via: :package_dependency }` edge
 - Package membership on other units (`metadata[:package]`, below) does not depend on how a unit was discovered: any registered unit with a file path under a package root is annotated. The undeclared cross-package edge report remains a follow-up, not this extractor. Discovery is the separate open gap: a pack-resident file-based unit is not yet found by `PathDispatcher` when only its `package.yml` changes (follow-up B-175), so it carries no membership only because it has no unit at all yet, not because membership skips it
@@ -530,7 +549,17 @@ Every app-owned unit under a package root carries `metadata[:package]` with the
 **Key details:**
 - Reads: `config/recurring.yml` (Solid Queue), `config/sidekiq_cron.yml` (Sidekiq Cron), `config/schedule.rb` (Whenever)
 - Extracts job class name, cron expression, queue, and any arguments
-- File-based (static read, no Rails introspection needed)
+- Resolves trusted application `recurring.yml` through Rails' configuration loader,
+  including ERB, filename-relative `require_relative`, and YAML aliases. On Rails
+  6.0 (before that loader existed), evaluates ERB with its filename and retains
+  safe YAML loading of scalars, hashes, arrays and symbols. ERB runs application
+  code in the extraction process; index only applications you trust.
+- Environment-wrapped task maps select the current Rails environment, including
+  custom names; an absent environment falls back to the first section, while an
+  explicitly empty section stays empty. Flat task maps remain supported.
+- Sidekiq-Cron remains safe-loaded YAML; Whenever remains a static DSL scan.
+  Invalid YAML/ERB, missing required files and runtime configuration errors are
+  logged and omit that schedule file. Source remains the original file text.
 - No per-file mapping, so incremental re-extraction re-runs `ScheduledJobExtractor` wholesale whenever one of the schedule files above changes
 
 ---
@@ -655,6 +684,7 @@ Every app-owned unit under a package root carries `metadata[:package]` with the
 **Key details:**
 - Excludes `lib/tasks/` (covered by RakeTaskExtractor) and `lib/generators/`
 - File-based scanning; no assumption about class hierarchy
+- `parent_class` and the generated Parent annotation describe the selected unit declaration only. Nested or sibling classes cannot supply its parent. An implicit `Object` parent, a dynamic superclass expression, or unparseable source produces `nil`; explicit constant-path parents retain their source names.
 
 ---
 
@@ -711,7 +741,16 @@ When written to disk, units also include:
 
 ### Git enrichment fields (`metadata[:git]`)
 
-If the host app is a git repo, the following are added to `metadata[:git]` after extraction:
+If the host app is a git repo, the following are added to `metadata[:git]` after extraction.
+History is limited to commits reachable from `HEAD` in the past 365 days,
+including merged branch history. Unmerged branches, remote refs, and tool
+checkpoint refs do not contribute. Commands run against the application root;
+when `WOODS_GIT_DIR` is set, `HEAD` belongs to that explicitly selected git
+directory, which may differ from a linked worktree's HEAD.
+
+After upgrading from a version that included all refs, run a full
+`woods:extract` to replace previously published git metadata. Incremental
+extraction refreshes only the units it rewrites.
 
 | Field | Description |
 |-------|-------------|

data/docs/FAQ.md

@@ -274,7 +274,13 @@ Verify the active v2 generation with `docker compose exec app bundle exec rake w
 
 ### How do I configure the Console Server with Docker?
 
-First set `config.console_mcp_enabled = true` in the Rails initializer after reviewing the live-data trust boundary. Stdio does not send a bearer token, but production Rails boot still requires a configured `console_mcp_token` of at least 32 characters whenever Console is enabled. Supply `WOODS_CONSOLE_MCP_TOKEN` through the container's secret mechanism; see [Console MCP setup](CONSOLE_MCP_SETUP.md#option-a-stdio-via-rake-recommended). Then, for the embedded mode (9 Tier 1 tools), point the MCP client at `docker compose exec -T` so Compose does not allocate a pseudo-TTY:
+First enable the master `console_mcp_enabled` switch after reviewing the
+live-data trust boundary. For stdio-only use, explicitly set
+`console_mcp_http_enabled = false`; HTTP remains enabled by default for
+compatibility and requires its bearer token while enabled. Follow
+[Console MCP setup](CONSOLE_MCP_SETUP.md#option-a-stdio-via-rake-recommended)
+for the configuration. Then point the embedded-mode client (9 Tier 1 tools)
+at `docker compose exec -T` so Compose does not allocate a pseudo-TTY:
 
 ```json
 {
@@ -414,7 +420,7 @@ When you run `rake woods:embed`, Woods generates embedding vectors for each extr
 Several options for tuning retrieval:
 
 - **Increase `max_context_tokens`** to include more units per query (at the cost of larger LLM context).
-- **Lower `similarity_threshold`** (default 0.7) to include less similar results.
+- **Use explicit retrieval scopes** and inspect ranking evidence. `similarity_threshold` is deprecated and does not filter results; see [retrieval tuning](RETRIEVAL_GUIDE.md#tuning).
 - **Enable framework sources** (`include_framework_sources: true`) if Rails internals are relevant to your queries.
 - **Use retrieval feedback only in a custom embedded server** that wires a feedback store. The normal packaged executable does not register feedback tools.
 
@@ -442,16 +448,9 @@ Snapshots prefer their own SQLite database (`woods.sqlite3` in the output direct
 
 The session tracer is middleware that records which Rails actions are invoked during a browser session, assembles the relevant extracted units, and makes that context available via the `session_trace` MCP tool. It is useful for giving an AI tool accurate context about what code path was active during a specific user interaction.
 
-Session tracing is disabled by default. To enable it:
-
-```ruby
-config.session_tracer_enabled = true
-config.session_store = Woods::SessionTracer::FileStore.new(
-  Rails.root.join('tmp/session_traces')
-)
-```
-
-The `session_store` option is required, there is no default store.
+Session tracing is disabled by default and requires an explicit `session_store`.
+Follow the [canonical configuration example](CONFIGURATION_REFERENCE.md#session-tracer-options)
+for store construction and review trace retention and access controls before enabling it.
 
 ---
 

data/docs/GETTING_STARTED.md

@@ -12,7 +12,19 @@ If an agent will perform the installation, use the safety and handoff checklist
 
 ## 1. Install the gem
 
-Add Woods to the development group:
+Use the [README release table](../README.md) to choose a version, then confirm
+that **exact version is published** on the [RubyGems versions page](https://rubygems.org/gems/woods/versions)
+before editing the Gemfile. A prepared release checkout can update the README
+before its gem is published; if the version is absent, choose an available
+version or wait for publication.
+
+If the published 2.x line has only beta or release-candidate versions, use an
+exact pin to the published prerelease, following the README's prerelease
+instructions; `~> 2.0` does not select prereleases. Follow the selected version's
+tag documentation. The `main` guides may describe features absent from the
+published gem.
+
+Once a stable 2.x release is published, add Woods to the development group with:
 
 ```ruby
 # Gemfile
@@ -108,13 +120,13 @@ For example:
 
 > Use Woods to find `Order`, inspect its resolved callbacks and associations, and list the first two levels of code that depend on it. Cite the Woods identifiers you used.
 
-The Index schema inventory totals 29 schemas. Fourteen register as tools in a normal packaged launch; `codebase_retrieve` is among them but returns a configuration error until embeddings are enabled. The other structural tools work immediately. See [Agent guide](AGENT_GUIDE.md) for a reliable query workflow.
+The Index schema inventory totals 29 schemas. Fourteen register as tools in a normal packaged launch; `codebase_retrieve` is among them but needs embeddings in the default semantic mode, or explicit [lexical retrieval](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval) over extraction output. The other structural tools work immediately. See [Agent guide](AGENT_GUIDE.md) for a reliable query workflow.
 
 ## Optional next steps
 
 ### Add semantic search
 
-Structural search, exact lookup, dependency traversal, graph analysis, and flow tracing do not need embeddings. Add embeddings only when agents need natural-language retrieval.
+Structural search, exact lookup, dependency traversal, graph analysis, and flow tracing do not need embeddings. For ranked natural-language retrieval, choose explicit [lexical mode](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval) without providers, or configure embeddings for semantic matching.
 
 The local preset uses SQLite metadata, persisted in-memory vectors, and a local Ollama service. Add `gem "sqlite3"` to the application bundle if it is not already present. MySQL/PostgreSQL applications that do not want that dependency can use the `:shared_filesystem` preset instead; it still uses Ollama but persists all stores beneath the Woods output directory.
 
@@ -154,7 +166,7 @@ When dependencies, initializers, database configuration, credentials, or schema
 
 The watcher maintains the structural index. If semantic retrieval is enabled, also run `bin/rails woods:embed_incremental` to update vectors. Without a resident watcher, run `bin/rails woods:incremental` after changes. Use a full `woods:extract` after major upgrades or when validation reports drift. CI and shared-artifact patterns are covered in [Incremental extraction](INCREMENTAL_EXTRACTION.md).
 
-On Rails 8.1, `config/ci.rb` can refresh the index before any gate that reads it: `step "Woods: refresh", "bin/rails woods:incremental"`. With the Claude Code plugin installed, an opt-in `PostToolUse` hook refreshes the index after graph-changing edits and an opt-in `SessionStart` hook warns when it predates the last commit; set `WOODS_HOOKS_ENABLED=1` to turn them on. See [Watch daemon](WATCH_DAEMON.md#hooks-for-agent-sessions).
+On Rails 8.1, `config/ci.rb` can refresh the index before any gate that reads it: `step "Woods: refresh", "bin/rails woods:incremental"`. With the Claude Code plugin installed, an opt-in `PostToolUse` hook refreshes the index after graph-changing edits and an opt-in `SessionStart` hook warns about source-content drift or unknown evidence; set `WOODS_HOOKS_ENABLED=1` to turn them on. See [Watch daemon](WATCH_DAEMON.md#hooks-for-agent-sessions).
 
 ### Enable the Console Server
 
@@ -171,7 +183,7 @@ If live-data queries are necessary, review its allowlists, blocked tables, crede
 | Rails fails during extraction | Boot and eager-load Rails with the same environment variables | [Troubleshooting](TROUBLESHOOTING.md) |
 | Validation reports missing or stale units | Run a full extraction, then validate again | [Incremental extraction](INCREMENTAL_EXTRACTION.md) |
 | MCP reports no index or zero units | Confirm `cwd`, the host-visible `tmp/woods` path, and `woods:stats` output | [MCP servers](MCP_SERVERS.md) |
-| `codebase_retrieve` says it is disabled | Configure an embedding provider and run `woods:embed`, or use `search` | [Retrieval guide](RETRIEVAL_GUIDE.md) |
+| `codebase_retrieve` says it is disabled | Choose lexical mode or configure embeddings and run `woods:embed`; `search` also works | [Retrieval guide](RETRIEVAL_GUIDE.md) |
 | Docker extraction succeeds but MCP cannot see it | Translate the container output path to its host-mounted path | [Docker setup](DOCKER_SETUP.md) |
 
 ## Where to go next

data/docs/INCREMENTAL_EXTRACTION.md

@@ -28,12 +28,24 @@ Three differences are tolerated, and nothing else:
 | Ordering inside a unit's `dependents` | Full extraction appends in extractor order, incremental in graph order. Same multiset. |
 | PageRank beyond six decimal places | Iterative floating point accumulated in each run's registration order. Scores are compared as values; only the last bits are forgiven. |
 
+The unit-file write skip ignores only Woods' top-level `extracted_at` stamp.
+A nested metadata field with the same name is application data: changing it
+rewrites the unit in both compact and pretty JSON output.
+
 `graph_analysis.json` used to be a fourth row, tolerating list ordering. It no
 longer is: the analyzer is order-independent and the oracle compares the file
 exactly. Tolerating the ordering there meant the harness, the only test that
 compares a full run against an incremental one, could not see the very
 dependence the analyzer's determinism work existed to remove.
 
+Graph targets use string identifiers even when an extractor emits a symbolic
+external target such as `:http_api`. Full and incremental runs retain every
+reverse dependency across JSON restoration and re-registration, including
+contributions from units sharing an identifier under different types. Unit
+dependency metadata retains its extractor-provided values. If an older version
+already lost reverse dependencies on these targets, run a full extraction once
+to restore them; loading the damaged graph cannot recover discarded entries.
+
 This matters most for **incremental CI chains**: restore the previous graph,
 run `woods:incremental` per merge. There, a unit that goes missing propagates
 forward run over run instead of being erased by the next full rebuild.
@@ -52,6 +64,10 @@ failure a CI chain cannot afford:
 | The range fails otherwise | Actionable error naming the range, **exit 1**. |
 | There is no `git` binary at all | Same two rows as above: the failure reads `git unavailable: …` and takes the daemon-coverage decision, rather than dying with an `Errno::ENOENT` backtrace. |
 
+Changed paths are normalized lexically before dispatch: trailing root slashes,
+duplicate separators and `.`/`..` segments do not create separate changes or
+bypass matching. Missing files remain representable; symlinks are not resolved.
+
 The range comes from `CI_COMMIT_BEFORE_SHA..CI_COMMIT_SHA` (GitLab),
 `origin/$GITHUB_BASE_REF...HEAD` (GitHub Actions), or `HEAD~1` (default). An
 unresolvable range — a GitLab zero-SHA on a new branch, an unfetched base ref,
@@ -76,6 +92,29 @@ The diff itself is rooted at the extracted application (`git -C Rails.root`),
 so it cannot read whatever checkout the process happened to start in — the
 same rooting rule the manifest's git provenance follows.
 
+Named, source-defined app modules included by runtime models are tracked as concern units even
+outside `concerns/` directories. Changing their source refreshes their includers,
+including inlined code and callback analysis. Multiple runtime mixins sharing a source
+file retain separate identities and refresh all their includers. Run a full extraction after upgrading
+to populate these previously missing source mappings.
+
+## Handled source errors and retry
+
+Unreleased after `2.0.0.beta3`: when an incremental extraction or named refresh
+records a handled consumer error (for example malformed locale or schedule
+YAML), it raises `Woods::ExtractionError` before publishing. Empty output from
+that failed consumer does not authorize replacing or deleting its last-good
+units. The published generation and source provenance remain unchanged,
+including when other files in the batch extracted successfully.
+
+Fix the source error named in the extraction log, then retry the **complete
+batch**, or the same named refresh. The watch daemon reports degraded and keeps
+the failed batch pending for retry. An error on one file does not mark a later
+successful file as failed, but the batch still cannot publish until all handled
+errors are resolved. This does not change full extraction's existing tolerance
+for handled consumer errors; its source-freshness report marks those scopes
+unverified.
+
 ## What a run does, in order
 
 `Extractor#extract_changed` is order-sensitive; each step exists because of the
@@ -87,8 +126,8 @@ step before it.
 2. **Reconcile changed paths.** Every changed path that still exists is handed
    to the file-based extractors that claim it (`PathDispatcher`), and units the
    path no longer produces are dropped. This is what indexes a file the index
-   has never seen, and what lets a task removed from a multi-task `.rake` file
-   actually go away.
+   has never seen, and what removes definitions deleted from a surviving
+   source file. Multi-file Rake tasks use wholesale reconciliation below.
 3. **Re-extract the rest of the blast radius**: units whose own file did not
    change but which depend on something that did.
 4. **Reconcile class-based types** against each extractor's
@@ -120,6 +159,11 @@ step before it.
    string literal, and an unrelated addition in the same batch do not.
    Idempotent when nothing was pruned.
 
+Git enrichment uses the same eligibility checks in full and incremental runs:
+existing app-owned files under `Rails.root`, excluding `vendor/`, `node_modules/`,
+and framework/gem source units. Each typed unit resolves its own file history,
+even when its identifier is shared by another type.
+
 Then the second pass: `dependents` and `metadata.git` are refreshed on every
 touched unit (the incremental equivalents of full extraction's phases 2 and 4),
 type indexes are regenerated, the graph, `graph_analysis.json` and the
@@ -174,13 +218,13 @@ automatically.
 | `app/decorators`, `app/presenters`, `app/form_objects` | decorators |
 | `app/managers` / `app/policies` / `app/validators` | managers / policies + pundit_policies / validators |
 | `app/**/concerns/**/*.rb` | concerns |
-| `app/models/**/*.rb` (outside `concerns/`) | poros, caching |
+| `app/models/**/*.rb` (outside `concerns/`) | poros, caching; concerns when runtime model inclusion confirms a mixin |
+| `app/**/*.rb`, `lib/**/*.rb` (outside `concerns/`) | runtime model mixins also dispatch to concerns |
 | `app/controllers/**/*.rb` | caching |
 | `app/views/**/*.erb` | view_templates, caching |
 | `config/locales/**/*.yml` | i18n |
 | `config/initializers`, `config/environments` | configurations |
 | `db/migrate/*.rb` (top level only) | migrations |
-| `lib/tasks/**/*.rake` | rake_tasks |
 | `lib/**/*.rb` (outside `tasks/`, `generators/`) | libs |
 | `spec/**/*_spec.rb`, `test/**/*_test.rb` | test_mappings |
 
@@ -191,12 +235,13 @@ serializer and decorator extractors, and all matching rules run.
 ### Wholesale re-runs
 
 `PathDispatcher.whole_app_rules` → `Extractor::WHOLE_APP_EXTRACTORS`. These
-extractors have no per-file entry point: they introspect the runtime or scan a
-whole directory in one pass. In an already-booted process re-running them is
+extractors need a complete runtime or directory view, even when a low-level
+per-file reader exists. In an already-booted process re-running them is
 cheap, which is what makes wholesale replacement the right shape.
 
 | Trigger | Re-runs |
 |---|---|
+| `lib/tasks/**/*.rake` | rake_tasks (all definitions of every task) |
 | `config/routes.rb`, `config/routes/**` | routes, engines, **and** controllers, mailers, components, view components, view templates |
 | `Gemfile.lock` | engines, middleware, rails_source (gated by `include_framework_sources`) |
 | `config/application.rb`, `config/initializers/**`, `config/environments/**` | middleware |
@@ -207,7 +252,14 @@ cheap, which is what makes wholesale replacement the right shape.
 | `db/views/**/*.sql` | database_views |
 | any `package.yml`, `packwerk.yml` | packages |
 
-Three of these deserve a note:
+Four of these deserve a note:
+
+- **Rake tasks merge definitions across files.** Any changed or deleted `.rake`
+  file reruns the task extractor over all task files. Removing the primary
+  definition preserves surviving definitions; removing a secondary definition
+  drops its source and dependencies from the shared unit. After upgrading from
+  the old per-file rules, run a full extraction to establish a source-freshness
+  baseline with the new rule fingerprint.
 
 - **Routes cascade.** `ROUTE_CONSUMER_EXTRACTORS` embed the route table, controllers write each action's routes into unit metadata and into the action
   chunks, and everything using `RouteHelperResolver` resolves navigation edges
@@ -258,6 +310,34 @@ oracle compares against emits it too, both sides agree, wrongly. The coverage
 is in `spec/extractor_spec.rb`, driving the reconciler with a shrinking
 discovery set.
 
+### Runtime removals and bundle updates
+
+Jobs discovered through `ApplicationJob.descendants` supplement the job-file
+scan, but jobs are not part of `CLASS_BASED_DISCOVERY` removal reconciliation.
+If a dynamically defined or gem-owned job disappears without a tracked source
+path changing, its unit can survive subsequent incremental runs. A full
+extraction in a fresh Rails process removes it; an in-process full extraction
+can still see an old constant retained by that process (B-165).
+
+After adding, removing, or updating bundled gems, boot the updated bundle in a
+fresh process and run:
+
+```bash
+bundle exec rake woods:extract woods:validate
+```
+
+A `Gemfile.lock` change refreshes engines, middleware, and optional framework
+sources. It does not refresh every gem-owned model, job, or other runtime unit.
+Their recorded paths or metadata can remain stale, including absolute paths to
+a removed gem version and paths under `vendor/`. A full extraction rebuilds
+those units against the installed bundle (B-166).
+
+An absent external source path can also mean the validator runs on a different
+host or mount from extraction. Confirm the bundle and filesystem context before
+rebuilding; a full run in one container does not make its gem paths visible on
+another host. Validation warnings identify missing paths, but do not prove a
+retained unit matches the currently installed gem when its path still exists.
+
 ### Deletion
 
 - Paths named in the change set that no longer exist are **authoritative** for
@@ -423,6 +503,66 @@ owns the definition of "the two indexes agree" and documents every exclusion.
 
 **Run it before and after any change to the incremental path.**
 
+## Profiling fixed costs
+
+Set `WOODS_PROFILE=1` to time extraction phases. Git enrichment and unit JSON
+finalization are separate from incremental re-extraction; runtime discovery,
+whole-app reruns and pruning appear under `reconciliation`. `payload sync`,
+`publish` (the generation pointer write), and `payload prune` (retention) are
+separate, additive phases. Older versions included sync and retention inside
+`publish`, so do not sum those older lines without subtracting nested sync.
+
+`[profile total]` reports time inside the extraction call, including setup and
+failed runs. It is not another phase to sum. Compare it with phase durations
+to find unaccounted work; each line rounds to hundredths of a second. Rails
+boot, Bundler and watch reload work outside the call require separate wall
+measurements. Do not infer that all unaccounted time is boot.
+
+Both full and incremental runs currently seed the prior payload. Cloning uses
+per-file hardlinks (or copies where links are unsupported); updated files are
+replaced atomically, preserving previous generations. Full-run carry-forward
+and generation retention remain unchanged. The seed walker classifies each entry
+once, avoiding duplicate file metadata lookups; it still links each file
+separately. Generation directories remain independent, so pruning an old generation cannot remove a newer one's files.
+Compare repeated runs on the actual index filesystem before attributing latency
+to extraction or changing the index layout.
+
+For a repeatable component comparison from a source checkout:
+
+```bash
+# BENCH_ROOT is an existing scratch parent on the index filesystem.
+# The script creates and removes only its own temporary directory there.
+BENCH_ROOT=/path/on/index/filesystem WOODS_SOURCE=/path/to/baseline \
+  ruby bench/payload_seed.rb
+BENCH_ROOT=/path/on/index/filesystem WOODS_SOURCE=/path/to/candidate \
+  ruby bench/payload_seed.rb
+```
+
+The benchmark clones 8,335 synthetic 2 KiB files across 35 directories, checks
+file counts and bytes outside timing, and reports seven samples plus medians.
+This measures the seed component only; it does not boot Rails or establish an
+end-to-end host improvement. Filesystem metadata latency and the copy fallback
+can dominate differently from local hardlink results. A first full extraction
+has no previous payload, so include a repeated full run when measuring seed cost.
+Full seeding also preserves on-demand framework units when framework extraction
+is disabled, and non-JSON files in existing type directories; dropping the seed
+wholesale would change that behavior.
+
+### Choosing full versus incremental for CI
+
+Measure repeated full and representative-day incremental runs with
+`WOODS_PROFILE=1` on the same resulting application tree, configuration and index
+filesystem. Restore the same baseline index before each incremental trial;
+otherwise a second trial may be a no-op. Include Rails boot in both wall times
+when comparing separate task invocations, and compare resident watcher cycles
+separately. Validate each resulting index with `woods:validate`.
+
+Use full extraction for that workload when its median wall time is no greater
+than the representative incremental run. There is no universal changed-file
+threshold: shared dependencies and whole-app extractor triggers change the work
+per file. Re-measure after substantial application or Woods changes. A fast leaf
+edit does not establish that a day of commits is below the crossover.
+
 ## Boundaries and open work
 
 - **Reloaded deletion is supported.** The resident watcher reloads changed