woods 1.6.1 → 2.0.0.beta2

data/docs/GETTING_STARTED.md

@@ -0,0 +1,183 @@
+# Getting started with Woods 2.0
+
+This guide takes a Rails application from no Woods installation to a validated index that an MCP client can query. The default path is structural only: it does not require OpenAI, Ollama, a vector database, or access to application records.
+
+## Before you begin
+
+Woods 2.0 requires Ruby 3.0 or later, Rails 6.0 through 8.x, a Rails environment that can boot and connect to its database, and Bundler. An MCP-capable client is only needed when an agent will query the result.
+
+The install generator adds an initializer plus a legacy compatibility migration for older/custom integrations. Shipped v2 paths do not use that migration's application tables; a new default install removes it without running it. Extraction boots Rails and writes a generated index under `tmp/woods/` by default; it does not read rows from your application's business tables.
+
+If an agent will perform the installation, use the safety and handoff checklist in [Agent setup](AGENT_SETUP.md).
+
+## 1. Install the gem
+
+Add Woods to the development group:
+
+```ruby
+# Gemfile
+group :development do
+  gem "woods", "~> 2.0"
+end
+```
+
+Then install and confirm the resolved version:
+
+```bash
+bundle install
+bundle exec ruby -rwoods/version -e 'puts Woods::VERSION'
+```
+
+If your application runs through Docker Compose, run Rails and Bundler commands inside its application service. See [Docker setup](DOCKER_SETUP.md) before configuring MCP paths.
+
+## 2. Generate and review configuration
+
+```bash
+bin/rails generate woods:install
+```
+
+Review the generated files:
+
+- `config/initializers/woods.rb` documents supported configuration;
+- `db/migrate/*_create_woods_tables.rb` is a legacy application migration for `woods_units`, `woods_edges`, and `woods_embeddings`.
+
+**For a new default installation, do not run that migration.** Woods 2's shipped structural index and storage backends do not read or write those application tables. Remove it before the next Rails boot. Keep and run it only when deliberately preserving an older/custom integration that uses the tables; first check for name conflicts and obtain normal migration approval.
+
+The generated initializer defaults are enough for structural extraction. Do not choose a storage preset or configure an embedding provider unless you want semantic search.
+
+## 3. Extract the application
+
+```bash
+bin/rails woods:extract
+```
+
+Woods boots and eager-loads the Rails application, runs its extractors, builds dependency edges, and publishes one complete generation under `tmp/woods/`. A reader stays on the previous complete generation until the new one is published.
+
+If Rails only boots with environment variables, provide the same variables here. Do not work around a boot failure inside Woods configuration. First confirm that the application can boot and eager-load with the same environment:
+
+```bash
+bin/rails runner 'puts Rails.application.class.name'
+bin/rails runner 'Rails.application.eager_load!; puts "eager load ok"'
+```
+
+## 4. Validate and inspect the index
+
+Use the Woods tasks rather than depending on internal filenames:
+
+```bash
+bin/rails woods:validate
+bin/rails woods:stats
+```
+
+Validation should finish successfully. Statistics should report non-zero units for the types your application contains. The exact distribution varies by application.
+
+The generated directory is disposable build output. Add `tmp/woods/` to `.gitignore` unless your team deliberately publishes it as an artifact.
+
+## 5. Connect the Index Server
+
+The Index Server reads the generated index. It does not boot Rails and does not query application records.
+
+For a project-scoped stdio MCP client, add the equivalent server entry at the application root (for clients that support `.mcp.json`, use this shape):
+
+```json
+{
+  "mcpServers": {
+    "woods": {
+      "command": "bundle",
+      "args": ["exec", "woods-mcp-start", "./tmp/woods"],
+      "cwd": "/absolute/path/to/your-rails-app"
+    }
+  }
+}
+```
+
+Use an absolute `cwd`. It ensures Bundler resolves the application's Woods version and makes the relative index path unambiguous. Reconnect or restart the MCP client after changing its configuration.
+
+For client-specific configuration locations, Docker, direct `woods-mcp`, and HTTP transport, use [MCP servers](MCP_SERVERS.md).
+
+## 6. Verify from the client
+
+Ask the client to call these tools in order:
+
+1. `woods_status` — confirm the index path, generation, unit counts, and retrieval status.
+2. `search` with a known class name — discover its exact Woods identifier.
+3. `lookup` with that identifier — inspect its runtime source, metadata, and relationships.
+4. `dependents` with that identifier — see what may be affected by a change.
+
+For example:
+
+> Use Woods to find `Order`, inspect its resolved callbacks and associations, and list the first two levels of code that depend on it. Cite the Woods identifiers you used.
+
+The Index schema inventory totals 29 schemas. Fourteen register as tools in a normal packaged launch; `codebase_retrieve` is among them but returns a configuration error until embeddings are enabled. The other structural tools work immediately. See [Agent guide](AGENT_GUIDE.md) for a reliable query workflow.
+
+## Optional next steps
+
+### Add semantic search
+
+Structural search, exact lookup, dependency traversal, graph analysis, and flow tracing do not need embeddings. Add embeddings only when agents need natural-language retrieval.
+
+The local preset uses SQLite metadata, persisted in-memory vectors, and a local Ollama service. Add `gem "sqlite3"` to the application bundle if it is not already present. MySQL/PostgreSQL applications that do not want that dependency can use the `:shared_filesystem` preset instead; it still uses Ollama but persists all stores beneath the Woods output directory.
+
+```ruby
+# config/initializers/woods.rb
+Woods.configure_with_preset(:local)
+```
+
+Install and start Ollama, then pull the default model and build embeddings:
+
+```bash
+ollama pull nomic-embed-text
+bin/rails woods:embed
+```
+
+For dense Ruby source, add `gem "tokenizers", "~> 0.5"` for exact WordPiece token counting. Without it, Woods falls back to character-based estimation, which can over-pack some Ollama chunks.
+
+Reconnect the MCP server and check `woods_status`. For OpenAI, pgvector, Qdrant, model dimensions, and provider changes, read the [Retrieval guide](RETRIEVAL_GUIDE.md) and [Backend matrix](BACKEND_MATRIX.md).
+
+### Keep the index current
+
+For automatic maintenance, keep a watcher running beside the Rails development process:
+
+```bash
+bin/rails woods:watch
+```
+
+```text
+# Procfile.dev
+web:   bin/rails server
+woods: bundle exec rake woods:watch
+```
+
+On startup it reconciles changes made since the last successful generation. While running it batches file events, reloads Rails code when safe, extracts affected units, and publishes atomically. The Index Server detects the new generation on its next call and reloads automatically. After the initial extraction, ordinary code edits need no manual extraction or MCP restart.
+
+When dependencies, initializers, database configuration, credentials, or schema change, Rails cannot safely reload all captured state. The watcher records a degraded reason and exits with status 75 so the process manager can restart it. Docker bind mounts may require polling; follow [Watch daemon](WATCH_DAEMON.md).
+
+The watcher maintains the structural index. If semantic retrieval is enabled, also run `bin/rails woods:embed_incremental` to update vectors. Without a resident watcher, run `bin/rails woods:incremental` after changes. Use a full `woods:extract` after major upgrades or when validation reports drift. CI and shared-artifact patterns are covered in [Incremental extraction](INCREMENTAL_EXTRACTION.md).
+
+On Rails 8.1, `config/ci.rb` can refresh the index before any gate that reads it: `step "Woods: refresh", "bin/rails woods:incremental"`. With the Claude Code plugin installed, an opt-in `PostToolUse` hook refreshes the index after graph-changing edits and an opt-in `SessionStart` hook warns when it predates the last commit; set `WOODS_HOOKS_ENABLED=1` to turn them on. See [Watch daemon](WATCH_DAEMON.md#hooks-for-agent-sessions).
+
+### Enable the Console Server
+
+The Console Server is separate from the Index Server. It boots Rails and can read live model data. Do not enable it merely to inspect code structure.
+
+If live-data queries are necessary, review its allowlists, blocked tables, credential scanning, redaction, SQL validation, and environment boundary in [Console MCP setup](CONSOLE_MCP_SETUP.md). The default surface is nine tools; `console_sql` and `console_query` require an explicit read-tools opt-in.
+
+## First-run problems
+
+| Symptom | Check first | Continue with |
+|---|---|---|
+| Bundler cannot resolve Woods | Ruby/Rails requirements and the lockfile's selected gem version | [Upgrade guide](UPGRADING_TO_2.md) |
+| Generator reports existing files | Diff the existing initializer and migration; do not overwrite blindly | [Configuration reference](CONFIGURATION_REFERENCE.md) |
+| Rails fails during extraction | Boot and eager-load Rails with the same environment variables | [Troubleshooting](TROUBLESHOOTING.md) |
+| Validation reports missing or stale units | Run a full extraction, then validate again | [Incremental extraction](INCREMENTAL_EXTRACTION.md) |
+| MCP reports no index or zero units | Confirm `cwd`, the host-visible `tmp/woods` path, and `woods:stats` output | [MCP servers](MCP_SERVERS.md) |
+| `codebase_retrieve` says it is disabled | Configure an embedding provider and run `woods:embed`, or use `search` | [Retrieval guide](RETRIEVAL_GUIDE.md) |
+| Docker extraction succeeds but MCP cannot see it | Translate the container output path to its host-mounted path | [Docker setup](DOCKER_SETUP.md) |
+
+## Where to go next
+
+- [Agent guide](AGENT_GUIDE.md): teach an agent to use the index effectively.
+- [MCP servers](MCP_SERVERS.md): client configuration and exact callable surfaces.
+- [Configuration reference](CONFIGURATION_REFERENCE.md): supported settings and defaults.
+- [Upgrade to Woods 2.0](UPGRADING_TO_2.md): migrate an existing 1.x installation.
+- [Troubleshooting](TROUBLESHOOTING.md): diagnose extraction, storage, MCP, and Docker failures.

data/docs/INCREMENTAL_EXTRACTION.md

@@ -0,0 +1,455 @@
+# Incremental Extraction
+
+`woods:incremental` re-indexes only what changed. This page states the
+correctness contract it is held to, the inventory of which path class triggers
+which work, how to run the differential harness that enforces the contract, and
+what is still out of scope.
+
+Background: [#164](https://github.com/lost-in-the/woods/issues/164).
+
+## The contract
+
+> After any sequence of file create / modify / delete / rename operations, an
+> index maintained purely by `extract_changed` is **indistinguishable from a
+> cold `extract_all` of the same tree**.
+
+Indistinguishable means: the same unit identifiers, the same per-unit JSON
+content, the same `_index.json` per type, the same graph nodes / edges /
+reverse edges / file map / type index / stats, PageRank recomputed, the
+same manifest counts and `graph_analysis.json` — and, when flow
+precomputation is enabled, the same `flows/flow_index.json`, the same
+flow documents, and the same flow annotations on controller units.
+
+Three differences are tolerated, and nothing else:
+
+| Tolerated | Why |
+|---|---|
+| Wall-clock stamps (`extracted_at`, `generated_at`, and the digest over it) | A unit an incremental run correctly left alone keeps an older stamp. |
+| Ordering inside a unit's `dependents` | Full extraction appends in extractor order, incremental in graph order. Same multiset. |
+| PageRank beyond six decimal places | Iterative floating point accumulated in each run's registration order. Scores are compared as values; only the last bits are forgiven. |
+
+`graph_analysis.json` used to be a fourth row, tolerating list ordering. It no
+longer is: the analyzer is order-independent and the oracle compares the file
+exactly. Tolerating the ordering there meant the harness, the only test that
+compares a full run against an incremental one, could not see the very
+dependence the analyzer's determinism work existed to remove.
+
+This matters most for **incremental CI chains**: restore the previous graph,
+run `woods:incremental` per merge. There, a unit that goes missing propagates
+forward run over run instead of being erased by the next full rebuild.
+
+## Exit behavior in CI chains
+
+The task decides what to do before it extracts, and the exit code is part of
+that decision — a green job that silently skipped the sync is exactly the
+failure a CI chain cannot afford:
+
+| Situation | Behavior |
+|---|---|
+| `CHANGED_FILES` is set | Used verbatim (comma-separated paths); git is not consulted. |
+| The git range resolves | Current behavior: extract the changed paths, or exit 0 with `No relevant files changed` when nothing relevant changed. |
+| The range fails **and** a `:running` watch daemon maintains the index | Stand down with a printed reason, exit 0 — the daemon's start-up catch-up covers whatever changed. |
+| The range fails otherwise | Actionable error naming the range, **exit 1**. |
+| There is no `git` binary at all | Same two rows as above: the failure reads `git unavailable: …` and takes the daemon-coverage decision, rather than dying with an `Errno::ENOENT` backtrace. |
+
+The range comes from `CI_COMMIT_BEFORE_SHA..CI_COMMIT_SHA` (GitLab),
+`origin/$GITHUB_BASE_REF...HEAD` (GitHub Actions), or `HEAD~1` (default). An
+unresolvable range — a GitLab zero-SHA on a new branch, an unfetched base ref,
+a shallow clone with no `HEAD~1` — reads as "nothing changed" to git, which is
+why a failed range must not be mistaken for an empty one: the sync never ran,
+and CI drift would stay unbounded. A degraded daemon covers nothing, so it
+does not stand the run down. `WOODS_IGNORE_WATCH=1` removes daemon coverage
+too — with it set, a failed range exits 1. A slim image with no `git` binary
+resolves to the same decision rather than a raw `Errno::ENOENT`: the failure is
+reported as `git unavailable: …`, so a daemon-covered tree still stands down
+and an uncovered one still gets the remediation text.
+
+Recovery choices, in the order they are worth trying:
+
+1. Repair or provide the range: fetch the base ref (`fetch-depth: 2` or more),
+   or correct the CI environment variables that build it.
+2. Set `CHANGED_FILES` explicitly from your CI platform, bypassing git range
+   resolution entirely.
+3. Run a full `woods:extract` when the range cannot be repaired this run.
+
+The diff itself is rooted at the extracted application (`git -C Rails.root`),
+so it cannot read whatever checkout the process happened to start in — the
+same rooting rule the manifest's git provenance follows.
+
+## What a run does, in order
+
+`Extractor#extract_changed` is order-sensitive; each step exists because of the
+step before it.
+
+1. **Blast radius** from the *pre-change* graph, so dependents of a file that
+   just disappeared still get re-extracted. Unbounded by default; see
+   [Bounding the blast radius](#bounding-the-blast-radius).
+2. **Reconcile changed paths.** Every changed path that still exists is handed
+   to the file-based extractors that claim it (`PathDispatcher`), and units the
+   path no longer produces are dropped. This is what indexes a file the index
+   has never seen, and what lets a task removed from a multi-task `.rake` file
+   actually go away.
+3. **Re-extract the rest of the blast radius**: units whose own file did not
+   change but which depend on something that did.
+4. **Reconcile class-based types** against each extractor's
+   `#discoverable_classes`, classes added since the last extraction, and
+   classes the graph still holds that the set no longer contains. Exact by
+   construction: it is the same discovery code a full extraction uses, so
+   there is no path-to-constant guessing.
+5. **Re-run whole-app extractors** whose trigger paths changed, replacing that
+   unit type wholesale.
+6. **Prune vanished units**, so anything steps 2–5 resurrected against a
+   deleted file is swept in the same run rather than surviving as a ghost.
+7. **Reconcile class-based types once more**, because pruning can un-know a
+   class the first pass skipped: a class-based file moved between autoload
+   directories with its constant unchanged still looks known when step 4 runs,
+   so it is not re-extracted, and step 6 then removes it for its vanished old
+   path. This pass re-adds it in the same run instead of waiting for some later
+   run to notice. It skips everything else step 6 pruned (`except:`), because
+   without a reload a constant outlives the file that defined it, otherwise
+   deleting `app/models/user.rb` would prune `User` only for this pass to find
+   it still in `ActiveRecord::Base.descendants` and re-register it against a
+   path nothing can ever remove again. What separates the two shapes is
+   loader-derived constant identity, not a textual class-name match: a pruned
+   identifier is re-added only when the active Zeitwerk loader governs a
+   changed file for exactly that constant (`cpath_expected_at` — the loader's
+   inflector, ignores, and root namespaces decide — and the file declares it).
+   A loader non-claim is authoritative: an unmanaged or declined path re-adds
+   nothing. So a moved file whose governed constant matches qualifies, while
+   another namespace's same-demodulized file, a mention in a comment or
+   string literal, and an unrelated addition in the same batch do not.
+   Idempotent when nothing was pruned.
+
+Then the second pass: `dependents` and `metadata.git` are refreshed on every
+touched unit (the incremental equivalents of full extraction's phases 2 and 4),
+type indexes are regenerated, the graph, `graph_analysis.json` and the
+manifest are written, and — with flow precomputation enabled — the run's
+controller delta gets the same flow treatment a full run gives (see
+[Flow artifacts](#flow-artifacts)).
+
+A run that changed nothing **does not rewrite the manifest**. The manifest
+timestamp drives `woods_status.staleness_seconds`, and touching it after a no-op
+would report the index as freshly synced when nothing was re-read.
+
+## Bounding the blast radius
+
+`incremental_blast_radius_depth` caps how many reverse hops step 1 walks.
+`nil`, the default, keeps the unbounded transitive closure: every unit that
+reaches the changed file, at any depth, is re-extracted.
+
+The reason to cap it is cost. A unit's extracted content is mostly a function
+of its own source and its own reflection, so on most graphs the deep half of
+the closure re-derives bytes that do not change. On a 200-service chain in the
+dummy app, one leaf edit re-extracts 200 units unbounded and 2 at a depth
+of 1.
+
+The reason the default is not capped is that "mostly" is not "always". An STI
+grandchild reads its grandparent's reflection: `SportsCar < Car < Vehicle`
+inherits `Vehicle`'s associations, validations and callback chain, and a
+nested `has_many :through` resolves through the same kind of chain. The graph
+records only the one-hop superclass reference each source file mentions, so
+that grandchild sits two hops out while its content depends on hop zero. Set
+the key on a tree you know has neither shape.
+
+What the cap never affects is `dependents`. Edges change only when a unit is
+re-extracted, and the run marks every target of a re-extracted unit's edges,
+before and after registration, so a unit that gains or loses an inbound edge
+is rewritten by the second pass whether or not the walk reached it.
+`spec/integration/incremental_equivalence_spec.rb` holds a depth of 1 to
+full-extraction equivalence, including that case.
+
+## Dispatch inventory
+
+### Per-file
+
+Routed by `PathDispatcher.file_rules`. Rules reference each extractor's own
+`*_DIRECTORIES` constant, so adding a directory there flows through
+automatically.
+
+| Path class | Extractor |
+|---|---|
+| `app/services`, `app/interactors`, `app/operations`, `app/commands`, `app/use_cases` | services |
+| `app/jobs`, `app/workers`, `app/sidekiq` | jobs |
+| `app/serializers`, `app/blueprinters`, `app/decorators` | serializers |
+| `app/decorators`, `app/presenters`, `app/form_objects` | decorators |
+| `app/managers` / `app/policies` / `app/validators` | managers / policies + pundit_policies / validators |
+| `app/**/concerns/**/*.rb` | concerns |
+| `app/models/**/*.rb` (outside `concerns/`) | poros, caching |
+| `app/controllers/**/*.rb` | caching |
+| `app/views/**/*.erb` | view_templates, caching |
+| `config/locales/**/*.yml` | i18n |
+| `config/initializers`, `config/environments` | configurations |
+| `db/migrate/*.rb` (top level only) | migrations |
+| `lib/tasks/**/*.rake` | rake_tasks |
+| `lib/**/*.rb` (outside `tasks/`, `generators/`) | libs |
+| `spec/**/*_spec.rb`, `test/**/*_test.rb` | test_mappings |
+
+A path can match several rules, `app/policies` is claimed by both
+`PolicyExtractor` and `PunditExtractor`, `app/decorators` by both the
+serializer and decorator extractors, and all matching rules run.
+
+### Wholesale re-runs
+
+`PathDispatcher.whole_app_rules` → `Extractor::WHOLE_APP_EXTRACTORS`. These
+extractors have no per-file entry point: they introspect the runtime or scan a
+whole directory in one pass. In an already-booted process re-running them is
+cheap, which is what makes wholesale replacement the right shape.
+
+| Trigger | Re-runs |
+|---|---|
+| `config/routes.rb`, `config/routes/**` | routes, engines, **and** controllers, mailers, components, view components, view templates |
+| `Gemfile.lock` | engines, middleware, rails_source (gated by `include_framework_sources`) |
+| `config/application.rb`, `config/initializers/**`, `config/environments/**` | middleware |
+| `config/recurring.yml`, `config/sidekiq_cron.yml`, `config/schedule.rb` | scheduled_jobs |
+| `app/models/**/*.rb` | state_machines |
+| `app/**/*.rb` | events |
+| `spec/factories/**`, `test/factories/**` | factories |
+| `db/views/**/*.sql` | database_views |
+| any `package.yml`, `packwerk.yml` | packages |
+
+Three of these deserve a note:
+
+- **Routes cascade.** `ROUTE_CONSUMER_EXTRACTORS` embed the route table, controllers write each action's routes into unit metadata and into the action
+  chunks, and everything using `RouteHelperResolver` resolves navigation edges
+  against it. The graph cannot express this, because a route unit depends *on*
+  its controller, not the other way round, so walking dependents from
+  `config/routes.rb` never reaches them.
+- **Database views are wholesale, not per file.** Scenic keeps only the highest
+  `_vNN` of each view, so pointing the per-file method at
+  `db/views/foo_v01.sql` would index a version a full extraction drops.
+- **Packages don't yet claim their members.** `PackageExtractor` (#280) only
+  produces `package` units from `package.yml`; it does not annotate which
+  package every other unit belongs to. A pack-resident file-based unit is not
+  discovered by `PathDispatcher` through its package boundary today
+  (follow-up B-175).
+
+### Class-based types
+
+Models, controllers, mailers, components, view components and channels are
+**not** dispatched by path. They are reconciled against
+`#discoverable_classes` on their own extractor, in **both** directions:
+additions are that set minus the graph, removals are the graph minus that set.
+
+For all six, `extract_all` is literally `discoverable_classes.map { … }.compact`,
+so absence from the set is exactly "a full extraction would not produce this", the equivalence the incremental path is held to.
+
+Removal is gated on the eager load having **completed**, and that gate carries
+the whole safety argument:
+
+- **A partial eager load.** The documented `NameError` fallback loads only
+  `EXTRACTION_DIRECTORIES`, so descendants are known-incomplete and the
+  difference would be most of the app. Deleting by the type is far worse than a
+  stale unit, so a partial load removes nothing.
+- **A constant outliving its file.** A resident daemon that has not reloaded
+  still holds a deleted class as a descendant, so it is *in* the set and not
+  stale, correct for that process. The subsequent reload is what makes it
+  removable.
+
+Without this, a class deleted from a file that still exists was never removed
+at all: path-keyed deletion sees no missing path, and a class-based unit
+records a *convention* path from its constant name, so a second model in one
+`.rb` was never attributed to the file it actually lived in. The unit outlived
+every subsequent incremental run.
+
+The booted harness cannot cover that case. Zeitwerk unloads only the constant a
+file is *expected* to define, so a class defined there as a side effect survives
+the reload, stays in `descendants`, and the in-process full extraction the
+oracle compares against emits it too, both sides agree, wrongly. The coverage
+is in `spec/extractor_spec.rb`, driving the reconciler with a shrinking
+discovery set.
+
+### Deletion
+
+- Paths named in the change set that no longer exist are **authoritative** for
+  any unit type. This covers deleted models and the old side of a rename.
+- A **sweep** over registered paths catches callers whose change set is
+  incomplete (a git diff that omits deletions, a missed unlink, a branch
+  switch). Being a heuristic, it is bounded twice:
+  - **To paths a file rule claims.** Some units name a *nominal* path rather
+    than a source file, `BehavioralProfile` names `config/application.rb`,
+    which no rule claims.
+  - **Away from class-based units entirely.** A class-based unit records a
+    convention path when its source location can't be resolved, and that path
+    need not exist. On Rails < 7.1, `ActiveRecord::SchemaMigration` and
+    `ActiveRecord::InternalMetadata` are real `ActiveRecord::Base` descendants
+    whose convention path (`app/models/active_record/schema_migration.rb`) no
+    application has, and *is* claimed by the PORO rule, so the first bound
+    doesn't cover it.
+
+  Sweeping either would delete units a full extraction still produces.
+- Only paths under `Rails.root` are considered either way: framework units point
+  at gem paths, and an index restored from a CI artifact can carry paths
+  produced under a different root.
+
+## Flow artifacts
+
+Everything in this section is gated on `precompute_flows` (default false).
+The family has three parts: `flows/flow_index.json` (entry point → relative
+document path), one document per controller action, and
+`metadata[:flow_paths]` on the controller units.
+
+A full extraction computes all three in one pass. An incremental run computes
+them for its **delta**:
+
+- **Re-extracted controllers** get their `metadata[:flow_paths]` back, their
+  flow documents are re-assembled from the units on disk, and their entries
+  replace whatever the previous index held for them — so an action removed
+  from a re-extracted controller leaves the index even though the file still
+  exists.
+- **Controllers the run pruned** (deleted or renamed) leave the index
+  entirely.
+- **Untouched controllers' entries carry forward** from the previous
+  generation, which payload seeding hardlinks into the run's payload
+  directory.
+
+Re-assembly is scoped further, because a flow document only reaches
+`FlowPrecomputer::DEFAULT_MAX_DEPTH` units. The run walks the pre-change graph
+to that same depth from its changed files; a re-extracted controller inside
+that radius is re-assembled, and one outside it takes its
+`metadata[:flow_paths]` back from the previous index without paying for the
+assembly. Three cases opt out and re-assemble every re-extracted controller: a
+targeted `Extractor#refresh`, which has no change set; a routes re-run, which
+replaces every controller and moves the route a flow document carries without
+touching any dependency edge; and a controller whose action set no longer
+matches the previous index, which is how an action inherited from further up a
+controller chain than the radius reaches still lands.
+
+After the index is rewritten, a **dedicated flow-artifact sweep** removes
+every `flows/` document no index entry references. It validates against
+`flow_index.json` and is deliberately separate from the unit sweep: flows/
+holds neither units nor an `_index.json`, so the unit sweep's in-memory
+contract does not describe it. The whole refresh is **fail closed**: a
+genuinely absent family (no `flows/` directory, or an empty one — typically
+an index built while the gate was off) skips the refresh, but a family that
+holds any artifact is authoritative. A missing `flow_index.json` among
+documents, a corrupt one, a failed rehydration, write, patch, or sweep
+raises, and the raise aborts the run **before** the generation publish —
+no generation bump, the preceding generation stays resolved and readable.
+`woods:validate` applies the same corruption rule (a populated family
+without its index is an error) and never demands `_index.json` from
+`flows/`. The full extraction path is fail closed too: `precompute_flows`
+raises before the manifest and generation publish.
+
+## Refreshing one extractor on demand
+
+`Extractor#refresh` re-runs named extractors wholesale against an
+already-booted app. Incremental runs reach the whole-app extractors by trigger
+path; this reaches them by name, for a caller that already knows what went
+stale.
+
+```ruby
+# After editing config/routes.rb, or from a resident process that just reloaded
+Woods::Extractor.new(output_dir: "tmp/woods").refresh(:routes)
+# => { types: [:routes, :controllers, :mailers, ...], touched: [...], unknown: [] }
+```
+
+```bash
+bundle exec rake "woods:refresh[routes]"
+bundle exec rake "woods:refresh[state_machines,factories]"
+bundle exec rake woods:refresh          # lists the valid keys
+```
+
+Any extractor key works, not only the whole-app ones, `refresh(:models)` is a
+legitimate way to re-derive every model after a schema change. A routes refresh
+cascades to `ROUTE_CONSUMER_EXTRACTORS` for the reason given above. Like an
+incremental run, `refresh` rewrites the graph, `graph_analysis.json`, the
+affected type indexes and the manifest, so the result is durable.
+
+## What a change actually requires: reload, restart, or neither
+
+`Woods::ReloadPolicy` answers the question a resident process has to ask before
+re-extracting: extraction reads the *runtime*, so "the file changed" is not the
+same question as "what has to happen before re-reading it is worth anything".
+
+| Action | Path classes | Why |
+|---|---|---|
+| `:reextract` | `config/locales/**`, `db/migrate/**`, `db/views/**`, `lib/tasks/**`, `spec/**`, `test/**`, `app/views/**` (non-Ruby), schedule files, `package.yml`, `packwerk.yml` | Woods reads bytes. No constant involved. |
+| `:reload` | `app/**/*.rb`, `lib/**/*.rb` (outside `tasks/`, `generators/`), `config/routes.rb`, `config/routes/**` | An autoloaded constant changed; introspecting the old class would be a lie. |
+| `:restart` | `Gemfile`, `Gemfile.lock`, `.ruby-version`, `.env*`, application/boot/environment files, initializers/environments/credentials, database/schema files, `config/settings*.yml`, and boot-captured service YAML | Captured at boot. Rails' reloader re-runs none of it. See the exact list below. |
+| `:ignore` | everything else | Not extraction input. |
+
+The `:restart` set is drawn generously on purpose. Rails' reloader replaces
+autoloaded constants and nothing else, it does not re-run initializers,
+re-resolve `Rails.application.config`, or rebuild the schema cache, all of
+which Woods captures (`BehavioralProfile`, `MiddlewareExtractor`, model column
+data). `rails/spring`'s staleness bugs came from under-scoping exactly this
+set.
+
+The exact additional boot-captured YAML set is `config/settings.yml`,
+`config/settings/*.yml`, and `config/{cable,storage,sidekiq,puma,cache,queue}.yml`
+(including `.yaml` spellings). Scheduled-job sources such as
+`config/recurring.yml` and `config/sidekiq_cron.yml` remain `:reextract` inputs,
+not restart triggers. `lib/woods/reload_policy.rb` is authoritative.
+
+Two version-sensitive behaviours sit *behind* the classification rather than in
+it, and belong to whoever implements the reload step:
+
+- `ActiveSupport::DescendantsTracker` internals changed across Rails 6.0–8.x, so
+  a reload can leave stale entries in a descendants set. Discovery-based
+  extraction must re-read descendants *after* the reload completes, never
+  across it.
+- A schema change needs `reset_column_information` plus schema-cache
+  invalidation to become visible. It is classified `:restart` rather than
+  `:reload` because getting that right in-process is subtle and schema changes
+  are rare.
+
+`Watch::Daemon` consumes the policy on every cycle: `classify_all` decides what
+the batch demands, and `paths_requiring(:restart)` names the offending paths in
+the restart message a supervisor sees. See `docs/WATCH_DAEMON.md`.
+
+## Running the differential harness
+
+`spec/integration/incremental_equivalence_spec.rb` is the oracle. It boots the
+`spec/dummy` app against a tmpdir copy, applies randomized
+create/modify/delete/rename sequences, and compares the maintained index to a
+cold full extraction at every step. It runs in CI on every Rails-matrix row.
+
+```bash
+# CI defaults: 60 operations x 3 seeds
+WOODS_RUN_BOOTED_APP=1 BUNDLE_GEMFILE=gemfiles/rails_8.0.gemfile \
+  bundle exec rspec spec/integration/incremental_equivalence_spec.rb
+
+# Soak run
+WOODS_RUN_BOOTED_APP=1 BUNDLE_GEMFILE=gemfiles/rails_8.0.gemfile \
+  WOODS_DIFF_OPS=1000 WOODS_DIFF_SEEDS=1,2,3,4,5 \
+  bundle exec rspec spec/integration/incremental_equivalence_spec.rb -e randomized
+
+# When a truncated delta isn't enough to see what moved
+WOODS_DIFF_BRIEF=4000 ...
+```
+
+Seeds are fixed, so a failure reproduces. `spec/support/index_comparison.rb`
+owns the definition of "the two indexes agree" and documents every exclusion.
+
+**Run it before and after any change to the incremental path.**
+
+## Boundaries and open work
+
+- **Reloaded deletion is supported.** The resident watcher reloads changed
+  constants before discovery-set reconciliation, so deleting a class-backed
+  file removes its unit. A caller using the lower-level incremental API without
+  the watcher must ensure the Rails runtime has been reloaded first.
+- **Identifier identity is typed.** Units of different types that share an
+  identifier remain separate graph nodes. Two source files that produce the
+  same type+identifier are not representable; full extraction fails closed
+  with both source paths instead of publishing a glob-order tie-break (resolved
+  B-063). Same-file re-derivation remains a legitimate deduplication case.
+- **Class-based units are never swept**: see [Deletion](#deletion) above for
+  why (the `SchemaMigration`/`InternalMetadata` convention-path case).
+  Deleting a class-based unit therefore requires either the caller naming the
+  path or the discovery-set reconciliation above; the sweep never infers it.
+- **Git metadata for untouched units.** An incremental run refreshes
+  `metadata.git` on the units it wrote. A unit nothing touched keeps the git
+  metadata from the last run that did, which goes stale as commits land on
+  other files. The same holds for the node attributes `commit_count` and
+  `change_frequency` that feed the `volatile_dependencies` report.
+- **Snapshots stay full-extraction-only.** They hash the full unit set, and an
+  incremental run only holds changed units in memory.
+- **A divergence floor is still worth keeping.** Incremental correctness is a
+  ratchet, not a proof: schedule a periodic full extraction and gate on
+  `woods:validate` so any undiscovered drift has a bounded lifetime.
+- **Phases 1–4** of #164, a public single-extractor re-run API
+  (`Extractor#refresh`), the resident `woods:watch` daemon, an MCP freshness
+  contract, and multi-worktree operation, all landed alongside this work
+  (B-064, resolved). `docs/WATCH_DAEMON.md` covers them, including the parts
+  that remain unmeasured.