woods 2.0.0.beta2 → 2.0.0.beta3

data/docs/GETTING_STARTED.md

@@ -12,7 +12,19 @@ If an agent will perform the installation, use the safety and handoff checklist
 
 ## 1. Install the gem
 
-Add Woods to the development group:
+Use the [README release table](../README.md) to choose a version, then confirm
+that **exact version is published** on the [RubyGems versions page](https://rubygems.org/gems/woods/versions)
+before editing the Gemfile. A prepared release checkout can update the README
+before its gem is published; if the version is absent, choose an available
+version or wait for publication.
+
+If the published 2.x line has only beta or release-candidate versions, use an
+exact pin to the published prerelease, following the README's prerelease
+instructions; `~> 2.0` does not select prereleases. Follow the selected version's
+tag documentation. The `main` guides may describe features absent from the
+published gem.
+
+Once a stable 2.x release is published, add Woods to the development group with:
 
 ```ruby
 # Gemfile
@@ -108,13 +120,13 @@ For example:
 
 > Use Woods to find `Order`, inspect its resolved callbacks and associations, and list the first two levels of code that depend on it. Cite the Woods identifiers you used.
 
-The Index schema inventory totals 29 schemas. Fourteen register as tools in a normal packaged launch; `codebase_retrieve` is among them but returns a configuration error until embeddings are enabled. The other structural tools work immediately. See [Agent guide](AGENT_GUIDE.md) for a reliable query workflow.
+The Index schema inventory totals 29 schemas. Fourteen register as tools in a normal packaged launch; `codebase_retrieve` is among them but needs embeddings in the default semantic mode, or explicit [lexical retrieval](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval) over extraction output. The other structural tools work immediately. See [Agent guide](AGENT_GUIDE.md) for a reliable query workflow.
 
 ## Optional next steps
 
 ### Add semantic search
 
-Structural search, exact lookup, dependency traversal, graph analysis, and flow tracing do not need embeddings. Add embeddings only when agents need natural-language retrieval.
+Structural search, exact lookup, dependency traversal, graph analysis, and flow tracing do not need embeddings. For ranked natural-language retrieval, choose explicit [lexical mode](RETRIEVAL_GUIDE.md#embedding-free-lexical-retrieval) without providers, or configure embeddings for semantic matching.
 
 The local preset uses SQLite metadata, persisted in-memory vectors, and a local Ollama service. Add `gem "sqlite3"` to the application bundle if it is not already present. MySQL/PostgreSQL applications that do not want that dependency can use the `:shared_filesystem` preset instead; it still uses Ollama but persists all stores beneath the Woods output directory.
 
@@ -154,7 +166,7 @@ When dependencies, initializers, database configuration, credentials, or schema
 
 The watcher maintains the structural index. If semantic retrieval is enabled, also run `bin/rails woods:embed_incremental` to update vectors. Without a resident watcher, run `bin/rails woods:incremental` after changes. Use a full `woods:extract` after major upgrades or when validation reports drift. CI and shared-artifact patterns are covered in [Incremental extraction](INCREMENTAL_EXTRACTION.md).
 
-On Rails 8.1, `config/ci.rb` can refresh the index before any gate that reads it: `step "Woods: refresh", "bin/rails woods:incremental"`. With the Claude Code plugin installed, an opt-in `PostToolUse` hook refreshes the index after graph-changing edits and an opt-in `SessionStart` hook warns when it predates the last commit; set `WOODS_HOOKS_ENABLED=1` to turn them on. See [Watch daemon](WATCH_DAEMON.md#hooks-for-agent-sessions).
+On Rails 8.1, `config/ci.rb` can refresh the index before any gate that reads it: `step "Woods: refresh", "bin/rails woods:incremental"`. With the Claude Code plugin installed, an opt-in `PostToolUse` hook refreshes the index after graph-changing edits and an opt-in `SessionStart` hook warns about source-content drift or unknown evidence; set `WOODS_HOOKS_ENABLED=1` to turn them on. See [Watch daemon](WATCH_DAEMON.md#hooks-for-agent-sessions).
 
 ### Enable the Console Server
 
@@ -171,7 +183,7 @@ If live-data queries are necessary, review its allowlists, blocked tables, crede
 | Rails fails during extraction | Boot and eager-load Rails with the same environment variables | [Troubleshooting](TROUBLESHOOTING.md) |
 | Validation reports missing or stale units | Run a full extraction, then validate again | [Incremental extraction](INCREMENTAL_EXTRACTION.md) |
 | MCP reports no index or zero units | Confirm `cwd`, the host-visible `tmp/woods` path, and `woods:stats` output | [MCP servers](MCP_SERVERS.md) |
-| `codebase_retrieve` says it is disabled | Configure an embedding provider and run `woods:embed`, or use `search` | [Retrieval guide](RETRIEVAL_GUIDE.md) |
+| `codebase_retrieve` says it is disabled | Choose lexical mode or configure embeddings and run `woods:embed`; `search` also works | [Retrieval guide](RETRIEVAL_GUIDE.md) |
 | Docker extraction succeeds but MCP cannot see it | Translate the container output path to its host-mounted path | [Docker setup](DOCKER_SETUP.md) |
 
 ## Where to go next

data/docs/INCREMENTAL_EXTRACTION.md

@@ -28,12 +28,24 @@ Three differences are tolerated, and nothing else:
 | Ordering inside a unit's `dependents` | Full extraction appends in extractor order, incremental in graph order. Same multiset. |
 | PageRank beyond six decimal places | Iterative floating point accumulated in each run's registration order. Scores are compared as values; only the last bits are forgiven. |
 
+The unit-file write skip ignores only Woods' top-level `extracted_at` stamp.
+A nested metadata field with the same name is application data: changing it
+rewrites the unit in both compact and pretty JSON output.
+
 `graph_analysis.json` used to be a fourth row, tolerating list ordering. It no
 longer is: the analyzer is order-independent and the oracle compares the file
 exactly. Tolerating the ordering there meant the harness, the only test that
 compares a full run against an incremental one, could not see the very
 dependence the analyzer's determinism work existed to remove.
 
+Graph targets use string identifiers even when an extractor emits a symbolic
+external target such as `:http_api`. Full and incremental runs retain every
+reverse dependency across JSON restoration and re-registration, including
+contributions from units sharing an identifier under different types. Unit
+dependency metadata retains its extractor-provided values. If an older version
+already lost reverse dependencies on these targets, run a full extraction once
+to restore them; loading the damaged graph cannot recover discarded entries.
+
 This matters most for **incremental CI chains**: restore the previous graph,
 run `woods:incremental` per merge. There, a unit that goes missing propagates
 forward run over run instead of being erased by the next full rebuild.
@@ -52,6 +64,10 @@ failure a CI chain cannot afford:
 | The range fails otherwise | Actionable error naming the range, **exit 1**. |
 | There is no `git` binary at all | Same two rows as above: the failure reads `git unavailable: …` and takes the daemon-coverage decision, rather than dying with an `Errno::ENOENT` backtrace. |
 
+Changed paths are normalized lexically before dispatch: trailing root slashes,
+duplicate separators and `.`/`..` segments do not create separate changes or
+bypass matching. Missing files remain representable; symlinks are not resolved.
+
 The range comes from `CI_COMMIT_BEFORE_SHA..CI_COMMIT_SHA` (GitLab),
 `origin/$GITHUB_BASE_REF...HEAD` (GitHub Actions), or `HEAD~1` (default). An
 unresolvable range — a GitLab zero-SHA on a new branch, an unfetched base ref,
@@ -76,6 +92,12 @@ The diff itself is rooted at the extracted application (`git -C Rails.root`),
 so it cannot read whatever checkout the process happened to start in — the
 same rooting rule the manifest's git provenance follows.
 
+Named, source-defined app modules included by runtime models are tracked as concern units even
+outside `concerns/` directories. Changing their source refreshes their includers,
+including inlined code and callback analysis. Multiple runtime mixins sharing a source
+file retain separate identities and refresh all their includers. Run a full extraction after upgrading
+to populate these previously missing source mappings.
+
 ## What a run does, in order
 
 `Extractor#extract_changed` is order-sensitive; each step exists because of the
@@ -120,6 +142,11 @@ step before it.
    string literal, and an unrelated addition in the same batch do not.
    Idempotent when nothing was pruned.
 
+Git enrichment uses the same eligibility checks in full and incremental runs:
+existing app-owned files under `Rails.root`, excluding `vendor/`, `node_modules/`,
+and framework/gem source units. Each typed unit resolves its own file history,
+even when its identifier is shared by another type.
+
 Then the second pass: `dependents` and `metadata.git` are refreshed on every
 touched unit (the incremental equivalents of full extraction's phases 2 and 4),
 type indexes are regenerated, the graph, `graph_analysis.json` and the
@@ -174,7 +201,8 @@ automatically.
 | `app/decorators`, `app/presenters`, `app/form_objects` | decorators |
 | `app/managers` / `app/policies` / `app/validators` | managers / policies + pundit_policies / validators |
 | `app/**/concerns/**/*.rb` | concerns |
-| `app/models/**/*.rb` (outside `concerns/`) | poros, caching |
+| `app/models/**/*.rb` (outside `concerns/`) | poros, caching; concerns when runtime model inclusion confirms a mixin |
+| `app/**/*.rb`, `lib/**/*.rb` (outside `concerns/`) | runtime model mixins also dispatch to concerns |
 | `app/controllers/**/*.rb` | caching |
 | `app/views/**/*.erb` | view_templates, caching |
 | `config/locales/**/*.yml` | i18n |
@@ -258,6 +286,34 @@ oracle compares against emits it too, both sides agree, wrongly. The coverage
 is in `spec/extractor_spec.rb`, driving the reconciler with a shrinking
 discovery set.
 
+### Runtime removals and bundle updates
+
+Jobs discovered through `ApplicationJob.descendants` supplement the job-file
+scan, but jobs are not part of `CLASS_BASED_DISCOVERY` removal reconciliation.
+If a dynamically defined or gem-owned job disappears without a tracked source
+path changing, its unit can survive subsequent incremental runs. A full
+extraction in a fresh Rails process removes it; an in-process full extraction
+can still see an old constant retained by that process (B-165).
+
+After adding, removing, or updating bundled gems, boot the updated bundle in a
+fresh process and run:
+
+```bash
+bundle exec rake woods:extract woods:validate
+```
+
+A `Gemfile.lock` change refreshes engines, middleware, and optional framework
+sources. It does not refresh every gem-owned model, job, or other runtime unit.
+Their recorded paths or metadata can remain stale, including absolute paths to
+a removed gem version and paths under `vendor/`. A full extraction rebuilds
+those units against the installed bundle (B-166).
+
+An absent external source path can also mean the validator runs on a different
+host or mount from extraction. Confirm the bundle and filesystem context before
+rebuilding; a full run in one container does not make its gem paths visible on
+another host. Validation warnings identify missing paths, but do not prove a
+retained unit matches the currently installed gem when its path still exists.
+
 ### Deletion
 
 - Paths named in the change set that no longer exist are **authoritative** for
@@ -423,6 +479,66 @@ owns the definition of "the two indexes agree" and documents every exclusion.
 
 **Run it before and after any change to the incremental path.**
 
+## Profiling fixed costs
+
+Set `WOODS_PROFILE=1` to time extraction phases. Git enrichment and unit JSON
+finalization are separate from incremental re-extraction; runtime discovery,
+whole-app reruns and pruning appear under `reconciliation`. `payload sync`,
+`publish` (the generation pointer write), and `payload prune` (retention) are
+separate, additive phases. Older versions included sync and retention inside
+`publish`, so do not sum those older lines without subtracting nested sync.
+
+`[profile total]` reports time inside the extraction call, including setup and
+failed runs. It is not another phase to sum. Compare it with phase durations
+to find unaccounted work; each line rounds to hundredths of a second. Rails
+boot, Bundler and watch reload work outside the call require separate wall
+measurements. Do not infer that all unaccounted time is boot.
+
+Both full and incremental runs currently seed the prior payload. Cloning uses
+per-file hardlinks (or copies where links are unsupported); updated files are
+replaced atomically, preserving previous generations. Full-run carry-forward
+and generation retention remain unchanged. The seed walker classifies each entry
+once, avoiding duplicate file metadata lookups; it still links each file
+separately. Generation directories remain independent, so pruning an old generation cannot remove a newer one's files.
+Compare repeated runs on the actual index filesystem before attributing latency
+to extraction or changing the index layout.
+
+For a repeatable component comparison from a source checkout:
+
+```bash
+# BENCH_ROOT is an existing scratch parent on the index filesystem.
+# The script creates and removes only its own temporary directory there.
+BENCH_ROOT=/path/on/index/filesystem WOODS_SOURCE=/path/to/baseline \
+  ruby bench/payload_seed.rb
+BENCH_ROOT=/path/on/index/filesystem WOODS_SOURCE=/path/to/candidate \
+  ruby bench/payload_seed.rb
+```
+
+The benchmark clones 8,335 synthetic 2 KiB files across 35 directories, checks
+file counts and bytes outside timing, and reports seven samples plus medians.
+This measures the seed component only; it does not boot Rails or establish an
+end-to-end host improvement. Filesystem metadata latency and the copy fallback
+can dominate differently from local hardlink results. A first full extraction
+has no previous payload, so include a repeated full run when measuring seed cost.
+Full seeding also preserves on-demand framework units when framework extraction
+is disabled, and non-JSON files in existing type directories; dropping the seed
+wholesale would change that behavior.
+
+### Choosing full versus incremental for CI
+
+Measure repeated full and representative-day incremental runs with
+`WOODS_PROFILE=1` on the same resulting application tree, configuration and index
+filesystem. Restore the same baseline index before each incremental trial;
+otherwise a second trial may be a no-op. Include Rails boot in both wall times
+when comparing separate task invocations, and compare resident watcher cycles
+separately. Validate each resulting index with `woods:validate`.
+
+Use full extraction for that workload when its median wall time is no greater
+than the representative incremental run. There is no universal changed-file
+threshold: shared dependencies and whole-app extractor triggers change the work
+per file. Re-measure after substantial application or Woods changes. A fast leaf
+edit does not establish that a day of commits is below the crossover.
+
 ## Boundaries and open work
 
 - **Reloaded deletion is supported.** The resident watcher reloads changed

data/docs/INDEX_LAYOUT.md

@@ -0,0 +1,382 @@
+# Published index layout for shell and Python readers
+
+Read `generation.json` at the configured index root, then read every structural
+artifact from the payload it names. Do not require `dependency_graph.json` or
+`manifest.json` at the root, and do not choose the highest directory in `payloads/`.
+A directory can be present before its generation is published.
+
+This is the filesystem contract for consumers without the Woods gem. Ruby callers
+can use [Woods::PublishedIndex](PUBLISHED_INDEX.md). These examples require a
+Woods 2.x payload index and a filesystem that supports the shared `flock` protocol
+below. They deliberately reject legacy flat indexes instead of claiming an atomic
+snapshot from them.
+
+## Entry point and files
+
+A typical structural index looks like this; optional entries need not exist:
+
+```text
+<index root>/
+├── generation.json
+├── payloads/
+│   └── gen-42/
+│       ├── manifest.json
+│       ├── source_inputs.json          # optional on older indexes
+│       ├── dependency_graph.json
+│       ├── graph_analysis.json
+│       ├── SUMMARY.md
+│       ├── models/
+│       │   ├── _index.json
+│       │   └── <unit filename>.json
+│       └── flows/
+│           ├── flow_index.json
+│           └── <flow filename>.json
+├── woods.json                 # optional configuration artifact
+├── dumps/                     # optional semantic-store snapshots
+└── ...                        # operational state and locks
+```
+
+The configured root can differ between a container and the host. Resolve the
+relative payload against the root visible to the reader, not the writer's path.
+
+`generation.json` is one JSON object, for example:
+
+```json
+{"number":42,"token":"978e69b524ac74fa","updated_at":"2026-09-15T12:00:00Z","reason":"incremental","payload":"payloads/gen-42"}
+```
+
+| Field | Contract |
+|---|---|
+| `number` | Positive integer, incremented on publication. It is local to this index; deleting/recreating the index can restart it. |
+| `token` | Opaque string, changed on each publication. Compare it as well as `number`; do not depend on its length or encoding. |
+| `updated_at` | ISO8601 publication timestamp, distinct from source modification times. |
+| `reason` | String or null describing the publication, such as `full`, `incremental`, or `refresh`. Treat values as extensible. |
+| `payload` | Relative directory name for this generation. Current writers use `payloads/gen-N`; follow the field instead of constructing it. Omitted/null means a flat layout. |
+
+Reject an absolute payload path or one whose resolved real path escapes the index
+root, including through a symlink. Unknown fields can be ignored. A missing
+pointer can mean an older flat index or no index at all; it is not evidence that a
+payload directory is published. An existing malformed pointer or a missing named
+payload is an error to investigate, not permission to silently serve root files.
+Some gem readers have permissive flat fallback for these failures; the publication
+gates below deliberately fail more strictly.
+
+### Payload artifacts
+
+| Artifact | Presence and meaning |
+|---|---|
+| `manifest.json` | Required for a complete structural publication. Counts by extractor directory, totals, extraction timestamp and provenance. Optional fields vary by writer/version; see [writer provenance](PUBLISHED_INDEX.md#manifest-writer-provenance). |
+| `source_inputs.json` | Versioned source-input identities and per-consumer provenance for this exact generation; old indexes may omit it. See [source freshness](SOURCE_FRESHNESS.md). |
+| `dependency_graph.json` | Required for a complete structural publication. Typed graph data; an empty graph is valid. |
+| `<type>/_index.json` and unit JSON | Present for extracted families. `_index.json` is an array of unit summaries; an empty array is valid. Disabled/unavailable families may be absent. Do not infer completeness from a fixed count of directories. |
+| `graph_analysis.json` | Derived graph analysis when produced. Treat absence as unavailable analysis, not an empty or corrupt unit index. |
+| `flows/flow_index.json` and flow documents | Optional precomputed flows. Use the index's relative paths; do not invent flow filenames. |
+| `SUMMARY.md` | Generated human-readable summary, not a machine schema. |
+
+Unit summaries identify units, but do not carry an artifact filename. Their
+`file_path` names the application's source file, not the unit JSON file. To avoid
+reimplementing filename normalization, scan JSON files in a needed type directory
+(excluding `_index.json`) and match the JSON `identifier` **and** `type`. The same
+identifier can exist in multiple types. The full unit fields are documented in
+[Extractor reference](EXTRACTOR_REFERENCE.md#extractedunit-field-reference).
+
+The graph's `nodes`, typed variants, forward/reverse relationships and relationship
+metadata belong to the graph format; preserve them when transporting the index.
+Extractor directories can contain multiple unit types: `graphql/` contains
+`graphql_type`, `graphql_mutation`, `graphql_resolver`, and `graphql_query`, while
+`rails_source/` contains `rails_source` and `gem_source`. Validate and retain the
+artifact's actual type instead of deriving it by singularizing the directory.
+
+Do not flatten typed variants into a single node per textual identifier. The
+static Woods self-map has the same publication envelope but different type
+families and `manifest.provenance.mode`; it is not Rails runtime evidence.
+
+### Semantic graph validation
+
+`woods:validate` checks raw graph data against the unit indexes and artifacts in
+one pinned published generation. This validation is unreleased after
+`2.0.0.beta2`. It checks the shapes of `nodes`, `edges`, `reverse`, `file_map`,
+`type_index`, and optional `variants`; typed identities must be unique and agree
+with the actual indexed units. Forward sources must exist. Reverse, file, and
+type memberships must match the union of primary and variant contributions.
+When present, `reverse_via` must preserve the typed forward records and their
+relationship attributes, including duplicate-record multiplicity.
+
+Cycles, recursion, shared paths, cross-type identifier collisions, nil file paths,
+and legacy bare-string edges are valid. Absent optional legacy fields remain
+legal. A target absent from the nodes is an unresolved reference, not automatically
+corruption: current metadata cannot distinguish an intentional external target
+from an internal node and unit that are both missing. Validation cannot certify a
+unique target type when several types share its name. A missing node whose typed
+unit remains indexed is detectable and is an error.
+
+The checker does not repair data, rerun extraction, or become a publication gate.
+It reports identity/path diagnostics through the existing validation report and
+nonzero task exit. It also supports the Woods static source map's explicitly
+recorded type families; that does not give the map Rails runtime fidelity.
+See [running validation from Ruby](PUBLISHED_INDEX.md#validate-a-published-generation)
+and [semantic error recovery](TROUBLESHOOTING.md#semantic-graph-validation-errors).
+
+### File profiles and file membership
+
+`file_map[path]` lists units associated with a source file, including whole-file
+profiles. It does not promise that every identifier names a Ruby constant. New
+writers mark graph nodes of types `caching`, `configuration`, `test_mapping`,
+`rails_source`, and `gem_source` with `"kind": "file_profile"`. The same field
+appears on non-primary typed variants.
+
+For example, `app/controllers/things_controller.rb` can map to both
+`ThingsController` and a caching unit named `app/controllers/things_controller.rb`.
+Inspect each typed node's `kind` to distinguish the profile; both retain their
+file membership so a source edit refreshes both units. Do not classify units by
+comparing the identifier with the path, and do not interpret a missing `kind` as
+proof that the unit names a constant.
+
+Older indexes omit the marker. Woods derives it from these known extractor types
+when loading and republishing a graph, including unchanged incremental nodes.
+Raw consumers of older indexes must treat absent markers as unclassified or use
+the documented type list. Identifiers, `file_map`, and type membership retain
+their existing shapes; older readers can ignore `kind`.
+
+### Reverse relationship records
+
+New writers add `reverse_via` to `dependency_graph.json`. Each target identifier
+maps to its incoming relationship records, including the owning source type:
+
+```json
+{
+  "reverse": { "Gadget": ["Widget"] },
+  "reverse_via": {
+    "Gadget": [{ "source": "Widget", "source_type": "service", "via": "render" }]
+  }
+}
+```
+
+The existing `reverse` arrays retain their bare identifiers. `reverse_via` includes
+edges from primary nodes and typed variants; several records can share a source
+and target while differing in type, relationship, or association attributes.
+Optional `through`, `through_db`, and `disable_joins` values match the forward
+edge. A `null` relationship means unknown legacy evidence. Target type remains
+unresolved when the target identifier belongs to multiple types; the source type
+does not resolve that ambiguity.
+
+These are recorded dependencies, not proof that changing a target breaks every
+source. For example, `factory_for` and migration `reference` edges describe
+different relationships from a runtime `render` edge. Consumers can inspect one
+target bucket without scanning the whole forward graph. Buckets and records are
+deterministically ordered, but consumers should treat the ordering as incidental.
+
+Older graphs omit `reverse_via`; absence means relationship detail must be derived
+from forward edges and variants, not that there are no dependents. Woods rebuilds
+this derived index from forward evidence when loading and republishing a graph.
+Existing readers can ignore the additive field; a subsequent changed extraction
+or full run publishes it. A no-op leaves the previous generation unchanged.
+
+### What is outside this structural snapshot
+
+`woods.json`, `dumps/`, embedding checkpoints, temporal snapshots, watch status,
+pending paths, MCP task records, exporter state and extraction/startup lock files
+have separate lifecycles. Their presence is configuration-dependent. Do not glob
+them into a structural payload or assume `generation.json` commits them together.
+A semantic/MCP deployment also needs its configured stores and artifacts; copying
+a structural payload alone does not clone that deployment.
+
+`<output>/.source-inputs.key` is private operational state used to verify source
+identities. Never include it in a published/exported structural snapshot; without
+it, a recipient can still read the index but source freshness is unknown.
+
+Temporary filenames, abandoned payloads, lock sidecars, retained-directory counts,
+and summary formatting are implementation details. Never remove or modify locks
+to make a reader proceed.
+
+## Atomic publication is not indefinite retention
+
+For a payload publication, Woods writes the payload, flushes it, and atomically
+replaces `generation.json` last. Capturing that pointer once selects a complete,
+immutable structural generation. Opening each file through a freshly reread pointer
+can mix generations and defeats that guarantee. A failed/no-op run does not advance
+the pointer. [Durability details](PUBLISHED_INDEX.md#durability-the-pointer-is-the-commit-point)
+explain the flush boundary.
+
+Retention can delete an older payload after the reader selects it. The default
+retains three generations, not three minutes. To keep a multi-file read or copy
+safe from Woods retention:
+
+1. Read and validate the pointer; resolve its payload inside the root.
+2. Open that payload's existing `manifest.json` **read-only** and take a shared
+   advisory `flock` on that open file. Do not create a new lock file.
+3. Re-read the pointer and confirm it is unchanged; verify the pathname still
+   names the open manifest inode. A pruner may have won between steps 1 and 2.
+4. Keep the handle/lock open for **all** reads or the complete copy. Use the single
+   captured payload path throughout. Once pinned, a later pointer advance is fine.
+5. Close the handle when done. On a race, discard partial results and retry the
+   entire operation from step 1, with a bounded retry count.
+
+Woods retention attempts a nonblocking **exclusive** `flock` on that same manifest
+before deletion and skips a payload held by readers. These are advisory filesystem
+locks, not Woods' writer-coordination locks. Ordinary readers need no exclusive
+lock and need not take `extraction.lock` or its guard. A lock-free reader must
+accept disappearance and restart the whole read; it must never silently replace
+missing files with files from another generation.
+
+This protocol protects against cooperating retention only. It does not protect
+against `woods:clean`, manual deletion, index replacement, or a filesystem that
+does not coordinate `flock` across its clients. Stop writers/cleanup and read an
+immutable snapshot if that protection is unavailable. Flat indexes (including
+full-extraction fallback when a payload cannot be created) have individually
+replaced files, not multi-file atomicity; read/copy them only with writers stopped.
+
+## Bash and jq: read one pinned generation
+
+Requires Bash, jq, GNU `realpath`, util-linux `flock`, and Linux `/proc`. Save as
+`read-woods.sh`, then run `bash read-woods.sh '/path with spaces/tmp/woods'`.
+The final command reads manifest and graph under the same lock. Substitute other
+reads or a complete copy **inside** the script before it exits. Printing a path
+and consuming it after the script exits does not keep it pinned.
+
+Exit 75 means a possible publication/retention race: retry the whole script a
+bounded number of times (for example three), discarding any previous output.
+Other nonzero exits need investigation. A persistent 75 can mean a broken index.
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+fail() { echo "$*" >&2; exit 1; }
+retry() { echo "$*; retry the whole read" >&2; exit 75; }
+root=$(realpath -e -- "${1:?provide the index root}")
+[[ -d "$root" ]] || fail 'Index root is not a directory'
+[[ -f "$root/generation.json" ]] || fail 'Missing pointer: legacy or unpublished index'
+marker=$(cat -- "$root/generation.json") || retry 'Cannot read pointer'
+jq -e '
+  if type != "object" then false
+  elif (.number | type) != "number" then false
+  else (.number >= 1 and (.number | floor) == .number)
+    and (.token | type == "string" and length > 0)
+    and (.payload | type == "string" and length > 0)
+    and (.payload | explode | all(. >= 32 and . != 127))
+  end
+' <<<"$marker" >/dev/null || fail 'Invalid pointer or flat layout'
+relative=$(jq -r '.payload' <<<"$marker")
+[[ "$relative" != /* ]] || fail 'Absolute payload path'
+payload=$(realpath -e -- "$root/$relative") || retry 'Missing payload'
+[[ -d "$payload" && "$payload" == "${root%/}/"* && "$payload" != "$root" ]] \
+  || fail 'Payload must resolve inside the index root'
+exec 9< "$payload/manifest.json" || retry 'Missing manifest'
+flock -sn 9 || retry 'Cannot pin manifest'
+current=$(cat -- "$root/generation.json") || retry 'Pointer disappeared'
+[[ "$current" == "$marker" && "$payload/manifest.json" -ef /proc/self/fd/9 ]] \
+  || retry 'Publication changed or retention removed the payload'
+jq -n --argjson generation "$marker" \
+  --slurpfile manifest "$payload/manifest.json" \
+  --slurpfile graph "$payload/dependency_graph.json" \
+  'if ($manifest | length) == 1 and ($manifest[0] | type) == "object"
+      and ($graph | length) == 1 and ($graph[0] | type) == "object"
+   then {generation: $generation, manifest: $manifest[0], dependency_graph: $graph[0]}
+   else error("Manifest and graph must each contain exactly one JSON object") end'
+# Descriptor 9 closes on exit, releasing the retention pin.
+```
+
+## Python: keep the pin while using the payload
+
+Requires Python 3.9+ on Unix with working `fcntl.flock`. Save as `read_woods.py`
+and run `python3 read_woods.py '/path with spaces/tmp/woods'`. The context manager
+retries acquisition three times. Errors during the read/copy propagate: discard
+partial output before retrying the entire operation. The scripts assume a trusted
+Woods-owned index; path containment is not a sandbox for hostile filesystem changes.
+
+```python
+import contextlib
+import fcntl
+import json
+import os
+from pathlib import Path
+import sys
+import time
+
+
+@contextlib.contextmanager
+def pinned_payload(index_root):
+    root = Path(index_root).resolve(strict=True)
+    pointer = root / "generation.json"
+    if not pointer.is_file():
+        raise ValueError("Missing pointer: legacy or unpublished index")
+    for attempt in range(3):
+        handle = None
+        try:
+            raw = pointer.read_bytes()
+            marker = json.loads(raw)
+            if not isinstance(marker, dict):
+                raise ValueError("Pointer must be an object")
+            number, token, name = (marker.get(k) for k in ("number", "token", "payload"))
+            if type(number) is not int or number < 1 or not isinstance(token, str) or not token:
+                raise ValueError("Invalid generation identity")
+            if not isinstance(name, str) or not name or Path(name).is_absolute():
+                raise ValueError("Invalid payload path or flat layout")
+            if any(ord(char) < 32 or ord(char) == 127 for char in name):
+                raise ValueError("Control character in payload path")
+            payload = (root / name).resolve(strict=True)
+            if not payload.is_dir() or root not in payload.parents:
+                raise ValueError("Payload must resolve inside the index root")
+            manifest = payload / "manifest.json"
+            handle = manifest.open("rb")
+            fcntl.flock(handle, fcntl.LOCK_SH | fcntl.LOCK_NB)
+            if pointer.read_bytes() != raw or not os.path.samestat(os.fstat(handle.fileno()), manifest.stat()):
+                raise BlockingIOError("Publication changed or payload was removed")
+        except (FileNotFoundError, BlockingIOError):
+            if handle is not None:
+                handle.close()
+            if attempt == 2:
+                raise
+            time.sleep(0.05)
+        except BaseException:
+            if handle is not None:
+                handle.close()
+            raise
+        else:
+            break
+    try:
+        yield marker, payload
+    finally:
+        handle.close()
+
+
+if __name__ == "__main__":
+    with pinned_payload(sys.argv[1]) as (generation, payload):
+        manifest = json.loads((payload / "manifest.json").read_text(encoding="utf-8"))
+        graph = json.loads((payload / "dependency_graph.json").read_text(encoding="utf-8"))
+        if not isinstance(manifest, dict) or not isinstance(graph, dict):
+            raise ValueError("Manifest and graph must each contain exactly one JSON object")
+        # Read more artifacts, or copy the whole payload, before leaving this block.
+        print(json.dumps({"generation": generation, "manifest": manifest, "dependency_graph": graph}))
+```
+
+## Shipping a structural snapshot
+
+While the pin is held, copy the selected payload into an unpublished staging
+location. Preserve the payload's relative path and pair it with the **captured**
+`generation.json`, not a later pointer reread from the live index. For example,
+a captured `payloads/gen-42` must still resolve to that directory in the exported
+root. Validate the copied manifest and graph, then publish the staged copy as a
+whole, or upload its payload first and switch the destination pointer last.
+Do not upload `generation.json` first or advertise a failed/partial copy.
+
+The retention lock is local coordination; do not copy its file descriptor or
+writer lock files. Copying `manifest.json` normally copies content, which is correct.
+Objects in a destination store do not inherit the source's locking guarantees;
+protect any later destination pruning with that destination's own reader protocol.
+
+## Compatibility within Woods 2.x
+
+Consumers may rely on the pointer field meanings, relative payload resolution,
+the complete-payload publication boundary, manifest/graph locations, and JSON unit
+identity described here. Additive JSON fields, new extractor families, new reason
+values, optional artifacts and additional root-level state may appear in 2.x.
+Ignore unknown fields and inspect available types instead of hardcoding a directory
+count. Existing field meanings and required-file locations are compatibility
+surfaces; incompatible changes require an explicit migration contract.
+
+Do not bind to temporary names, a fixed token length, directory listing order,
+retention count, internal lock sidecars, or Markdown summary formatting. Check the
+installed Woods version and its release's documentation when consuming older
+prereleases; this guide describes the current source contract, not a promise that
+every prerelease contains every optional field or retention improvement.

data/docs/INTERNALS.md

@@ -145,6 +145,11 @@ The `DependencyGraph` is a directed graph where nodes are `ExtractedUnit` identi
 - **Forward edges** (`@edges`): what each unit depends on, populated when units are registered
 - **Reverse edges** (`@reverse`): what depends on each unit, built during registration and in the resolve phase
 
+The published graph also carries additive `reverse_via` target buckets with
+typed source identities, relationship labels and association attributes. The
+bare-name `reverse` map remains compatible. See the
+[reverse relationship format](INDEX_LAYOUT.md#reverse-relationship-records).
+
 ```ruby
 graph = DependencyGraph.new
 graph.register(user_unit)     # adds User to nodes, adds User→Order edge (from belongs_to)
@@ -175,7 +180,7 @@ Scores feed into the retrieval ranker as one signal in the final ranking formula
 | **Cycles** | Circular dependencies, A→B→C→A. Detected via DFS, and capped: `graph_cycle_limit` (default 500) bounds how many are enumerated and `graph_cycle_max_length` (default 50) skips one longer than that. Either cap firing sets `stats.cycle_limit_reached`. Set both to `nil` for exhaustive enumeration. |
 | **Bridges** | Edges whose removal would disconnect the graph, high-risk structural connections |
 | **Cross-database edges** | Association or foreign-key edges whose two ends resolve to different databases. A `has_many :through` is reported as `join_through_across_databases` when `disable_joins` is false and `from_db`, `through_db` (the join model's database), or `to_db` disagree. A foreign key never resolves to an owner in the source database, even when another database also claims the table; when every owner sits elsewhere and they span more than one database, the entry comes back with `to: nil` and an `ambiguous_owners` list instead of guessing. Read from graph node and edge attributes, so full and incremental runs agree. Scoped to primary nodes (units registered in the graph), not variants. |
-| **Volatile dependencies** | Edges that point at a unit changing at least `volatile_dependency_ratio` times more often than the dependent (POODR: depend on things that change less often than you do). Dependencies with fewer than 5 commits or a `new` change frequency are skipped. Ranked by the dependency's PageRank; the persisted list keeps the top 20, while `stats.volatile_dependency_count` reports the full qualifying count and `stats.volatile_dependencies_limit` reports the cap. |
+| **Volatile dependencies** | Edges that point at a unit changing at least `volatile_dependency_ratio` times more often than the dependent (POODR: depend on things that change less often than you do). Dependencies with fewer than 5 commits or a `new` change frequency are skipped. Ranked by the dependency's PageRank; an optional `volatile_dependency_limit_per_target` selects edges per typed dependency before the persisted top 20, while `stats.volatile_dependency_count` reports the full qualifying count and `stats.volatile_dependencies_limit` reports the cap. |
 | **Undeclared package edges** | Edges that cross a Packwerk package boundary the source package does not list in `dependencies`. Membership comes from each unit's `package` node attribute, declarations from the package unit's own `package_dependency` edges. Woods reports the boundary; enforcement stays with `packwerk check` / `pks check`. |
 
 Analysis results are written to `graph_analysis.json` and surfaced in `SUMMARY.md`.
@@ -194,7 +199,7 @@ persisted rather than every hub in the graph.
 | `cycles` | `stats.cycle_count`, `stats.cycle_limit_reached` | yes, `graph_cycle_limit` cycles of at most `graph_cycle_max_length` nodes | the count is the persisted array; the flag says whether either cap fired |
 | `bridges` | none | yes, top 10 by score | not counted |
 | `cross_database_edges` | `stats.cross_database_edge_count` | no | every crossing edge |
-| `volatile_dependencies` | `stats.volatile_dependency_count`, `stats.volatile_dependencies_limit` | yes, top 20 by the dependency's PageRank | the count is every qualifying edge; the limit is the cap |
+| `volatile_dependencies` | `stats.volatile_dependency_count`, `stats.volatile_dependencies_limit`; optional `stats.volatile_dependencies_limit_per_target`, `stats.volatile_dependency_reported_count` | yes, optional per-target cap followed by top 20 by the dependency's PageRank | count is every qualifying edge before caps; reported count is the final array length and appears only when the per-target cap is enabled |
 | `undeclared_package_edges` | `stats.undeclared_package_edge_count` | no | every undeclared crossing |
 
 A capped section means the array on disk is a page, not the population.