woods 2.0.0.beta2 → 2.0.0.beta3

data/docs/EVALUATION.md

@@ -14,6 +14,177 @@ Three harnesses, three questions.
 
 `woods:evaluate:baseline[grep|random|file_level]` scores a naive strategy on the same query set for comparison.
 
+## Paired lexical comparison (#404 / #227)
+
+`bench/evaluation/lexical_comparison.rb` runs **all 28 existing questions at their
+existing budgets** through semantic retrieval, lexical-only retrieval, a fixed
+query-seeded graph experiment, and the existing identifier-substring grep baseline.
+Unlike the strategy groups below, these conditions answer the same questions.
+The semantic condition replays the same captured real MiniLM vectors; labels,
+source snapshot and the original quality floors are unchanged.
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/lexical_comparison.rb /tmp/woods-paired.json
+# Use the pinned tokenizer environment described below:
+python bench/evaluation/capture_tokens.py /tmp/woods-paired.json lexical_comparison_capture.json
+```
+
+The second command writes `bench/evaluation/lexical_comparison_capture.json`.
+It retains every query's outcomes, context hash and exact token count, including
+misses. Raw returned contexts remain in `/tmp/woods-paired.json`. The capture binds
+its corpus/vector hashes and tokenizer provenance. It is reviewed experimental
+evidence, not a replacement for the required semantic gate.
+
+Initial Ruby 4.0.6 comparison (five warm pipeline repetitions per question):
+
+| Condition | Precision@5 | Recall | MRR | Mean actual context tokens | Median query latency (ms) |
+|---|---:|---:|---:|---:|---:|
+| Existing semantic pipeline | 0.536 | 0.580 | 0.857 | 1,000.7 | 3.019 |
+| Explicit lexical | 0.492 | 0.666 | 0.780 | 1,152.6 | 0.407 |
+| Lexical + query-seeded graph experiment | 0.396 | 0.476 | 0.750 | 1,159.8 | 3.292 |
+| Existing identifier grep baseline | 0.342 | 0.288 | 0.336 | 592.9 | 0.628 |
+
+Lexical retrieval improves recall on this small set while losing precision and
+first-hit rank against the semantic pipeline. This supports an explicit offline
+option, not semantic equivalence or a new default. The seeded graph condition
+loses on these measures and stays **evaluation-only**. It uses 20 iterations,
+restart probability 0.15, normalized positive lexical seeds, bidirectional
+recorded relationships within type eligibility, and no seeds for a no-match
+query. These choices were fixed before scoring; no weights were tuned to labels.
+The grep baseline matches identifiers, not source text; it retains its existing
+ordering, with eligibility applied before its limit and the same budgeted renderer.
+
+Latencies are warm in-process measurements over 62 units. They exclude lexical
+snapshot construction, live embedding/network latency and Rails extraction; they
+are not host-scale performance claims. Exact tokens count returned context only,
+using the existing `cl100k_base` capture machinery. Agent task outcomes were not
+measured. Ambiguous typed identities, concern/callback metadata and unrelated-hub
+controls also have targeted regressions; those synthetic controls do not replace
+representative host evaluation. Broader production/provider/agent evidence remains
+open under #227.
+
+## Checked-in Canopy retrieval gate
+
+The required CI `coverage` job runs this offline command before its test suite:
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/runner.rb
+# Report: tmp/retrieval-evaluation.json (uploaded by CI)
+```
+
+`bench/evaluation/` contains a versioned, deliberately small baseline:
+
+- **Corpus:** 62 runtime-extracted Canopy units and 28 annotated queries covering
+  publishing, billing, newsletters, and support. Source and relationships come
+  from the committed fictional `woods-testbed` Rails application; no application
+  records are included. Exact application/Woods revisions are in `corpus.json`.
+  This snapshot selects models, services, jobs, controllers, Pundit policies, and
+  mailers; it does not represent every extractor or a large production index.
+- **Embeddings:** real `all-MiniLM-L6-v2` ONNX inference, captured with a pinned
+  model revision and file hashes. CI replays those vectors through the current
+  production classifier, stores, graph, ranker, type fallback, and assembler.
+  No hash-based fake embeddings or expected-answer vectors are used.
+- **Gate:** per-strategy precision at 5, recall, and MRR must remain at least 95%
+  of the first observed score (floors rounded down to six decimal places).
+  Corpus/vector/token-provenance mismatches, missing strategies, and unexpected
+  strategy selection also fail. The baseline format is developer-only and is
+  **not** the `EVAL_BASELINE_FILE` aggregate-threshold format.
+
+B-190/B-191 recapture on Ruby 4.0.6 (five warmed pipeline repetitions per query; Ruby 3.3.1
+and 3.4.10 replay the same answers):
+
+| Strategy | Queries | Precision@5 | Recall | MRR | Mean actual context tokens |
+|---|---:|---:|---:|---:|---:|
+| Keyword | 4 | 0.292 | 0.500 | 0.750 | 1,020.8 |
+| Vector | 4 | 0.313 | 0.375 | 0.625 | 1,041.2 |
+| Graph | 8 | 0.813 | 0.519 | 1.000 | 1,041.1 |
+| Hybrid | 4 | 0.750 | 0.396 | 1.000 | 1,058.8 |
+| Direct with type filtering | 4 | 0.375 | 0.750 | 0.625 | 551.0 |
+| Within-type vector fallback | 4 | 0.400 | 1.000 | 1.000 | 1,250.8 |
+
+Precision@5 divides relevant hits by the actual returned slice size (up to five),
+not always by five. Recall divides retrieved relevant units by all annotated
+relevant units. MRR is the mean reciprocal rank of the first relevant hit; a
+query with no relevant hit contributes zero.
+
+`profiles.json` binds six measured Ruby runtimes to a shared context capture:
+Ruby 3.0.7, 3.1.7, 3.2.11, 3.3.1, 3.4.10, and 4.0.6. Other Ruby engines or
+major/minor versions fail with a recapture instruction. All six produce identical
+retrieved identifiers, context bytes, token counts and quality metrics for every
+query. The original positive per-runtime quality floors remain unchanged in
+`baseline.json` and `baseline_legacy.json`.
+
+B-191 orders equal PageRank scores lexically by identifier. B-192 orders hybrid
+candidates by descending score, identifier, then source before truncation and
+RRF; the same ordering selects the three vector seeds for graph expansion.
+Only `hybrid-2` changes from the prior Ruby 3.3–4.0 capture: `Newsletter::DeliverJob`
+now precedes `Newsletter::Delivery`, matching the prior Ruby 3.0–3.2 answer.
+No quality metric or exact token count changes. This is production retrieval
+ordering, with no answer reordering in the evaluation harness.
+
+These are separate annotated query groups, **not a controlled comparison of six
+strategies on identical questions**. Relevance annotations were written from
+fixture source and its functional contract before scoring. Initial graph and
+direct probes remain in the corpus; additional probes exercise the supported
+snake-case graph roots and actual within-type fallback. The B-190 fix now
+resolves all four originally empty graph queries, including `trace Billing::Invoice`
+and `trace ReviewAssignment`. The B-190 change affected those four graph answers. B-191 additionally
+changes vector and within-type fallback ranks or membership; all original queries,
+relevance labels, vectors, and quality floors remain unchanged. Graph precision/recall/MRR increased from 0.417 / 0.238 / 0.500 to
+0.813 / 0.519 / 1.000. Low semantic recall and editorial misses are
+also retained. The floors prevent further regression; they are not release
+quality targets or evidence that retrieval is already good enough.
+
+`capture_report.json` records per-query ranks, quality, latency ranges, and exact
+`cl100k_base` counts of the returned context. Those counts are bound to context
+SHA-256 values; the tokenizer vocabulary asset hash, pattern, special-token map,
+and package version are recorded alongside them. Changed context requires explicit recapture and review. Woods'
+`tokens_used` remains a separate character-ratio estimate. Neither number counts
+agent prompts, generated answers, or total agent usage. Vector capture records
+actual model-input token counts (including special tokens after truncation),
+separately from output-context tokens. The model truncates inputs to 512 tokens,
+so long source units are only partially represented.
+
+Pipeline latency excludes model loading, downloads, live embedding requests, and
+network time. `vectors.json` separately records the warm embedding batch time,
+input counts, model/package provenance, and SHA-256 of each model asset. Timing
+is observational and has no shared-runner CI threshold. This baseline does not
+validate OpenAI/Ollama quality, their latency, production-scale storage, or agent
+outcomes. The baseline scope of #227 is complete; broader provider and
+production-scale evaluation remain separate follow-up work.
+
+### Reproduce or review a new capture
+
+Ordinary CI needs only the Ruby bundle and the checked-in files. Model refresh is
+an explicit maintenance operation with isolated Python dependencies:
+
+```bash
+uv venv /tmp/woods-eval-venv
+uv pip install --python /tmp/woods-eval-venv/bin/python -r bench/evaluation/requirements.txt
+/tmp/woods-eval-venv/bin/python bench/evaluation/capture.py /tmp/woods-eval-model
+```
+
+This overwrites `vectors.json` with real inference from the pinned Apache-2.0
+model. To capture changed context after an intentional pipeline change:
+
+```bash
+bundle exec ruby -Ilib -r./bench/evaluation/runner -e \
+  'r=RetrievalBaseline::Runner.new; File.write("/tmp/retrieval-capture.json", JSON.pretty_generate(r.run))'
+/tmp/woods-eval-venv/bin/python bench/evaluation/capture_tokens.py /tmp/retrieval-capture.json
+```
+
+Replay all six measured Ruby lines before replacing the shared capture. Retain
+the exact capture runtime in the report and preserve each runtime’s original
+quality floors. A newly observed runtime difference needs separate reviewed
+captures rather than being hidden by answer reordering.
+
+Review every changed answer and score before updating the matching baseline digests or
+floors. Never lower a floor merely to make CI pass. Keep labels independent of
+rankings and preserve hard or failed queries. When changing the application
+snapshot, use a fresh runtime `woods:extract` into a disposable output directory,
+retain only deliberate source/relationship fields, and record both repository
+SHAs. Do not copy a host's live index or application records into the repository.
+
 ## Agent-level ablation
 
 Retrieval scores say whether the right units come back. They do not say whether an agent finishes the job. The ablation runs the same task set twice per task, once with the Woods MCP server available and once without, and compares resolution rate, tokens, cost, and turns.
@@ -80,7 +251,7 @@ bin/rails "woods:evaluate:ablation[config/eval_ablation.json]"
 
 The report holds every attempt (`task_id`, `condition`, `resolved`, `total_tokens`, `cost_usd`, `turns`, `duration_ms`, `error`, `provenance`) and a summary per condition (`resolution_rate`, `mean_tokens`, `mean_cost_usd`, `mean_turns`, `tasks`, `errors`) plus a `delta` (on minus off, present only when both conditions ran). Tokens are the sum of input, output, cache-creation, and cache-read tokens. `mean_tokens` is `null` when every trial in that condition errored before the agent produced JSON.
 
-A timeout (`AblationRunner.new(..., timeout: seconds)`, default 600) bounds every command a trial runs, applied independently to each one rather than as a single budget shared across the trial: worktree add/remove, the optional `reset`, the agent invocation, and the check. A timed-out `reset` or `check` counts as an error the same way a timed-out agent invocation does. When the default subprocess executor is in use, a timed-out command is terminated (`TERM`, then `KILL` if still alive after a short grace period) so it never outlives the trial that started it.
+A timeout (`AblationRunner.new(..., timeout: seconds)`, default 600) bounds every command a trial runs, applied independently to each one rather than as a single budget shared across the trial: worktree add/remove, the optional `reset`, the agent invocation, and the check. A timed-out `reset` or `check` counts as an error the same way a timed-out agent invocation does. The default subprocess executor creates an owned process group. On timeout it sends `TERM`, then `KILL` after a short grace period if necessary, to that group, including ordinary descendants whose parent already exited. Commands must not deliberately detach into separate sessions/groups; those escape this cleanup. Custom executors exposing only a PID retain single-process cleanup; executors exposing no PID must handle their own subprocess lifecycle.
 
 ### Where to get tasks
 
@@ -89,3 +260,295 @@ The Rails Foundation's "Agents on Rails" benchmark (announced 2026-08-13, built
 ### Reading the numbers
 
 A negative `delta.mean_tokens` with an equal or higher `delta.resolution_rate` is the result that sells the index, on the tasks actually run. It does not, by itself, establish that the index causes the difference: run at least ten tasks, two conditions on five tasks is noise, and remember that agent runs are not perfectly reproducible even at a fixed baseline SHA. Keep the baseline SHA and Woods generation fixed across a comparison (both are recorded per result), and do not let the `on` agent run `woods:extract` mid-task.
+
+## Explicit scope comparison
+
+`bench/evaluation/scope_comparison.rb` reuses #227's 62-unit Canopy extraction,
+28 original questions, captured MiniLM vectors, unchanged gold labels, and the
+same 1200-token budgets. Fixed domain-word rules choose the existing billing,
+newsletter, or support directories across models, controllers, jobs, and use
+cases. Other questions are unscoped controls. This assigns **no package ownership**
+to Canopy. Expected units outside a requested directory remain in the gold labels,
+so restricting scope can reduce recall for questions about cross-boundary behavior.
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/scope_comparison.rb /tmp/woods-scope-raw.json
+python bench/evaluation/capture_tokens.py /tmp/woods-scope-raw.json scope_comparison_capture.json
+```
+
+Use the pinned Python requirements in this directory's existing evaluation runbook.
+The capture records per-query scope, outcomes, context hashes, exact cl100k counts,
+and the existing precision@5, recall, and MRR metrics. Warm replay latency includes
+scope preparation and excludes provider/network time. This is a filtering and
+ranking comparison, not a coding-task or live-host performance claim.
+
+`spec/retrieval/package_scope_spec.rb` separately builds a synthetic Packwerk
+filesystem fixture and obtains root, nested, and sibling ownership from the real
+`PackageExtractor`. It verifies package boundaries without inventing ownership in
+a captured host corpus. Live pgvector/Qdrant tests cover a small eligible package
+behind globally stronger matches, typed chunks, and old vectors without metadata
+payload fields. Broader agent task-quality evidence remains unmeasured by this
+scope comparison.
+
+The reviewed capture uses 11 scoped questions and 17 unchanged controls per mode.
+On the 11 questions with a requested directory scope:
+
+| Mode | Scope | Precision@5 | Recall | MRR | Mean exact context tokens |
+|---|---|---:|---:|---:|---:|
+| Semantic | none | 0.5758 | 0.6530 | 0.9545 | 907.55 |
+| Semantic | explicit paths | 0.6364 | 0.5621 | 0.9091 | 820.73 |
+| Lexical | none | 0.5394 | 0.7364 | 0.8636 | 1201.45 |
+| Lexical | explicit paths | 0.6970 | 0.6455 | 0.8030 | 936.73 |
+
+Scope improves precision here while losing cross-boundary expected units. It is
+an explicit intent filter, not a default ranking improvement. Scoped lexical
+median query time was 2.30 ms versus 0.41 ms unscoped on these 11 replay questions;
+preparing the full metadata snapshot is visible even on this small corpus.
+Exact token counts measure the returned context, not MCP metadata or agent prompts;
+Woods' 1200-token budget is an estimate and can differ from cl100k counts.
+
+## Compact evidence comparison (#403)
+
+`bench/evaluation/evidence_comparison.rb` replays the same #227 Canopy corpus,
+28 questions, expected unit labels, captured MiniLM vectors and per-question
+budgets with full, compact and outline evidence, separately for semantic and
+lexical retrieval. Only within-unit evidence formatting varies within each
+strategy. The checked capture binds all 168 returned contexts to exact
+`tiktoken 0.11.0` `cl100k_base` counts; full-mode hashes match the approved prior
+comparison. The original labels are unit-level labels, not method/span relevance
+or task-correctness labels.
+
+| Strategy / evidence | P@5 | Unit recall | MRR | Mean exact context tokens |
+|---|---:|---:|---:|---:|
+| Semantic / full | 0.5363 | 0.5798 | 0.8571 | 1000.68 |
+| Semantic / compact | 0.4786 | 0.6476 | 0.8571 | 1034.68 |
+| Semantic / outline | 0.4536 | 0.8304 | 0.8571 | 976.86 |
+| Lexical / full | 0.4917 | 0.6661 | 0.7798 | 1152.64 |
+| Lexical / compact | 0.4881 | 0.7077 | 0.7887 | 1063.79 |
+| Lexical / outline | 0.3857 | 0.7964 | 0.7887 | 1014.75 |
+
+These results do not justify a default change. Outlines can expose more unit names
+without their method bodies; 19 of 100 returned semantic compact units and 13 of
+103 lexical compact units have no selected source span (the response explicitly
+reports omissions and may contain runtime metadata). Per-unit selected/omitted
+span counts remain in the capture. Increased unit recall is not evidence that an
+agent saw the implementation it needed. Compact semantic context consumed more
+actual tokens on average. Structured provenance and MCP envelopes are additional
+output tokens, outside the context-only counts above.
+
+Median warm query time across questions was 3.24/11.80/11.44 ms for semantic
+full/compact/outline and 0.46/4.44/6.08 ms for lexical modes. This measures local
+replay and includes source parsing; semantic provider/network time is excluded.
+It does not establish live-host performance.
+
+A separate deterministic long-prefix regression fixes the target method before
+selection: 80 unrelated methods precede `refund(payment)`. Under the same small
+context allowance the compact selector includes its entire body and exact
+published byte/hash coordinates. This is a synthetic retention test, separate
+from the real-corpus unit metrics above. Unicode, shared-line declarations, nested
+blocks, commented inlined concern display, inherited metadata, unknown physical
+locations and unknown generation are covered by source-evidence contract tests.
+Installed stdio and HTTP checks exercise generation attribution and typed,
+SHA-guarded full lookup on a real Rails extraction.
+
+Reproduce the corpus comparison without changing labels:
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/evidence_comparison.rb /tmp/woods-evidence-raw.json
+python bench/evaluation/capture_tokens.py /tmp/woods-evidence-raw.json evidence_comparison_capture.json
+```
+
+Use the same tokenizer version/vocabulary recorded by the capture. Raw contexts
+are an external artifact; the checked file retains hashes, exact counts, unit
+outcomes and source-span counts. Candidate ranking and ownership diversity are
+outside this experiment.
+
+### Public Writebook task attempts
+
+[`public_task_evidence_capture.json`](../bench/evaluation/public_task_evidence_capture.json)
+records twelve actual agent attempts on the Rails Foundation's public
+[`ai-evals`](https://github.com/rails/ai-evals) Writebook tasks at revision
+`5327efe54dd767d662ab89332f6a4285fef07bd8` (Writebook 1.2.1):
+`ar-announce-once`, `aj-enqueue-after-commit`, and `ar-archive-book-access`.
+Each task ran twice per condition with the same task prompt, client 2.1.267,
+resolved model `claude-sonnet-5`, 40-turn cap and Woods source revision
+`33abd6da8e072c7f510d052292aaff4260c30c7f`. Hidden verifiers remained outside the
+agent workspace until the attempt finished. Baseline failure/reference success
+prechecks and prompt/artifact hashes are retained in the capture.
+
+The intervention changed the default evidence mode at the MCP boundary. Full
+lookup kept its existing complete output; compact lookup used a 1200-estimated-token
+budget. Explicit evidence choices, metadata-only reads and full-source follow-ups
+remained available. This measures a default-mode change, not a forced mode on
+every call. All actual evidence calls used `lookup` (26 full-condition calls and
+20 compact-condition calls); there were no `codebase_retrieve` calls, query hints,
+or explicit full follow-ups. Consequently this does not establish retrieval
+ranking or query-guided compact evidence benefits.
+
+| Condition | Hidden task suites passed | Median MCP text tokens (cl100k) | Median agent wall time |
+|---|---:|---:|---:|
+| Full | 5/6 | 8661 | 180.3 s |
+| Compact | 4/6 | 2316 | 132.8 s |
+
+Both compact announcement attempts failed the go-live-then-return-to-draft edge;
+the second full attempt passed it. Both conditions passed both repetitions of the
+other tasks. Two compact `Accessable` lookups returned not-found errors; these are
+retained as valid tool errors, not discarded trials. The earlier 20-turn calibration
+pair hit the turn cap and failed the same announcement edge in both modes; the
+fixed 40-turn primary protocol and all failed/setup pilots are recorded separately.
+
+Compact delivered less MCP text in this small sample but passed one fewer task.
+That is a reason to keep it explicit and retain full-source verification, not to
+change the default. Two trials could overlap, so wall times are descriptive rather
+than isolated performance estimates. `cl100k_base` measures representation size,
+not the Claude tokenizer or billing. The capture separately records serialized MCP
+response counts, client-reported usage and list-price estimates; these must not be
+substituted for model-visible context or billed charges. It contains metrics,
+arguments and hashes, without shipping application source or agent transcripts.
+
+
+## Source-owner overlap experiment (#412)
+
+`bench/evaluation/owner_overlap_comparison.rb` evaluates an offline selector after
+ranking, using the production context assembler's full-source appender, section
+allocations, truncation and token estimator. It does not add a runtime option.
+The strongest result stays first. A later candidate moves ahead only when the
+higher-scoring pending span is already completely covered by delivered source.
+Several disjoint methods from one file remain separate; typed units are never
+merged. Exact-target/pinpoint queries and sections with one verified owner bypass
+the policy. Unknown origins block promotion across their position. Truncated
+parents establish no complete span coverage. Candidate scores remain unchanged.
+
+Origins require a unique exact byte substring in an explicitly supplied immutable
+source snapshot. Display paths alone, ambiguous repeated text, and synthesized
+inlined concern display do not prove physical ownership. Original-file and
+published-source hashes bind the verified spans. There are no query-time host-file
+reads. These are offline fixture origins, not newly published runtime provenance.
+
+The fixed fixtures define necessary methods and budgets before selection. Seven
+synthetic cases and one real Woods source case use actual `RubyAnalyzer` output;
+the latter is static source evidence, not a booted Rails application. Synthetic
+executable oracles load only delivered snippets into an isolated namespace and
+assert their fixed multi-method behavior. The wrapper supplies the declared class
+name for a method; it supplies no absent implementation. These deterministic task
+checks are separate from an LLM answer/task benchmark.
+
+| Fixture | Necessary-method recall, baseline → overlap | Executable fixture, baseline → overlap | Exact context tokens, baseline → overlap |
+|---|---|---|---|
+| Crowded class/method overlap | 0.5 → 1.0 | fail → pass | 312 → 308 |
+| Single owner | 1.0 → 1.0 | pass → pass | 312 → 312 |
+| Exact target / pinpoint (each) | 0.5 → 0.5 | fail → fail | 312 → 312 |
+| Truncated parent | 0.0 → 0.0 | fail → fail | 165 → 165 |
+| Disjoint methods in one file | 1.0 → 1.0 | pass → pass | 97 → 97 |
+| Unknown inlined display origin | 0.5 → 0.5 | fail → fail | 325 → 325 |
+| Woods storage identity / filename source | 0.5 → 1.0 | not measured | 336 → 366 |
+
+Relevant-owner coverage equals necessary-method recall in these fixtures because
+each required owner has one method, except the disjoint-method case where both
+methods share one owner and coverage remains 1.0. The real-source case retains
+`StorageIdentity.key` and `FilenameUtils#collision_safe_filename`; that is source
+retention, not proof that every helper needed to execute a change is present.
+The budget is Woods' existing character estimate: for example, the 230-token
+crowded fixture delivers 312/308 cl100k tokens. Exact counts measure context text,
+not prompts, provenance envelopes or agent output. No new token calibration or
+section-budget borrowing is introduced.
+
+The same #227 28 Canopy questions, gold labels, vectors and budgets serve as a
+negative control. All 62 units have distinct display paths and lack verified
+original spans. Both conditions therefore return identical contexts, matching
+the approved semantic full capture byte for byte. This demonstrates safe bypass;
+it cannot demonstrate owner-selection benefit on a runtime Rails index.
+Equal scores, typed collisions, unknown-origin barriers and separate
+primary/supporting/framework allocations have additional regression coverage.
+
+**Decision: keep this evaluation-only.** Fixed overlap cases benefit, but a
+truncated strongest class still consumes the whole allowance before complementary
+evidence can be selected. Real runtime provenance and actual agent answer/task
+correctness under this selector remain unmeasured. The Writebook task attempts
+above tested evidence formatting, not this policy, and cannot fill that gap.
+A production proposal needs trustworthy published physical spans and a paired
+representative task evaluation; no default change, owner quota, or near-relevance
+promotion is justified by these fixtures.
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/owner_overlap_comparison.rb /tmp/woods-owner-overlap-raw.json
+python bench/evaluation/capture_tokens.py /tmp/woods-owner-overlap-raw.json owner_overlap_capture.json
+```
+
+The checked capture retains all 72 results, unchanged #227 gold labels, fixed-case
+requirements, source hashes, actual context counts/hashes, original scores and
+selection/deferral/budget-exhaustion traces. It retains failed cases. Use the
+recorded `tiktoken` version and vocabulary; keep raw contexts outside the checkout.
+
+
+## Matched optional context-hook tasks (#406)
+
+**Both conditions passed all four task attempts.** This small comparison does not
+establish a task-quality or efficiency improvement. Hints remained optional.
+The context condition returned less MCP text in aggregate, added its own context,
+and had a higher median agent wall time; individual repetitions varied.
+
+`bench/evaluation/hook_context_capture.json` records all eight preselected trials,
+returned-context hashes and actual token counts, official-client usage, emitted
+and delivered hints, callback latency and hidden-verifier outcomes. Two tasks
+from Rails Foundation's public `ai-evals` at
+`5327efe54dd767d662ab89332f6a4285fef07bd8` used disposable Writebook 1.2.1/Ruby 3.4.7
+images. Seeded defects failed and reference fixes passed before agent trials.
+Both arms used Claude Code 2.1.267, `claude-sonnet-5`, a 40-turn limit, the same
+prompt and source snapshot, full MCP evidence, and lexical retrieval. Refresh
+was disabled in both arms to isolate the independent context switch. Each
+condition ran twice per task; two trials ran concurrently. The hidden verifier
+was copied only after the agent exited.
+
+| Task attempt | Manual agent wall (s) | Context agent wall (s) | Hidden verifier, manual / context |
+|---|---:|---:|---|
+| `aj-enqueue-after-commit`, rep 1 | 111.3 | 113.6 | pass / pass |
+| `aj-enqueue-after-commit`, rep 2 | 173.4 | 154.9 | pass / pass |
+| `ar-atomic-import`, rep 1 | 105.3 | 210.4 | pass / pass |
+| `ar-atomic-import`, rep 2 | 126.5 | 119.9 | pass / pass |
+
+| Measure | Manual tools | Context enabled |
+|---|---:|---:|
+| Tasks passed | 4 / 4 | 4 / 4 |
+| Median agent wall (s) | 118.9 | 137.4 |
+| Total MCP text, actual cl100k tokens | 45,834 | 39,550 |
+| Additional delivered hint tokens, cl100k | 0 | 1,914 |
+| Emitted / delivered hints | 0 / 0 | 16 / 16 |
+| Client-reported input tokens | 160 | 158 |
+| Client-reported cache creation tokens | 217,920 | 206,056 |
+| Client-reported cache read tokens | 5,715,709 | 5,467,210 |
+| Client-reported output tokens | 40,884 | 42,000 |
+
+The 16 context callbacks took 0.590–0.818s (median
+0.613s); the largest complete JSON envelope was 1,067 bytes.
+Disabled callbacks took 1.75ms median and emitted nothing.
+Every emitted hint appeared in the official client's `hook_additional_context`
+attachments. A separate native-plugin probe also matched SessionStart and
+PostToolUse hint strings inside captured model requests. Attachment delivery does
+not establish that the model used a hint. Callback measurement wraps the actual
+entrypoint and excludes the instrumentation wrapper's own startup; agent wall
+includes the whole client run. OS scheduling and cold startup can still cause a
+silent timeout under the fixed deadline.
+
+An independent review checked all 44 rendered relationship claims against each
+trial's typed graph and found no incorrect claim. Useful owners were often
+already inspected; broad transitive candidates were sometimes redundant. One
+atomic-import run explicitly followed the DemoContent candidate and found no
+matching call site. New test files received honest unresolved-path notices.
+Source freshness was initially unknown because of `unverified_consumption_scopes`,
+then edits were marked drifted; truncated and pre-refresh qualifications remained
+visible. Correct candidates are not proof of task benefit or complete impact.
+
+Exact cl100k counts measure representation size, separately from the client's
+repeated input/cache/output accounting. The client's summed list-price estimates
+were $2.424 / $2.338; these are not billed charges or evidence of a
+repeatable saving. This two-task sample and overlapping execution cannot establish
+isolated performance or general quality. Raw traces remain local review artifacts;
+the checked capture binds their hashes. `diff.patch` covers tracked files only;
+new untracked tests are preserved through native Write/Edit records and separately
+labelled transcript reconstructions, not claimed as disk-captured final trees.
+
+The measured hook implementation is `21a578d6`; subsequent main integration left
+its helper and entrypoint bytes unchanged. The earlier smoke setup missing new
+bundle executable stubs is retained as an invalid setup attempt. The fixed
+primary trials all completed and none was discarded or replaced.

data/docs/EXTRACTOR_REFERENCE.md

@@ -60,9 +60,13 @@ Every extractor returns `Array<ExtractedUnit>`. An `ExtractedUnit` is a self-con
 
 **Key details:**
 - Uses `ActiveRecord::Base.descendants` for discovery (runtime introspection, not static parsing)
+- Named, source-defined app model mixins (for example `Card::Pinnable` in `app/models/card/pinnable.rb`) resolve through runtime source locations, with conventional concern paths as fallbacks. Included mixins also receive `:concern` units, so their actual files map to the includer through dependency edges. Gem-owned modules stay outside this discovery. Conventional concern files retain their existing identity even when nested helpers share the same file. Outside those directories, each included runtime mixin receives its own concern identity even when several share a source file; editing that file refreshes every includer.
 - Inlines concerns: all `include FooConcern` references are resolved and the concern source is appended to `source_code`. Inlined concern names are recorded in `metadata[:inlined_concerns]`
-- Extracts all 19 callback types: `before_validation`, `after_validation`, `before_save`, `after_save`, `around_save`, `before_create`, `after_create`, `around_create`, `before_update`, `after_update`, `around_update`, `before_destroy`, `after_destroy`, `around_destroy`, `after_commit`, `after_rollback`, `after_initialize`, `after_find`, `after_touch`
+- Reads Rails' per-event callback chains (`_save_callbacks`, `_create_callbacks`, and the other lifecycle events), preserving each chain's order and each entry's `kind`, filter and conditions. The public `type` combines kind and event, such as `before_save` or `after_create`; Rails' separate `before_commit` event is reported as `before_commit`, not `before_before_commit`. `callback_count` equals the emitted callback list's length. The list includes framework-registered callbacks; it is runtime metadata, not an application-only filter or a cross-event execution trace. Regenerate the index after upgrading to pick up corrected callback metadata.
+- Proc/lambda filters, including Rails-generated association callbacks, use stable source-site labels in both metadata and callback chunks: `#<Proc app/models/post.rb:12>` (or `lambda`). App paths are relative to `Rails.root`; external paths are retained and native procs use `native`. Rails 6's numeric filter identity is resolved through `raw_filter`. These labels describe location and callable kind, not captured closure state; callbacks are never executed. Model condition labels retain their existing format.
+- Default callback-object representations omit process addresses: an instance becomes `#<CleanupCallback>`, an anonymous class becomes `#<Class>`, and its instance becomes `#<#<Class>>`. Anonymous namespace prefixes are normalized too (for example, `#<Module>::CleanupCallback`). Named classes and custom `to_s` labels retain their text. These labels do not distinguish arbitrary object state; separate registered callbacks remain separate entries even when their descriptive labels match. Controller object-filter formatting is unchanged.
 - Callback side-effects are analyzed via `CallbackAnalyzer`: detects columns written (`self.col =`), jobs enqueued (`perform_later`), and services called
+- Reflects model class and instance methods after reading the schema, so Rails schema-loading optimizations produce the same method metadata in cold and warmed runs. Application-defined constructors remain visible; Rails versions that install an optimized singleton `new` during schema loading consistently include it in `class_methods`.
 - Automatically skips HABTM join models and anonymous classes
 - Chunks every model into semantic sections: `:summary`, `:associations`, `:callbacks`, `:validations`, `:scopes`, `:methods`
 - **Runtime-generated method detection:** Because extraction runs inside a booted Rails process, `instance_methods(false)` captures every method Rails generates dynamically, enum predicates (`status_active?`, `status_pending?`), association builders (`build_profile`, `create_line_item!`), attribute accessors, and dynamically registered scopes. Static analysis tools cannot see these methods because they only exist after Rails processes the DSL declarations at boot time
@@ -158,6 +162,8 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 - Route context is inlined in `source_code` as a comment header, not just in metadata
 - Chunks per-action: each action becomes a `:action` chunk with its applicable filters and route
 - Metadata includes permitted params (strong parameters), response formats, and applied filters per action
+- Inline callbacks use stable source-site labels in filter metadata, controller annotations and action chunks: `#<Proc app/controllers/posts_controller.rb:12>` (or `lambda`). The controller filter metadata and annotations use the same labels for `if`/`unless` procs. App paths are relative to `Rails.root`; external paths are retained and native procs use `native`. Labels describe the callable location and kind, not captured closure state, and never execute callbacks.
+- Route helper resolution accepts every live named controller/action route, including `file_path`, `image_url`, `download_path`, and `root_path`. Unknown filesystem/asset helpers produce no edge; matching names are conservative source references, not proof a call executes.
 - Extracts `redirect_to` navigation edges: named route helpers (`posts_path`, `users_url`) are resolved to controller targets via `RouteHelperResolver`, producing `:redirect_to` dependency edges (gated by `extract_navigation_edges` config)
 
 **Edge cases:**
@@ -194,6 +200,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 - Scans: `app/services`, `app/interactors`, `app/operations`, `app/commands`, `app/use_cases`
 - Extracts public entry points (`call`, `perform`, `execute`, `run`), custom error classes, and dependency references
 - File-based discovery (not class introspection), so it catches services with non-standard superclasses
+- `initialize_params` describes declared names, default presence and keyword status from Ruby syntax. Nested/comma-bearing defaults are not evaluated or treated as parameters; named rest, keyword-rest and block parameters retain their names. Anonymous forwarding has no name to report; malformed source produces an empty parameter list.
 
 **Example output (abbreviated):**
 
@@ -218,6 +225,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **Key details:**
 - Scans: `app/jobs`, `app/workers`, `app/sidekiq`
 - Extracts queue name, retry configuration, concurrency options, perform method arguments, and callbacks
+- `perform_params` uses the same syntax-aware signature parsing as service initializers and preserves its `name`, `splat` (`single`/`double`/null), and `has_default` fields. Keyword defaults do not invent additional argument names.
 - Records what triggers this job (reverse lookup via dependency graph after extraction)
 - Supports both ActiveJob and Sidekiq native workers
 
@@ -243,7 +251,8 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **What it captures:** ActionMailer classes with their mailer actions, defaults, template paths, callbacks, and helper usage.
 
 **Key details:**
-- Discovers via class introspection (`ActionMailer::Base.descendants`)
+- Discovers `ApplicationMailer.descendants` when that class exists, otherwise `ActionMailer::Base.descendants`; an app without ActionMailer contributes no mailer units.
+- Discovery and direct extraction accept only mailers backed by an existing app-owned source file, excluding dependency mailers and fabricated convention paths.
 - Each mailer action corresponds to an email template, template paths are recorded in metadata
 - Extracts `default from:`, `layout`, and per-action subject patterns
 
@@ -294,7 +303,8 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 
 **Key details:**
 - Extracts the entire stack as one unit (not one per middleware)
-- Records middleware class names, insertion order, and any initialization arguments
+- Records middleware class names, insertion order, and initialization arguments as readable strings
+- Argument rendering preserves literal strings and nested array/hash configuration. Procs use source locations; anonymous classes (including Ruby temporary names used by Rails executors/reloaders) use parent names and method source locations. Opaque objects using Ruby's default `to_s` are represented by class, without walking private runtime state. Custom `to_s` output is preserved, so application-defined nondeterministic renderers can still vary. Closure captures and opaque object internals are not serialized.
 - No per-file mapping, so incremental re-extraction re-runs `MiddlewareExtractor` wholesale when `config/application.rb`, `Gemfile.lock`, or a file under `config/initializers`/`config/environments` changes
 
 ---
@@ -331,6 +341,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **Key details:**
 - File-based scanning, no Rails boot needed for the actual file reading
 - Records which partials a template renders and which instance variables it expects
+- Loads the runtime route collection before caching named helpers, including Rails lazy route sets. Fresh-process incremental view extraction resolves the same navigation targets as full extraction.
 - Extracts navigation dependencies: `link_to` and `form_with`/`form_for` calls using `_path`/`_url` route helpers are resolved to controller targets via `RouteHelperResolver`
 - Navigation edges use `:link_to` and `:form_action` via types in the dependency array
 - Gated by `extract_navigation_edges` config (default: true)
@@ -469,6 +480,7 @@ class PageView < AnalyticsRecord; end   # metadata[:database] => "analytics"
 **Key details:**
 - Identifier is the package directory relative to `Rails.root` (`.` for the root package), the same name Packwerk uses
 - Honors `packwerk.yml` `package_paths` and `exclude`; without one, `**/` with the Packwerk default excludes (`bin`, `node_modules`, `script`, `tmp`, `vendor`)
+- With those exact defaults, excluded top-level directories are pruned before discovery descends into them, so an index or snapshots under `tmp/` do not add package-scan work. Custom patterns or exclusions retain their configured glob behavior. Hidden directories and symlink directories are not recursively followed by default.
 - `metadata`: `name`, `dependencies` (sorted), `enforce_dependencies` (`true`, `false`, or `"strict"`), `enforce_privacy`, `layer` (pks), `public_path`, `owner`
 - Each declared dependency becomes a `{ type: :package, target: <name>, via: :package_dependency }` edge
 - Package membership on other units (`metadata[:package]`, below) does not depend on how a unit was discovered: any registered unit with a file path under a package root is annotated. The undeclared cross-package edge report remains a follow-up, not this extractor. Discovery is the separate open gap: a pack-resident file-based unit is not yet found by `PathDispatcher` when only its `package.yml` changes (follow-up B-175), so it carries no membership only because it has no unit at all yet, not because membership skips it
@@ -530,7 +542,17 @@ Every app-owned unit under a package root carries `metadata[:package]` with the
 **Key details:**
 - Reads: `config/recurring.yml` (Solid Queue), `config/sidekiq_cron.yml` (Sidekiq Cron), `config/schedule.rb` (Whenever)
 - Extracts job class name, cron expression, queue, and any arguments
-- File-based (static read, no Rails introspection needed)
+- Resolves trusted application `recurring.yml` through Rails' configuration loader,
+  including ERB, filename-relative `require_relative`, and YAML aliases. On Rails
+  6.0 (before that loader existed), evaluates ERB with its filename and retains
+  safe YAML loading of scalars, hashes, arrays and symbols. ERB runs application
+  code in the extraction process; index only applications you trust.
+- Environment-wrapped task maps select the current Rails environment, including
+  custom names; an absent environment falls back to the first section, while an
+  explicitly empty section stays empty. Flat task maps remain supported.
+- Sidekiq-Cron remains safe-loaded YAML; Whenever remains a static DSL scan.
+  Invalid YAML/ERB, missing required files and runtime configuration errors are
+  logged and omit that schedule file. Source remains the original file text.
 - No per-file mapping, so incremental re-extraction re-runs `ScheduledJobExtractor` wholesale whenever one of the schedule files above changes
 
 ---
@@ -711,7 +733,16 @@ When written to disk, units also include:
 
 ### Git enrichment fields (`metadata[:git]`)
 
-If the host app is a git repo, the following are added to `metadata[:git]` after extraction:
+If the host app is a git repo, the following are added to `metadata[:git]` after extraction.
+History is limited to commits reachable from `HEAD` in the past 365 days,
+including merged branch history. Unmerged branches, remote refs, and tool
+checkpoint refs do not contribute. Commands run against the application root;
+when `WOODS_GIT_DIR` is set, `HEAD` belongs to that explicitly selected git
+directory, which may differ from a linked worktree's HEAD.
+
+After upgrading from a version that included all refs, run a full
+`woods:extract` to replace previously published git metadata. Incremental
+extraction refreshes only the units it rewrites.
 
 | Field | Description |
 |-------|-------------|

data/docs/FAQ.md

@@ -274,7 +274,13 @@ Verify the active v2 generation with `docker compose exec app bundle exec rake w
 
 ### How do I configure the Console Server with Docker?
 
-First set `config.console_mcp_enabled = true` in the Rails initializer after reviewing the live-data trust boundary. Stdio does not send a bearer token, but production Rails boot still requires a configured `console_mcp_token` of at least 32 characters whenever Console is enabled. Supply `WOODS_CONSOLE_MCP_TOKEN` through the container's secret mechanism; see [Console MCP setup](CONSOLE_MCP_SETUP.md#option-a-stdio-via-rake-recommended). Then, for the embedded mode (9 Tier 1 tools), point the MCP client at `docker compose exec -T` so Compose does not allocate a pseudo-TTY:
+First enable the master `console_mcp_enabled` switch after reviewing the
+live-data trust boundary. For stdio-only use, explicitly set
+`console_mcp_http_enabled = false`; HTTP remains enabled by default for
+compatibility and requires its bearer token while enabled. Follow
+[Console MCP setup](CONSOLE_MCP_SETUP.md#option-a-stdio-via-rake-recommended)
+for the configuration. Then point the embedded-mode client (9 Tier 1 tools)
+at `docker compose exec -T` so Compose does not allocate a pseudo-TTY:
 
 ```json
 {
@@ -414,7 +420,7 @@ When you run `rake woods:embed`, Woods generates embedding vectors for each extr
 Several options for tuning retrieval:
 
 - **Increase `max_context_tokens`** to include more units per query (at the cost of larger LLM context).
-- **Lower `similarity_threshold`** (default 0.7) to include less similar results.
+- **Use explicit retrieval scopes** and inspect ranking evidence. `similarity_threshold` is deprecated and does not filter results; see [retrieval tuning](RETRIEVAL_GUIDE.md#tuning).
 - **Enable framework sources** (`include_framework_sources: true`) if Rails internals are relevant to your queries.
 - **Use retrieval feedback only in a custom embedded server** that wires a feedback store. The normal packaged executable does not register feedback tools.
 
@@ -442,16 +448,9 @@ Snapshots prefer their own SQLite database (`woods.sqlite3` in the output direct
 
 The session tracer is middleware that records which Rails actions are invoked during a browser session, assembles the relevant extracted units, and makes that context available via the `session_trace` MCP tool. It is useful for giving an AI tool accurate context about what code path was active during a specific user interaction.
 
-Session tracing is disabled by default. To enable it:
-
-```ruby
-config.session_tracer_enabled = true
-config.session_store = Woods::SessionTracer::FileStore.new(
-  Rails.root.join('tmp/session_traces')
-)
-```
-
-The `session_store` option is required, there is no default store.
+Session tracing is disabled by default and requires an explicit `session_store`.
+Follow the [canonical configuration example](CONFIGURATION_REFERENCE.md#session-tracer-options)
+for store construction and review trace retention and access controls before enabling it.
 
 ---