woods 2.0.0.beta1 → 2.0.0.beta3

data/docs/DOCKER_SETUP.md

@@ -87,6 +87,12 @@ docker compose exec app bundle exec rake woods:extract_framework
 
 Run the watcher as its own development service or process-manager entry, not as a one-off terminal command. Docker Desktop bind mounts may not deliver reliable native filesystem events; set `WOODS_WATCH_POLL=1` for polling when needed. The watcher updates structural generations automatically, while semantic vectors still require `woods:embed_incremental`.
 
+When host-side tasks or one-off containers read the daemon's shared index,
+`WOODS_WATCH_TRUST_FOREIGN_HOST=1` lets those readers trust its recent heartbeat.
+Set it in each reader process; Docker does not forward host variables by default.
+See [cross-host liveness](WATCH_DAEMON.md#cross-host-liveness) for the 15-minute
+crash-detection bound and single-supervisor requirement.
+
 ### Index persistence
 
 Persist `tmp/woods/` if the index should survive container replacement. A bind mount also makes it available to optional host-side tools:
@@ -185,6 +191,16 @@ Use this only when the application bundle, a supported Ruby, and the Woods execu
 | **Survives container replacement** | Only with a bind/named volume | Yes, on host disk |
 | **Needs Ruby/Woods bundle on host** | No | Yes |
 
+### Index filesystem performance
+
+Bind mounts backed by virtiofs or FUSE can make each hardlink, rename and
+metadata lookup costly. Profile `payload seed`, writes and `payload prune`
+before tuning extraction. A container volume can reduce these costs, but it
+changes host visibility: use a distinct index location per worktree, run MCP
+where that path is visible, and update export/archive paths together. A single
+shared volume for every worktree would mix their indexes. See
+[incremental profiling](INCREMENTAL_EXTRACTION.md#profiling-fixed-costs).
+
 ## Console Server Setup
 
 The Console Server queries live Rails state. There are two launch paths for the same embedded server.
@@ -194,13 +210,16 @@ Before either path can start, deliberately enable live-data access in the Rails
 ```ruby
 Woods.configure do |config|
   config.console_mcp_enabled = true
-  config.console_mcp_token = ENV["WOODS_CONSOLE_MCP_TOKEN"]
+  config.console_mcp_http_enabled = false # stdio-only
 end
 ```
 
 The process exits with status 1 while this master switch is false. Review [Console MCP setup and security](CONSOLE_MCP_SETUP.md) before enabling it.
 
-Stdio does not send the bearer token, but production Rails boot still requires `WOODS_CONSOLE_MCP_TOKEN` to contain at least 32 characters whenever Console is enabled. Provide it to the container through the application's normal secret mechanism. Outside production, omitting it warns and leaves the Console HTTP endpoint guarded with 401.
+This explicitly disables HTTP Console while retaining stdio access; no HTTP
+token is needed at boot. Existing configurations default to HTTP enabled.
+For HTTP deployment, enable the HTTP flag and configure its token, origins
+and TLS using the [Console setup guide](CONSOLE_MCP_SETUP.md#option-c-http-rack-middleware).
 
 ### Comparison
 

data/docs/EVALUATION.md

@@ -14,6 +14,177 @@ Three harnesses, three questions.
 
 `woods:evaluate:baseline[grep|random|file_level]` scores a naive strategy on the same query set for comparison.
 
+## Paired lexical comparison (#404 / #227)
+
+`bench/evaluation/lexical_comparison.rb` runs **all 28 existing questions at their
+existing budgets** through semantic retrieval, lexical-only retrieval, a fixed
+query-seeded graph experiment, and the existing identifier-substring grep baseline.
+Unlike the strategy groups below, these conditions answer the same questions.
+The semantic condition replays the same captured real MiniLM vectors; labels,
+source snapshot and the original quality floors are unchanged.
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/lexical_comparison.rb /tmp/woods-paired.json
+# Use the pinned tokenizer environment described below:
+python bench/evaluation/capture_tokens.py /tmp/woods-paired.json lexical_comparison_capture.json
+```
+
+The second command writes `bench/evaluation/lexical_comparison_capture.json`.
+It retains every query's outcomes, context hash and exact token count, including
+misses. Raw returned contexts remain in `/tmp/woods-paired.json`. The capture binds
+its corpus/vector hashes and tokenizer provenance. It is reviewed experimental
+evidence, not a replacement for the required semantic gate.
+
+Initial Ruby 4.0.6 comparison (five warm pipeline repetitions per question):
+
+| Condition | Precision@5 | Recall | MRR | Mean actual context tokens | Median query latency (ms) |
+|---|---:|---:|---:|---:|---:|
+| Existing semantic pipeline | 0.536 | 0.580 | 0.857 | 1,000.7 | 3.019 |
+| Explicit lexical | 0.492 | 0.666 | 0.780 | 1,152.6 | 0.407 |
+| Lexical + query-seeded graph experiment | 0.396 | 0.476 | 0.750 | 1,159.8 | 3.292 |
+| Existing identifier grep baseline | 0.342 | 0.288 | 0.336 | 592.9 | 0.628 |
+
+Lexical retrieval improves recall on this small set while losing precision and
+first-hit rank against the semantic pipeline. This supports an explicit offline
+option, not semantic equivalence or a new default. The seeded graph condition
+loses on these measures and stays **evaluation-only**. It uses 20 iterations,
+restart probability 0.15, normalized positive lexical seeds, bidirectional
+recorded relationships within type eligibility, and no seeds for a no-match
+query. These choices were fixed before scoring; no weights were tuned to labels.
+The grep baseline matches identifiers, not source text; it retains its existing
+ordering, with eligibility applied before its limit and the same budgeted renderer.
+
+Latencies are warm in-process measurements over 62 units. They exclude lexical
+snapshot construction, live embedding/network latency and Rails extraction; they
+are not host-scale performance claims. Exact tokens count returned context only,
+using the existing `cl100k_base` capture machinery. Agent task outcomes were not
+measured. Ambiguous typed identities, concern/callback metadata and unrelated-hub
+controls also have targeted regressions; those synthetic controls do not replace
+representative host evaluation. Broader production/provider/agent evidence remains
+open under #227.
+
+## Checked-in Canopy retrieval gate
+
+The required CI `coverage` job runs this offline command before its test suite:
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/runner.rb
+# Report: tmp/retrieval-evaluation.json (uploaded by CI)
+```
+
+`bench/evaluation/` contains a versioned, deliberately small baseline:
+
+- **Corpus:** 62 runtime-extracted Canopy units and 28 annotated queries covering
+  publishing, billing, newsletters, and support. Source and relationships come
+  from the committed fictional `woods-testbed` Rails application; no application
+  records are included. Exact application/Woods revisions are in `corpus.json`.
+  This snapshot selects models, services, jobs, controllers, Pundit policies, and
+  mailers; it does not represent every extractor or a large production index.
+- **Embeddings:** real `all-MiniLM-L6-v2` ONNX inference, captured with a pinned
+  model revision and file hashes. CI replays those vectors through the current
+  production classifier, stores, graph, ranker, type fallback, and assembler.
+  No hash-based fake embeddings or expected-answer vectors are used.
+- **Gate:** per-strategy precision at 5, recall, and MRR must remain at least 95%
+  of the first observed score (floors rounded down to six decimal places).
+  Corpus/vector/token-provenance mismatches, missing strategies, and unexpected
+  strategy selection also fail. The baseline format is developer-only and is
+  **not** the `EVAL_BASELINE_FILE` aggregate-threshold format.
+
+B-190/B-191 recapture on Ruby 4.0.6 (five warmed pipeline repetitions per query; Ruby 3.3.1
+and 3.4.10 replay the same answers):
+
+| Strategy | Queries | Precision@5 | Recall | MRR | Mean actual context tokens |
+|---|---:|---:|---:|---:|---:|
+| Keyword | 4 | 0.292 | 0.500 | 0.750 | 1,020.8 |
+| Vector | 4 | 0.313 | 0.375 | 0.625 | 1,041.2 |
+| Graph | 8 | 0.813 | 0.519 | 1.000 | 1,041.1 |
+| Hybrid | 4 | 0.750 | 0.396 | 1.000 | 1,058.8 |
+| Direct with type filtering | 4 | 0.375 | 0.750 | 0.625 | 551.0 |
+| Within-type vector fallback | 4 | 0.400 | 1.000 | 1.000 | 1,250.8 |
+
+Precision@5 divides relevant hits by the actual returned slice size (up to five),
+not always by five. Recall divides retrieved relevant units by all annotated
+relevant units. MRR is the mean reciprocal rank of the first relevant hit; a
+query with no relevant hit contributes zero.
+
+`profiles.json` binds six measured Ruby runtimes to a shared context capture:
+Ruby 3.0.7, 3.1.7, 3.2.11, 3.3.1, 3.4.10, and 4.0.6. Other Ruby engines or
+major/minor versions fail with a recapture instruction. All six produce identical
+retrieved identifiers, context bytes, token counts and quality metrics for every
+query. The original positive per-runtime quality floors remain unchanged in
+`baseline.json` and `baseline_legacy.json`.
+
+B-191 orders equal PageRank scores lexically by identifier. B-192 orders hybrid
+candidates by descending score, identifier, then source before truncation and
+RRF; the same ordering selects the three vector seeds for graph expansion.
+Only `hybrid-2` changes from the prior Ruby 3.3–4.0 capture: `Newsletter::DeliverJob`
+now precedes `Newsletter::Delivery`, matching the prior Ruby 3.0–3.2 answer.
+No quality metric or exact token count changes. This is production retrieval
+ordering, with no answer reordering in the evaluation harness.
+
+These are separate annotated query groups, **not a controlled comparison of six
+strategies on identical questions**. Relevance annotations were written from
+fixture source and its functional contract before scoring. Initial graph and
+direct probes remain in the corpus; additional probes exercise the supported
+snake-case graph roots and actual within-type fallback. The B-190 fix now
+resolves all four originally empty graph queries, including `trace Billing::Invoice`
+and `trace ReviewAssignment`. The B-190 change affected those four graph answers. B-191 additionally
+changes vector and within-type fallback ranks or membership; all original queries,
+relevance labels, vectors, and quality floors remain unchanged. Graph precision/recall/MRR increased from 0.417 / 0.238 / 0.500 to
+0.813 / 0.519 / 1.000. Low semantic recall and editorial misses are
+also retained. The floors prevent further regression; they are not release
+quality targets or evidence that retrieval is already good enough.
+
+`capture_report.json` records per-query ranks, quality, latency ranges, and exact
+`cl100k_base` counts of the returned context. Those counts are bound to context
+SHA-256 values; the tokenizer vocabulary asset hash, pattern, special-token map,
+and package version are recorded alongside them. Changed context requires explicit recapture and review. Woods'
+`tokens_used` remains a separate character-ratio estimate. Neither number counts
+agent prompts, generated answers, or total agent usage. Vector capture records
+actual model-input token counts (including special tokens after truncation),
+separately from output-context tokens. The model truncates inputs to 512 tokens,
+so long source units are only partially represented.
+
+Pipeline latency excludes model loading, downloads, live embedding requests, and
+network time. `vectors.json` separately records the warm embedding batch time,
+input counts, model/package provenance, and SHA-256 of each model asset. Timing
+is observational and has no shared-runner CI threshold. This baseline does not
+validate OpenAI/Ollama quality, their latency, production-scale storage, or agent
+outcomes. The baseline scope of #227 is complete; broader provider and
+production-scale evaluation remain separate follow-up work.
+
+### Reproduce or review a new capture
+
+Ordinary CI needs only the Ruby bundle and the checked-in files. Model refresh is
+an explicit maintenance operation with isolated Python dependencies:
+
+```bash
+uv venv /tmp/woods-eval-venv
+uv pip install --python /tmp/woods-eval-venv/bin/python -r bench/evaluation/requirements.txt
+/tmp/woods-eval-venv/bin/python bench/evaluation/capture.py /tmp/woods-eval-model
+```
+
+This overwrites `vectors.json` with real inference from the pinned Apache-2.0
+model. To capture changed context after an intentional pipeline change:
+
+```bash
+bundle exec ruby -Ilib -r./bench/evaluation/runner -e \
+  'r=RetrievalBaseline::Runner.new; File.write("/tmp/retrieval-capture.json", JSON.pretty_generate(r.run))'
+/tmp/woods-eval-venv/bin/python bench/evaluation/capture_tokens.py /tmp/retrieval-capture.json
+```
+
+Replay all six measured Ruby lines before replacing the shared capture. Retain
+the exact capture runtime in the report and preserve each runtime’s original
+quality floors. A newly observed runtime difference needs separate reviewed
+captures rather than being hidden by answer reordering.
+
+Review every changed answer and score before updating the matching baseline digests or
+floors. Never lower a floor merely to make CI pass. Keep labels independent of
+rankings and preserve hard or failed queries. When changing the application
+snapshot, use a fresh runtime `woods:extract` into a disposable output directory,
+retain only deliberate source/relationship fields, and record both repository
+SHAs. Do not copy a host's live index or application records into the repository.
+
 ## Agent-level ablation
 
 Retrieval scores say whether the right units come back. They do not say whether an agent finishes the job. The ablation runs the same task set twice per task, once with the Woods MCP server available and once without, and compares resolution rate, tokens, cost, and turns.
@@ -80,7 +251,7 @@ bin/rails "woods:evaluate:ablation[config/eval_ablation.json]"
 
 The report holds every attempt (`task_id`, `condition`, `resolved`, `total_tokens`, `cost_usd`, `turns`, `duration_ms`, `error`, `provenance`) and a summary per condition (`resolution_rate`, `mean_tokens`, `mean_cost_usd`, `mean_turns`, `tasks`, `errors`) plus a `delta` (on minus off, present only when both conditions ran). Tokens are the sum of input, output, cache-creation, and cache-read tokens. `mean_tokens` is `null` when every trial in that condition errored before the agent produced JSON.
 
-A timeout (`AblationRunner.new(..., timeout: seconds)`, default 600) bounds every command a trial runs, applied independently to each one rather than as a single budget shared across the trial: worktree add/remove, the optional `reset`, the agent invocation, and the check. A timed-out `reset` or `check` counts as an error the same way a timed-out agent invocation does. When the default subprocess executor is in use, a timed-out command is terminated (`TERM`, then `KILL` if still alive after a short grace period) so it never outlives the trial that started it.
+A timeout (`AblationRunner.new(..., timeout: seconds)`, default 600) bounds every command a trial runs, applied independently to each one rather than as a single budget shared across the trial: worktree add/remove, the optional `reset`, the agent invocation, and the check. A timed-out `reset` or `check` counts as an error the same way a timed-out agent invocation does. The default subprocess executor creates an owned process group. On timeout it sends `TERM`, then `KILL` after a short grace period if necessary, to that group, including ordinary descendants whose parent already exited. Commands must not deliberately detach into separate sessions/groups; those escape this cleanup. Custom executors exposing only a PID retain single-process cleanup; executors exposing no PID must handle their own subprocess lifecycle.
 
 ### Where to get tasks
 
@@ -89,3 +260,295 @@ The Rails Foundation's "Agents on Rails" benchmark (announced 2026-08-13, built
 ### Reading the numbers
 
 A negative `delta.mean_tokens` with an equal or higher `delta.resolution_rate` is the result that sells the index, on the tasks actually run. It does not, by itself, establish that the index causes the difference: run at least ten tasks, two conditions on five tasks is noise, and remember that agent runs are not perfectly reproducible even at a fixed baseline SHA. Keep the baseline SHA and Woods generation fixed across a comparison (both are recorded per result), and do not let the `on` agent run `woods:extract` mid-task.
+
+## Explicit scope comparison
+
+`bench/evaluation/scope_comparison.rb` reuses #227's 62-unit Canopy extraction,
+28 original questions, captured MiniLM vectors, unchanged gold labels, and the
+same 1200-token budgets. Fixed domain-word rules choose the existing billing,
+newsletter, or support directories across models, controllers, jobs, and use
+cases. Other questions are unscoped controls. This assigns **no package ownership**
+to Canopy. Expected units outside a requested directory remain in the gold labels,
+so restricting scope can reduce recall for questions about cross-boundary behavior.
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/scope_comparison.rb /tmp/woods-scope-raw.json
+python bench/evaluation/capture_tokens.py /tmp/woods-scope-raw.json scope_comparison_capture.json
+```
+
+Use the pinned Python requirements in this directory's existing evaluation runbook.
+The capture records per-query scope, outcomes, context hashes, exact cl100k counts,
+and the existing precision@5, recall, and MRR metrics. Warm replay latency includes
+scope preparation and excludes provider/network time. This is a filtering and
+ranking comparison, not a coding-task or live-host performance claim.
+
+`spec/retrieval/package_scope_spec.rb` separately builds a synthetic Packwerk
+filesystem fixture and obtains root, nested, and sibling ownership from the real
+`PackageExtractor`. It verifies package boundaries without inventing ownership in
+a captured host corpus. Live pgvector/Qdrant tests cover a small eligible package
+behind globally stronger matches, typed chunks, and old vectors without metadata
+payload fields. Broader agent task-quality evidence remains unmeasured by this
+scope comparison.
+
+The reviewed capture uses 11 scoped questions and 17 unchanged controls per mode.
+On the 11 questions with a requested directory scope:
+
+| Mode | Scope | Precision@5 | Recall | MRR | Mean exact context tokens |
+|---|---|---:|---:|---:|---:|
+| Semantic | none | 0.5758 | 0.6530 | 0.9545 | 907.55 |
+| Semantic | explicit paths | 0.6364 | 0.5621 | 0.9091 | 820.73 |
+| Lexical | none | 0.5394 | 0.7364 | 0.8636 | 1201.45 |
+| Lexical | explicit paths | 0.6970 | 0.6455 | 0.8030 | 936.73 |
+
+Scope improves precision here while losing cross-boundary expected units. It is
+an explicit intent filter, not a default ranking improvement. Scoped lexical
+median query time was 2.30 ms versus 0.41 ms unscoped on these 11 replay questions;
+preparing the full metadata snapshot is visible even on this small corpus.
+Exact token counts measure the returned context, not MCP metadata or agent prompts;
+Woods' 1200-token budget is an estimate and can differ from cl100k counts.
+
+## Compact evidence comparison (#403)
+
+`bench/evaluation/evidence_comparison.rb` replays the same #227 Canopy corpus,
+28 questions, expected unit labels, captured MiniLM vectors and per-question
+budgets with full, compact and outline evidence, separately for semantic and
+lexical retrieval. Only within-unit evidence formatting varies within each
+strategy. The checked capture binds all 168 returned contexts to exact
+`tiktoken 0.11.0` `cl100k_base` counts; full-mode hashes match the approved prior
+comparison. The original labels are unit-level labels, not method/span relevance
+or task-correctness labels.
+
+| Strategy / evidence | P@5 | Unit recall | MRR | Mean exact context tokens |
+|---|---:|---:|---:|---:|
+| Semantic / full | 0.5363 | 0.5798 | 0.8571 | 1000.68 |
+| Semantic / compact | 0.4786 | 0.6476 | 0.8571 | 1034.68 |
+| Semantic / outline | 0.4536 | 0.8304 | 0.8571 | 976.86 |
+| Lexical / full | 0.4917 | 0.6661 | 0.7798 | 1152.64 |
+| Lexical / compact | 0.4881 | 0.7077 | 0.7887 | 1063.79 |
+| Lexical / outline | 0.3857 | 0.7964 | 0.7887 | 1014.75 |
+
+These results do not justify a default change. Outlines can expose more unit names
+without their method bodies; 19 of 100 returned semantic compact units and 13 of
+103 lexical compact units have no selected source span (the response explicitly
+reports omissions and may contain runtime metadata). Per-unit selected/omitted
+span counts remain in the capture. Increased unit recall is not evidence that an
+agent saw the implementation it needed. Compact semantic context consumed more
+actual tokens on average. Structured provenance and MCP envelopes are additional
+output tokens, outside the context-only counts above.
+
+Median warm query time across questions was 3.24/11.80/11.44 ms for semantic
+full/compact/outline and 0.46/4.44/6.08 ms for lexical modes. This measures local
+replay and includes source parsing; semantic provider/network time is excluded.
+It does not establish live-host performance.
+
+A separate deterministic long-prefix regression fixes the target method before
+selection: 80 unrelated methods precede `refund(payment)`. Under the same small
+context allowance the compact selector includes its entire body and exact
+published byte/hash coordinates. This is a synthetic retention test, separate
+from the real-corpus unit metrics above. Unicode, shared-line declarations, nested
+blocks, commented inlined concern display, inherited metadata, unknown physical
+locations and unknown generation are covered by source-evidence contract tests.
+Installed stdio and HTTP checks exercise generation attribution and typed,
+SHA-guarded full lookup on a real Rails extraction.
+
+Reproduce the corpus comparison without changing labels:
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/evidence_comparison.rb /tmp/woods-evidence-raw.json
+python bench/evaluation/capture_tokens.py /tmp/woods-evidence-raw.json evidence_comparison_capture.json
+```
+
+Use the same tokenizer version/vocabulary recorded by the capture. Raw contexts
+are an external artifact; the checked file retains hashes, exact counts, unit
+outcomes and source-span counts. Candidate ranking and ownership diversity are
+outside this experiment.
+
+### Public Writebook task attempts
+
+[`public_task_evidence_capture.json`](../bench/evaluation/public_task_evidence_capture.json)
+records twelve actual agent attempts on the Rails Foundation's public
+[`ai-evals`](https://github.com/rails/ai-evals) Writebook tasks at revision
+`5327efe54dd767d662ab89332f6a4285fef07bd8` (Writebook 1.2.1):
+`ar-announce-once`, `aj-enqueue-after-commit`, and `ar-archive-book-access`.
+Each task ran twice per condition with the same task prompt, client 2.1.267,
+resolved model `claude-sonnet-5`, 40-turn cap and Woods source revision
+`33abd6da8e072c7f510d052292aaff4260c30c7f`. Hidden verifiers remained outside the
+agent workspace until the attempt finished. Baseline failure/reference success
+prechecks and prompt/artifact hashes are retained in the capture.
+
+The intervention changed the default evidence mode at the MCP boundary. Full
+lookup kept its existing complete output; compact lookup used a 1200-estimated-token
+budget. Explicit evidence choices, metadata-only reads and full-source follow-ups
+remained available. This measures a default-mode change, not a forced mode on
+every call. All actual evidence calls used `lookup` (26 full-condition calls and
+20 compact-condition calls); there were no `codebase_retrieve` calls, query hints,
+or explicit full follow-ups. Consequently this does not establish retrieval
+ranking or query-guided compact evidence benefits.
+
+| Condition | Hidden task suites passed | Median MCP text tokens (cl100k) | Median agent wall time |
+|---|---:|---:|---:|
+| Full | 5/6 | 8661 | 180.3 s |
+| Compact | 4/6 | 2316 | 132.8 s |
+
+Both compact announcement attempts failed the go-live-then-return-to-draft edge;
+the second full attempt passed it. Both conditions passed both repetitions of the
+other tasks. Two compact `Accessable` lookups returned not-found errors; these are
+retained as valid tool errors, not discarded trials. The earlier 20-turn calibration
+pair hit the turn cap and failed the same announcement edge in both modes; the
+fixed 40-turn primary protocol and all failed/setup pilots are recorded separately.
+
+Compact delivered less MCP text in this small sample but passed one fewer task.
+That is a reason to keep it explicit and retain full-source verification, not to
+change the default. Two trials could overlap, so wall times are descriptive rather
+than isolated performance estimates. `cl100k_base` measures representation size,
+not the Claude tokenizer or billing. The capture separately records serialized MCP
+response counts, client-reported usage and list-price estimates; these must not be
+substituted for model-visible context or billed charges. It contains metrics,
+arguments and hashes, without shipping application source or agent transcripts.
+
+
+## Source-owner overlap experiment (#412)
+
+`bench/evaluation/owner_overlap_comparison.rb` evaluates an offline selector after
+ranking, using the production context assembler's full-source appender, section
+allocations, truncation and token estimator. It does not add a runtime option.
+The strongest result stays first. A later candidate moves ahead only when the
+higher-scoring pending span is already completely covered by delivered source.
+Several disjoint methods from one file remain separate; typed units are never
+merged. Exact-target/pinpoint queries and sections with one verified owner bypass
+the policy. Unknown origins block promotion across their position. Truncated
+parents establish no complete span coverage. Candidate scores remain unchanged.
+
+Origins require a unique exact byte substring in an explicitly supplied immutable
+source snapshot. Display paths alone, ambiguous repeated text, and synthesized
+inlined concern display do not prove physical ownership. Original-file and
+published-source hashes bind the verified spans. There are no query-time host-file
+reads. These are offline fixture origins, not newly published runtime provenance.
+
+The fixed fixtures define necessary methods and budgets before selection. Seven
+synthetic cases and one real Woods source case use actual `RubyAnalyzer` output;
+the latter is static source evidence, not a booted Rails application. Synthetic
+executable oracles load only delivered snippets into an isolated namespace and
+assert their fixed multi-method behavior. The wrapper supplies the declared class
+name for a method; it supplies no absent implementation. These deterministic task
+checks are separate from an LLM answer/task benchmark.
+
+| Fixture | Necessary-method recall, baseline → overlap | Executable fixture, baseline → overlap | Exact context tokens, baseline → overlap |
+|---|---|---|---|
+| Crowded class/method overlap | 0.5 → 1.0 | fail → pass | 312 → 308 |
+| Single owner | 1.0 → 1.0 | pass → pass | 312 → 312 |
+| Exact target / pinpoint (each) | 0.5 → 0.5 | fail → fail | 312 → 312 |
+| Truncated parent | 0.0 → 0.0 | fail → fail | 165 → 165 |
+| Disjoint methods in one file | 1.0 → 1.0 | pass → pass | 97 → 97 |
+| Unknown inlined display origin | 0.5 → 0.5 | fail → fail | 325 → 325 |
+| Woods storage identity / filename source | 0.5 → 1.0 | not measured | 336 → 366 |
+
+Relevant-owner coverage equals necessary-method recall in these fixtures because
+each required owner has one method, except the disjoint-method case where both
+methods share one owner and coverage remains 1.0. The real-source case retains
+`StorageIdentity.key` and `FilenameUtils#collision_safe_filename`; that is source
+retention, not proof that every helper needed to execute a change is present.
+The budget is Woods' existing character estimate: for example, the 230-token
+crowded fixture delivers 312/308 cl100k tokens. Exact counts measure context text,
+not prompts, provenance envelopes or agent output. No new token calibration or
+section-budget borrowing is introduced.
+
+The same #227 28 Canopy questions, gold labels, vectors and budgets serve as a
+negative control. All 62 units have distinct display paths and lack verified
+original spans. Both conditions therefore return identical contexts, matching
+the approved semantic full capture byte for byte. This demonstrates safe bypass;
+it cannot demonstrate owner-selection benefit on a runtime Rails index.
+Equal scores, typed collisions, unknown-origin barriers and separate
+primary/supporting/framework allocations have additional regression coverage.
+
+**Decision: keep this evaluation-only.** Fixed overlap cases benefit, but a
+truncated strongest class still consumes the whole allowance before complementary
+evidence can be selected. Real runtime provenance and actual agent answer/task
+correctness under this selector remain unmeasured. The Writebook task attempts
+above tested evidence formatting, not this policy, and cannot fill that gap.
+A production proposal needs trustworthy published physical spans and a paired
+representative task evaluation; no default change, owner quota, or near-relevance
+promotion is justified by these fixtures.
+
+```bash
+bundle exec ruby -Ilib bench/evaluation/owner_overlap_comparison.rb /tmp/woods-owner-overlap-raw.json
+python bench/evaluation/capture_tokens.py /tmp/woods-owner-overlap-raw.json owner_overlap_capture.json
+```
+
+The checked capture retains all 72 results, unchanged #227 gold labels, fixed-case
+requirements, source hashes, actual context counts/hashes, original scores and
+selection/deferral/budget-exhaustion traces. It retains failed cases. Use the
+recorded `tiktoken` version and vocabulary; keep raw contexts outside the checkout.
+
+
+## Matched optional context-hook tasks (#406)
+
+**Both conditions passed all four task attempts.** This small comparison does not
+establish a task-quality or efficiency improvement. Hints remained optional.
+The context condition returned less MCP text in aggregate, added its own context,
+and had a higher median agent wall time; individual repetitions varied.
+
+`bench/evaluation/hook_context_capture.json` records all eight preselected trials,
+returned-context hashes and actual token counts, official-client usage, emitted
+and delivered hints, callback latency and hidden-verifier outcomes. Two tasks
+from Rails Foundation's public `ai-evals` at
+`5327efe54dd767d662ab89332f6a4285fef07bd8` used disposable Writebook 1.2.1/Ruby 3.4.7
+images. Seeded defects failed and reference fixes passed before agent trials.
+Both arms used Claude Code 2.1.267, `claude-sonnet-5`, a 40-turn limit, the same
+prompt and source snapshot, full MCP evidence, and lexical retrieval. Refresh
+was disabled in both arms to isolate the independent context switch. Each
+condition ran twice per task; two trials ran concurrently. The hidden verifier
+was copied only after the agent exited.
+
+| Task attempt | Manual agent wall (s) | Context agent wall (s) | Hidden verifier, manual / context |
+|---|---:|---:|---|
+| `aj-enqueue-after-commit`, rep 1 | 111.3 | 113.6 | pass / pass |
+| `aj-enqueue-after-commit`, rep 2 | 173.4 | 154.9 | pass / pass |
+| `ar-atomic-import`, rep 1 | 105.3 | 210.4 | pass / pass |
+| `ar-atomic-import`, rep 2 | 126.5 | 119.9 | pass / pass |
+
+| Measure | Manual tools | Context enabled |
+|---|---:|---:|
+| Tasks passed | 4 / 4 | 4 / 4 |
+| Median agent wall (s) | 118.9 | 137.4 |
+| Total MCP text, actual cl100k tokens | 45,834 | 39,550 |
+| Additional delivered hint tokens, cl100k | 0 | 1,914 |
+| Emitted / delivered hints | 0 / 0 | 16 / 16 |
+| Client-reported input tokens | 160 | 158 |
+| Client-reported cache creation tokens | 217,920 | 206,056 |
+| Client-reported cache read tokens | 5,715,709 | 5,467,210 |
+| Client-reported output tokens | 40,884 | 42,000 |
+
+The 16 context callbacks took 0.590–0.818s (median
+0.613s); the largest complete JSON envelope was 1,067 bytes.
+Disabled callbacks took 1.75ms median and emitted nothing.
+Every emitted hint appeared in the official client's `hook_additional_context`
+attachments. A separate native-plugin probe also matched SessionStart and
+PostToolUse hint strings inside captured model requests. Attachment delivery does
+not establish that the model used a hint. Callback measurement wraps the actual
+entrypoint and excludes the instrumentation wrapper's own startup; agent wall
+includes the whole client run. OS scheduling and cold startup can still cause a
+silent timeout under the fixed deadline.
+
+An independent review checked all 44 rendered relationship claims against each
+trial's typed graph and found no incorrect claim. Useful owners were often
+already inspected; broad transitive candidates were sometimes redundant. One
+atomic-import run explicitly followed the DemoContent candidate and found no
+matching call site. New test files received honest unresolved-path notices.
+Source freshness was initially unknown because of `unverified_consumption_scopes`,
+then edits were marked drifted; truncated and pre-refresh qualifications remained
+visible. Correct candidates are not proof of task benefit or complete impact.
+
+Exact cl100k counts measure representation size, separately from the client's
+repeated input/cache/output accounting. The client's summed list-price estimates
+were $2.424 / $2.338; these are not billed charges or evidence of a
+repeatable saving. This two-task sample and overlapping execution cannot establish
+isolated performance or general quality. Raw traces remain local review artifacts;
+the checked capture binds their hashes. `diff.patch` covers tracked files only;
+new untracked tests are preserved through native Write/Edit records and separately
+labelled transcript reconstructions, not claimed as disk-captured final trees.
+
+The measured hook implementation is `21a578d6`; subsequent main integration left
+its helper and entrypoint bytes unchanged. The earlier smoke setup missing new
+bundle executable stubs is retained as an invalid setup attempt. The fixed
+primary trials all completed and none was discarded or replaced.