@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/runtime/output-router-benchmark-1-report.md

@@ -0,0 +1,81 @@
+# Output Router Benchmark Report - Iteration 1
+
+Date: 2026-06-19
+
+## Scope
+
+Deterministic, CI-friendly tool benchmark for Freeflow Router retrieval behavior. The runner compares a native text-search proxy, a pre-hardening Freeflow-style proxy, and the improved Freeflow Router implementation. Optional external comparators are recorded as skipped rather than failed.
+
+Reduction percentages compare routed/context bytes and approximate tokens against the raw source or direct output for that mode. Negative reduction means structured routing overhead is larger than the tiny raw output; native exact-search proxies can still be smaller than the router for simple lookups.
+
+## Baseline Caveat
+
+The pre-hardening Freeflow mode is a deterministic proxy for the old line-scoring behavior, not a checkout of an older runtime. It is useful for stable regression pressure, not historical performance archaeology.
+
+## Command
+
+```sh
+npm run bench:router
+```
+
+The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
+
+## Summary
+
+- Iterations per mode: 3
+- Fixtures: 7
+- Improved Freeflow Router gated pass: 7/7
+- Native baseline proxy pass: 3/7
+- Pre-hardening Freeflow proxy pass: 2/7
+- Generated false positives observed: 4/21 mode results
+- Improved generated false positives: 0/7
+- Improved weighted byte/token reduction: 98.54% / 98.54% (511618/127907 raw to 7473/1870 routed)
+- Improved average byte/token reduction: -235.00% / -231.39%
+- Improved median byte/token reduction: -105.94% / -105.77%
+- Improved path/span/excerpt checks: 7/7 path, 7/7 span, 7/7 excerpt
+- Sandbox failure fixed: yes
+
+## Results
+
+| fixture | mode | correctness | checks | path | lines | raw bytes/tokens | routed bytes/tokens | byte/token reduction | latency p50/p95 ms | recovery | notes |
+| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
+| exact-copied-text-block | native-baseline-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 138/35 | 78/20 | 43.48%/42.86% | 0.28/5.15 | not-applicable | score=28.00 |
+| exact-copied-text-block | pre-hardening-freeflow-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 138/35 | 78/20 | 43.48%/42.86% | 0.24/0.26 | failed | score=28.00 |
+| exact-copied-text-block | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 138/35 | 903/226 | -554.35%/-545.71% | 0.80/2.55 | passed | Deterministic repo retrieval selected 1 candidate(s); top result target.md:1-3 (matched exact normalized query phrase in section chunk).; recovery=passed: Verified repo retrieve target.md:1-3. |
+| markdown-heading-nearby-body | native-baseline-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 169/43 | 44/11 | 73.96%/74.42% | 0.18/0.22 | not-applicable | score=20.00 |
+| markdown-heading-nearby-body | pre-hardening-freeflow-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 169/43 | 44/11 | 73.96%/74.42% | 0.17/0.27 | failed | score=20.00 |
+| markdown-heading-nearby-body | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 3-6 | 169/43 | 954/239 | -464.50%/-455.81% | 0.67/0.68 | passed | Deterministic repo retrieval selected 1 candidate(s); top result target.md:3-6 (BM25-style scored section chunk with 6/6 query-token coverage).; recovery=passed: Verified repo retrieve target.md:3-6. |
+| generated-artifact-decoy | native-baseline-proxy | fail | path ✗ span ✓ excerpt ✓ gen-fp ✗ | graphify-out/graph.html | 1-3 | 310220/77555 | 310058/77515 | 0.05%/0.05% | 1.42/1.57 | not-applicable | score=140000.00 |
+| generated-artifact-decoy | pre-hardening-freeflow-proxy | fail | path ✗ span ✓ excerpt ✓ gen-fp ✗ | graphify-out/graph.html | 1-3 | 310220/77555 | 310058/77515 | 0.05%/0.05% | 1.37/1.43 | failed | score=140000.00 |
+| generated-artifact-decoy | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-9 | 310220/77555 | 1396/349 | 99.55%/99.55% | 1.09/1.24 | passed | Deterministic repo retrieval selected 1 candidate(s); top result docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-9 (BM25-style scored section chunk with 7/7 query-token coverage).; recovery=passed: Verified repo retrieve docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-9. |
+| huge-single-line-decoy | native-baseline-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✗ | debug.log | 1-1 | 192091/48023 | 192013/48004 | 0.04%/0.04% | 0.41/0.46 | not-applicable | score=24000.00 |
+| huge-single-line-decoy | pre-hardening-freeflow-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✗ | debug.log | 1-1 | 192091/48023 | 192013/48004 | 0.04%/0.04% | 0.43/0.57 | failed | score=24000.00 |
+| huge-single-line-decoy | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | docs/router.md | 1-3 | 192091/48023 | 946/237 | 99.51%/99.51% | 0.73/0.81 | passed | Deterministic repo retrieval selected 1 candidate(s); top result docs/router.md:1-3 (BM25-style scored section chunk with 6/6 query-token coverage).; recovery=passed: Verified repo retrieve docs/router.md:1-3. |
+| ambiguous-multi-file-query | native-baseline-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 8262/2066 | 8100/2025 | 1.96%/1.98% | 0.26/0.28 | not-applicable | score=3600.00 |
+| ambiguous-multi-file-query | pre-hardening-freeflow-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 8262/2066 | 8100/2025 | 1.96%/1.98% | 0.26/0.30 | failed | score=3600.00 |
+| ambiguous-multi-file-query | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 8262/2066 | 947/237 | 88.54%/88.53% | 1.03/1.05 | passed | Deterministic repo retrieval selected 1 candidate(s); top result target.md:1-3 (BM25-style scored section chunk with 6/6 query-token coverage).; recovery=passed: Verified repo retrieve target.md:1-3. |
+| vaulted-output-query | native-baseline-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | ffout_91e882b51880dd5b9bbbc3ac:stderr | 1-3 | 115/29 | 115/29 | 0.00%/0.00% | 0.01/0.07 | not-applicable | score=12.00 |
+| vaulted-output-query | pre-hardening-freeflow-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | ffout_91e882b51880dd5b9bbbc3ac:stderr | 1-3 | 115/29 | 115/29 | 0.00%/0.00% | 0.00/0.01 | failed | score=12.00 |
+| vaulted-output-query | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | ffout_91e882b51880dd5b9bbbc3ac:stderr | 1-3 | 115/29 | 1044/261 | -807.83%/-800.00% | 0.39/0.64 | passed | Deterministic lexical retrieval selected vaulted outputId=ffout_91e882b51880dd5b9bbbc3ac stream=stderr lines=1-3.; recovery=passed: Verified vault retrieve ffout_91e882b51880dd5b9bbbc3ac:stderr:1-3. |
+| expand-narrow-evidence | native-baseline-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-24 | 623/156 | 623/156 | 0.00%/0.00% | 0.06/0.08 | not-applicable |  |
+| expand-narrow-evidence | pre-hardening-freeflow-proxy | fail | path ✗ span ✓ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 3824/956 | 3200/800 | 16.32%/16.32% | 0.17/0.18 | failed | score=800.00 |
+| expand-narrow-evidence | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-24 | 623/156 | 1283/321 | -105.94%/-105.77% | 0.87/0.93 | passed | Expanded repo evidence ev_0ba9afec918a3334 to target.md:1-24.; recovery=passed: Verified repo retrieve target.md:1-24. |
+
+## Not Yet Measured
+
+- recall@3 / alternate candidates: not measured in this first deterministic runner.
+- explanation quality: not measured beyond route reason capture.
+- command-output parser benchmarks: deferred to the command benchmark track.
+
+## Skipped External Comparators
+
+- Graphify: Optional external comparator is not required for CI-friendly router benchmarks.
+- Claude Context: Optional semantic/hybrid search comparator is skipped unless configured separately.
+- RTK: Command-output comparator belongs to the later command benchmark track.
+- Squeez: Session-efficiency comparator belongs to the later command/session benchmark track.
+
+## Regression Status
+
+Improved Freeflow Router passed all gated benchmark fixtures.
+
+The generated-artifact decoy benchmark preserves the original Sandbox Permissions false-positive shape and records it as fixed when the improved router selects the docs target instead of `graphify-out/graph.html`.

package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md

@@ -0,0 +1,35 @@
+# Output Router Capture/Derive/Provider Eval - Iteration 1
+
+Date: 2026-06-23
+
+## Scope
+
+Targeted deterministic eval for Slice 9 of the universal output capture/derive work. It compares direct/raw evidence shapes against explicit Freeflow routing for read-only provider capture, web-shaped capture, deterministic derive, and provider-summary tool choice.
+
+This eval does not claim broad superiority. It verifies bounded evidence, exact recovery where promised, lineage, category-scoped provider summaries, and non-injection of invalid custom manifest text for these fixtures.
+
+## Command
+
+```sh
+npm run build && node evals/scripts/run-output-router-capture-derive-eval.js
+```
+
+## Summary
+
+- Fixtures: 4
+- Objective gates passed: 14/14
+- Direct host-tool capture: not evaluated and remains off by default.
+- Mutating provider tools: not mediated by `freeflow_capture`.
+
+## Results
+
+| fixture | direct/raw baseline | Freeflow routed behavior | status | gates |
+| --- | --- | --- | --- | --- |
+| direct-readonly-provider-vs-freeflow-capture | 1268 raw bytes would enter context from direct provider output | 233 evidence bytes entered context; raw recovery=exact | pass | readOnlyCaptured ✓; boundedEvidence ✓; exactRecovery ✓ |
+| web-shaped-capture-and-recovery | 1682 raw bytes from web-shaped output | 301 evidence bytes plus exact recovery | pass | webProducerCaptured ✓; boundedEvidence ✓; exactRecovery ✓ |
+| long-log-manual-inspection-vs-freeflow-derive | 7729 raw log bytes for manual inspection | 176 filtered evidence bytes; countMatches=2; exact derived recovery | pass | manualBaselineWouldReadWholeLog ✓; filteredTargetFacts ✓; countMatches ✓; lineage ✓ |
+| provider-summary-tool-choice-accuracy | Raw provider docs/manuals are not injected | 458 summary bytes; diagnostics category only; invalid custom guidance omitted | pass | compactSummary ✓; categoryScoped ✓; invalidCustomNotInjected ✓; mutationBoundaryPresent ✓ |
+
+## Result
+
+All targeted Slice 9 capture/derive/provider-summary gates passed for these deterministic fixtures.

package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md

@@ -0,0 +1,150 @@
+# Output Router Codex Large Repo Index/SQLite Ad Hoc Benchmark
+
+Date: 2026-06-18
+
+## Scope
+
+Ad hoc large-repo benchmark requested during Slice 8 follow-up. This is not an adoption decision and did not change product defaults.
+
+Repository:
+
+- URL: `https://github.com/openai/codex`
+- Local path: `/tmp/freeflow-codex-large-benchmark/codex`
+- Commit: `e12dd73`
+- Git-tracked files: 5,118
+- Working tree size after shallow clone: ~70 MB
+
+Modes measured:
+
+1. Existing `freeflow_retrieve` scanner default.
+2. Slice 8 no-dependency experimental local index.
+3. Improved no-dependency prototype with slimmer symbol/section index.
+4. Python stdlib SQLite FTS5 ad hoc comparator.
+5. SQLite FTS5 with deterministic reranking and tuned candidate limit.
+6. Scanner-hardening changes derived from this benchmark.
+
+SQLite/FTS used only Python standard-library `sqlite3` with FTS5 available. No npm/package dependency was added.
+
+## Baseline Result
+
+| mode | build/load | query p50 | query p95 | strict path pass | avg context bytes | notes |
+| --- | ---: | ---: | ---: | ---: | ---: | --- |
+| scanner default | none | 5,663 ms | 8,929 ms | 3/8 | 1,230 | live scan each query |
+| Slice 8 no-dep local index | failed after 10,013 ms | n/a | n/a | n/a | n/a | `RangeError: Invalid string length` while serializing cache |
+| SQLite FTS5 AND | 3,934 ms | 0.18 ms | 2.24 ms | 2/8 | 135 | very fast but low recall from strict AND query construction |
+| SQLite FTS5 OR + camel split | 8,145 ms | 42.97 ms | 50.94 ms | 3/8 | 322 | same strict pass count as scanner, much faster query-only |
+
+## Improvement Attempts
+
+Changes tried in temp prototypes:
+
+- no-dep index: remove duplicated line-window text; use symbol/Markdown chunks plus coarse blocks; store token counts and metadata only.
+- no-dep ranking: camel/Pascal split, exact compound boosts, source/test path priors, symbol-definition boosts, path-intent boosts, phrase-sequence boosts, candidate prefiltering.
+- SQLite: OR query for recall, ordered FTS candidate set, deterministic reranking with the same scoring features, candidate-limit tuning.
+
+Best observed variants:
+
+| mode | build/load | query p50 | query p95 | strict path pass | avg context bytes | cache/size | notes |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| no-dep V1 slim | 42,496 ms | 23,814 ms | 42,197 ms | 5/8 | 1,397 | 169.5 MB JSON estimate | accuracy improved, speed unacceptable |
+| no-dep V2 candidate prefilter | 73,755 ms | 1,973 ms | 2,933 ms | 5/8 | 1,542 | 156.4 MB JSON estimate | query faster, build/cache unacceptable |
+| no-dep V3 symbol/section only | 41,048 ms | 1,484 ms | 2,720 ms | 5/8 | 1,413 | 97.5 MB JSON estimate | best no-dep profile, still heavy |
+| SQLite FTS rerank limit=1000 | 1,936 ms | 137 ms | 287 ms | 5/8 | 1,128 | in-memory FTS | best overall prototype |
+
+## Scanner Hardening Applied
+
+Changes applied to the default scanner from this evidence:
+
+- skip broad traversal of `generated/` directories while preserving explicit generated-path retrieval,
+- add code symbol chunks for enum/struct/function/impl-style matches,
+- split CamelCase/PascalCase/snake/path identifiers for scoring,
+- preserve stopwords for exact phrase matching while keeping scoring stopword-filtered,
+- add bounded symbol chunk evidence instead of narrow ±2-line snippets,
+- add source/test path priors,
+- add definition, path-intent, ordered-phrase, and complete-query-coverage boosts,
+- use structural chunks first and line-window chunks only as a fallback for unstructured files.
+
+Follow-up speed pass kept the same scanner result while reducing overhead:
+
+- build BM25 stats only for chunks that pass the cheap `chunkMightMatch` prefilter,
+- memoize token and phrase-sequence normalization per chunk,
+- reuse tokenized chunk content for ordered-phrase scoring,
+- read eligible repo text files with bounded concurrency while preserving deterministic traversal order,
+- skip expensive identifier-splitting regexes for plain tokens without separators or case boundaries.
+
+Measured scanner-hardening result on the same Codex repo/query set:
+
+| mode | query p50 | query p95 | strict path+excerpt pass | avg context bytes | notes |
+| --- | ---: | ---: | ---: | ---: | --- |
+| scanner before hardening | 5,663 ms | 8,929 ms | 3/8 | 1,230 | baseline live scan |
+| scanner after hardening | 8,776 ms | 12,668 ms | 5/8 | 2,178 | more accurate but slower and larger context |
+| scanner after speed pass | 5,112 ms | 8,269 ms | 5/8 | 2,307 | accuracy kept; latency below baseline scanner p50/p95 |
+| scanner after review-fix pass | 5,473 ms | 8,611 ms | 6/8 | 2,082 | restored regressed source-definition queries and kept MCP test-query improvement |
+| scanner after deep-module cleanup rerun | 4,305 ms | 6,773 ms | 6/8 | 2,082 | final verification after `RepoTraversalPolicy`, exact line ranges, `EvidenceRangeSelector`, vault locking, command benchmark observation changes, and the post-review exact-phrase fixes |
+| scanner after architecture-deepening report refresh | 4,272 ms | 6,679 ms | 6/8 | 2,082 | report-refresh verification after bounded evidence, BenchmarkHarness, EvidenceSearch, RouterContract, and experimental-index capsule slices |
+
+Interpretation: scanner hardening produced a real accuracy gain (**+2/8 strict fixtures**) but initially regressed latency/context. The speed pass recovered the latency regression and kept the accuracy gain, although context remained larger because symbol chunks return fuller evidence. The later review-fix pass improved strict accuracy to **6/8** by constraining symbol complete-coverage boosts, preserving exact-phrase evidence ranges, adding Markdown preamble chunks, and tightening generated/media-path filtering while keeping scanner latency near the speed-pass result.
+
+## Before/After Summary
+
+Against scanner default:
+
+- Scanner hardening plus the review-fix pass improved accuracy from **3/8** to **6/8**; the latest report-refresh rerun kept **6/8** and kept query p50 below the original scanner baseline (**5,663 ms → 4,272 ms**).
+- Accuracy improved from **3/8** strict expected-path matches to **5/8** for no-dep V3, SQLite FTS rerank, and scanner hardening/speed pass, then to **6/8** for the default scanner after review fixes.
+- SQLite FTS rerank query p50 improved from **5,663 ms** to **137 ms**: about **41× faster query-only**.
+- SQLite FTS rerank query p95 improved from **8,929 ms** to **287 ms**: about **31× faster query-only**.
+- Scanner total for eight baseline queries was about **47.6 s**; initial scanner-hardening total was about **78.1 s**; after the speed pass it was about **45.5 s**; after review fixes it was about **53.5 s** in the observed run; the latest report-refresh rerun was about **38.6 s** for the scanner rows.
+- SQLite FTS rerank total was about **3.3 s including build**: about **14× faster end-to-end for this query batch** versus baseline scanner.
+- SQLite FTS rerank context was slightly smaller than baseline scanner: **1,230 avg bytes → 1,128 avg bytes** (~8% lower). Scanner hardening increased context to **2,178-2,307 avg bytes** because symbol chunks return fuller evidence.
+- No-dep V3 warm queries were faster than baseline scanner, but cold build/cache costs made it unattractive on this repo.
+
+## Misses Remaining
+
+The best strict result is now **6/8**. The repeated misses indicate ranking/query-intent problems, not just speed problems:
+
+- Some broad queries legitimately have multiple plausible targets.
+- Test/source intent is hard to infer from query text alone.
+- Path-intent queries like `apply_patch prompt codex prompts lib` remain sensitive: current scanner evidence includes the expected excerpt, but strict path grading still fails because it selects the deeper implementation handler instead of the expected prompt source.
+- FTS candidate generation and deterministic reranking need a more principled query classifier before automatic routing is safe.
+
+## Interpretation
+
+- The current no-dependency local index is **not ready for large repos**. It needs a more compact cache format and fewer or more targeted chunks before it is worth productizing.
+- SQLite/FTS is **promising enough to keep as an optional benchmark/comparator**. It produced better strict accuracy than scanner and much better speed in this ad hoc run.
+- SQLite/FTS is **not ready to become product behavior**. Accuracy is still mid, and adding SQLite/FTS has packaging/portability implications that need an explicit dependency decision.
+- A future `hybrid:auto` could be valuable only after a proper benchmark shows reliable query classification, fallback behavior, and no accuracy regression.
+
+## Decision
+
+- Keep scanner as default for now, with the applied hardening guarded by tests.
+- Keep no-dependency local index experimental; redesign its large-repo chunk/cache strategy before further adoption discussion.
+- Do not add SQLite/FTS as a product dependency yet.
+- Keep SQLite/FTS as an optional large-repo benchmark comparator candidate.
+- Do not implement `hybrid:auto` yet; the evidence supports future research, not default behavior.
+- Before further accuracy hardening, keep the speed-pass benchmark in the loop so ranking gains do not silently become latency regressions.
+
+## Raw Evidence Pointers
+
+Temporary files from this ad hoc run:
+
+- `/tmp/freeflow-codex-large-benchmark/scanner-sqlite-results.json`
+- `/tmp/freeflow-codex-large-benchmark/sqlite-results.json`
+- `/tmp/freeflow-codex-large-benchmark/sqlite-or-results.json`
+- `/tmp/freeflow-codex-large-benchmark/improved-index-results.json`
+- `/tmp/freeflow-codex-large-benchmark/improved-index-v2-results.json`
+- `/tmp/freeflow-codex-large-benchmark/improved-index-v3-results.json`
+- `/tmp/freeflow-codex-large-benchmark/results-symbols-only.json`
+- `/tmp/freeflow-codex-large-benchmark/scanner-sqlite-final-green-output.json`
+- `/tmp/freeflow-codex-large-benchmark/scanner-sqlite-speed-pass-output.json`
+- Post-review-fix rerun command: `node /tmp/freeflow-codex-large-benchmark/bench-scanner-sqlite.mjs` on 2026-06-18/19 session; observed scanner summary `pass=6/8`, `p50=5,473 ms`, `p95=8,611 ms`, `avgContextBytes=2,082`.
+- Deep-module cleanup final verification command: `node /tmp/freeflow-codex-large-benchmark/bench-scanner-sqlite.mjs`; observed scanner summary `pass=6/8`, `p50=4,305 ms`, `p95=6,773 ms`, `avgContextBytes=2,082`.
+- Architecture-deepening report-refresh command: `node /tmp/freeflow-codex-large-benchmark/bench-scanner-sqlite.mjs`; observed scanner summary `pass=6/8`, `p50=4,272 ms`, `p95=6,679 ms`, `avgContextBytes=2,082`, scanner row total about `38.6 s`.
+- Latest report-refresh scanner rows:
+  - `sandbox-permissions-enum`: pass, `codex-rs/protocol/src/models.rs`, lines `47-66`, 1,759 context bytes.
+  - `mcp-tool-call-event`: pass, `codex-rs/analytics/src/analytics_client_tests.rs`, lines `4207-4211`, 1,409 context bytes.
+  - `approval-mode-cli-arg`: pass, `codex-rs/utils/cli/src/approval_mode_cli_arg.rs`, lines `9-28`, 1,845 context bytes.
+  - `turn-diff-tracker`: fail strict path, selected `codex-rs/core/src/tools/handlers/unified_exec_tests.rs`, lines `3-35`, excerpt complete, 2,350 context bytes.
+  - `model-provider-info`: pass, `codex-rs/ollama/src/lib.rs`, lines `63-79`, 1,467 context bytes.
+  - `network-proxy-spec`: pass, `codex-rs/core/src/config/network_proxy_spec.rs`, lines `24-31`, 1,296 context bytes.
+  - `codex-http-client`: pass, `codex-rs/codex-client/src/default_client.rs`, lines `21-50`, 1,794 context bytes.
+  - `apply-patch-prompt`: fail strict path, selected `codex-rs/core/src/tools/handlers/apply_patch.rs`, lines `330-409`, excerpt complete, 4,737 context bytes.

package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md

@@ -0,0 +1,46 @@
+# Codex Structured Q&A Macro Benchmark Report - Iteration 1
+
+Date: 2026-06-19
+
+## Scope
+
+Fast, deterministic Stage 1 macro benchmark for the Codex CLI agent-harness research corpus. The benchmark asks one structured Q&A question derived from the Sandbox Permissions pass and grades retrieval-backed answer quality, citation quality, evidence quality, context size, and latency.
+
+The fixture uses existing Freeflow research docs as oracle scaffolding and includes a generated `graphify-out` decoy to preserve the original broad-retrieval failure shape. Upstream Codex source citation comparison is marked skipped until a source snapshot is explicitly supplied.
+
+## Command
+
+```sh
+npm run bench:router:codex-qa
+```
+
+The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
+
+## Summary
+
+- Iterations per mode: 1
+- Fixtures: 1
+- Improved Freeflow Router gated pass: 1/1
+- Native broad-search proxy pass: 0/1
+- Sandbox failure fixed: yes
+- Generated false positives observed: 1/2 mode results
+- Improved answer/citation/evidence: 1/1 answer, 1/1 citation, 1/1 evidence
+- Improved weighted byte/token reduction: 99.50% / 99.50% (580499/145125 raw to 2892/723 context)
+
+## Results
+
+| fixture | mode | correctness | checks | path | lines | raw bytes/tokens | context bytes/tokens | byte/token reduction | latency p50/p95 ms | proxy calls | answer | notes |
+| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- | --- |
+|sandbox-permissions-structured-qa | native-broad-search-proxy | fail | answer ✗ citation ✗ evidence ✗ gen-fp ✗ | graphify-out/graph.html | 1-3 | 580499/145125 | 580058/145015 | 0.08%/0.08% | 11.67/11.67 | 1 | Sandbox Permissions evidence from graphify-out/graph.html:1-3 did not include enough plain-language definitions to answer. | lexical score=40000 |
+|sandbox-permissions-structured-qa | improved-freeflow-router | pass | answer ✓ citation ✓ evidence ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 1-28 | 580499/145125 | 2892/723 | 99.50%/99.50% | 3.66/3.66 | 2 | Sandbox Permissions are documented at docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:1-28. UseDefault: run with the turn's normal sandbox. RequireEscalated: request unsandboxed execution. WithAdditionalPermissions: stay sandboxed but widen permissions for this one command. | Deterministic repo retrieval selected 1 candidate(s); top result docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-17 (BM25-style scored section chunk with 14/14 query-token coverage).; Expanded repo evidence ev_178fc507730e20fc to docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:1-28. |
+
+## Fixture Questions
+
+- sandbox-permissions-structured-qa: Find the Sandbox Permissions section, report file/lines, and explain UseDefault, RequireEscalated, and WithAdditionalPermissions.
+  - expected doc: `docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md`
+  - upstream source citation comparison: skipped-unavailable
+
+## Skipped External Comparators
+
+- Graphify: Skipped unless a fresh graph for the benchmark fixture is explicitly supplied.
+- Claude Context: Skipped unless an index is configured and fresh for the benchmark fixture.

package/evals/reports/runtime/output-router-command-benchmark-1-report.md

@@ -0,0 +1,62 @@
+# Output Router Command Benchmark Report - Iteration 1
+
+Date: 2026-06-19
+
+## Scope
+
+Deterministic, CI-friendly command-output benchmark for `freeflow_run`. The runner compares native direct output against the improved Freeflow command router and records optional RTK/Squeez comparators as skipped unless a caller supplies configured comparator hooks.
+
+Reduction percentages compare routed/context bytes and approximate tokens against raw command output bytes for each fixture. Correctness is gated on execution status, exit code, exact key fact preservation, and Freeflow raw vault recovery.
+
+## Command
+
+```sh
+npm run bench:router:commands
+```
+
+The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
+
+## Summary
+
+- Iterations per mode: 3
+- Fixtures: 8
+- Improved Freeflow run gated pass: 8/8
+- Native baseline proxy pass: 8/8
+- Improved exact fact preservation: 8/8
+- Improved recovery pass: 8/8
+- Improved weighted byte/token reduction: 85.03% / 85.02% (71893/17977 raw to 10760/2693 routed)
+- Improved average byte/token reduction: -291.55% / -288.41%
+- Improved median byte/token reduction: -451.81% / -444.90%
+- Failed command facts preserved: yes
+
+## Command Output Fixtures
+
+| fixture | mode | correctness | checks | status/code | parser | raw bytes/tokens | routed bytes/tokens | byte/token reduction | latency p50/p95 ms | recovery | outputId | notes |
+| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- | --- |
+| noisy-success | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 2439/610 | 2439/610 | 0.00%/0.00% | 0.00/0.03 | not-applicable | - |  |
+| noisy-success | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | generic 0.35 exact | 2439/610 | 1095/274 | 55.10%/55.08% | 1.60/5.45 | passed | ffout_e3bdc8fdb3c6bda34cfdb163 | Large successful command output (2439 bytes, 81 lines) was vaulted; bounded important lines were returned instead of full output (parser=generic confidence=0.35 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_e3bdc8fdb3c6bda34cfdb163. |
+| failed-stack-trace | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | - | 120/30 | 120/30 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
+| failed-stack-trace | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | generic 0.35 exact | 120/30 | 956/239 | -696.67%/-696.67% | 1.29/1.69 | passed | ffout_7070abadf7eb9e9d6671c150 | Command failed or did not complete (executionStatus=failed exitCode=1); selected failure evidence was returned and raw output was vaulted before routing (parser=generic confidence=0.35 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_7070abadf7eb9e9d6671c150. |
+| test-summary | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | - | 127/32 | 127/32 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
+| test-summary | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | test-runner 0.92 exact | 127/32 | 1110/278 | -774.02%/-768.75% | 1.27/1.91 | passed | ffout_78debcd7489ddf0ac2d17210 | Command failed or did not complete (executionStatus=failed exitCode=1); selected failure evidence was returned and raw output was vaulted before routing (parser=test-runner confidence=0.92 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_78debcd7489ddf0ac2d17210. |
+| diagnostics | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/2 | - | 169/43 | 169/43 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
+| diagnostics | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/2 | typescript-lint 0.88 exact | 169/43 | 1366/342 | -708.28%/-695.35% | 1.12/1.13 | passed | ffout_694685d2e031abd0deca0b5c | Command failed or did not complete (executionStatus=failed exitCode=2); selected failure evidence was returned and raw output was vaulted before routing (parser=typescript-lint confidence=0.88 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_694685d2e031abd0deca0b5c. |
+| git-output | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 193/49 | 193/49 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
+| git-output | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | git-status-diffstat 0.76 exact | 193/49 | 1065/267 | -451.81%/-444.90% | 1.04/1.34 | passed | ffout_b3241210e6a212f818865c01 | Large successful command output (193 bytes, 4 lines) was vaulted; bounded important lines were returned instead of full output (parser=git-status-diffstat confidence=0.76 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_b3241210e6a212f818865c01. |
+| repetitive-log | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 4365/1092 | 4365/1092 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
+| repetitive-log | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | generic 0.35 exact | 4365/1092 | 1144/286 | 73.79%/73.81% | 1.23/1.24 | passed | ffout_a68393c538a09a60bb271f44 | Large successful command output (4365 bytes, 121 lines) was vaulted; bounded important lines were returned instead of full output (parser=generic confidence=0.35 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_a68393c538a09a60bb271f44. |
+| huge-json-table | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 60090/15023 | 60090/15023 | 0.00%/0.00% | 0.00/0.01 | not-applicable | - |  |
+| huge-json-table | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | generic 0.35 lossy | 60090/15023 | 2897/725 | 95.18%/95.17% | 1.62/1.80 | passed | ffout_264e9a7aceed6ad38635bc21 | Large successful command output (60090 bytes, 1 lines) was vaulted; bounded important lines were returned instead of full output (parser=generic confidence=0.35 fidelity=lossy).; recovery=passed: Verified exact combined output recovery for ffout_264e9a7aceed6ad38635bc21. |
+| repeated-command-output | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 4390/1098 | 4390/1098 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - | Exact duplicate detection returns a compact note while current and prior raw outputs remain recoverable. |
+| repeated-command-output | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | duplicate-output 1.00 exact | 4390/1098 | 1127/282 | 74.33%/74.32% | 2.21/2.31 | passed | ffout_3142c5e358f7098c9be2225c | Exact duplicate command output matched previous outputId=ffout_857f56abd6351518d5393719 by exact output hash and command/cwd/status fingerprint (executionStatus=success exitCode=0); returning a compact duplicate note instead of re-injecting repeated output. Current raw output was vaulted as outputId=ffout_3142c5e358f7098c9be2225c.; Exact duplicate detection returns a compact note while current and prior raw outputs remain recoverable.; repeatedRuns=2; outputIds=ffout_ddd734b516d927ca72811bb3,ffout_3142c5e358f7098c9be2225c; recovery=passed: Verified exact combined output recovery for ffout_3142c5e358f7098c9be2225c. |
+
+## Skipped Optional Command Compressors
+
+- RTK: Optional command compressor comparator is skipped unless a caller supplies a configured comparator hook.
+- Squeez: Optional session/output compressor comparator is skipped unless a caller supplies a configured comparator hook.
+
+## Regression Status
+
+Improved Freeflow command routing passed all gated command-output benchmark fixtures.
+
+Failed command facts preserved: yes.

package/evals/reports/runtime/output-router-index-benchmark-1-report.md

@@ -0,0 +1,64 @@
+# Output Router Index Benchmark Report - Iteration 1
+
+Date: 2026-06-28
+
+## Scope
+
+Repo Search Backend Benchmark for `freeflow_search`. The scanner remains the product default; this benchmark compares scanner-only retrieval, the no-dependency local lexical index, a conservative hybrid scanner+index path, and records whether an FTS5/BM25/trigram candidate is available.
+
+The index cache is keyed by repo root and stores outside the repo by default. No external service, vector DB, or native dependency is required for the local lexical index.
+
+## Command
+
+```sh
+npm run bench:router:index
+```
+
+The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
+
+## Summary
+
+- Fixtures: 3
+- Scanner remains default: yes
+- Index adopted by default: no
+- Scanner pass: 3/3 (recall@3: 3/3)
+- Index warm pass: 3/3 (recall@3: 3/3)
+- FTS5/BM25/trigram pass: 3/3 (skipped: 0; recall@3: 3/3)
+- Hybrid warm pass: 3/3 (recall@3: 3/3)
+- FTS5/BM25/trigram candidate: available — Node node:sqlite is available and supports SQLite FTS5 trigram tokenization with bm25 ranking.
+- Generated false positives: 0/18
+- Cold build p50/p95: 0.56/0.60 ms
+- Warm query p50/p95: 0.04/0.05 ms
+- Stale refresh p50/p95: 0.60/0.77 ms
+- Index warm context reduction: 99.54% (310411 raw bytes to 1441 context bytes)
+
+## Results
+
+| fixture | mode | correctness | checks | path | lines | candidates | raw/context bytes | context reduction | latency p50/p95 ms | build/query ms | index mode | notes |
+| --- | --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
+| index-exact-copied-text | scanner-default | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/1201 | -751.77% | 1.84/8.82 | 0.00/0.49 | - | Deterministic repo retrieval selected 2 candidate(s); top result target.md:1-3 (matched exact normalized query phrase in section chunk). |
+| index-exact-copied-text | index-cold | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/678 | -380.85% | 1.45/2.77 | 0.56/0.05 | cold-built | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-z1mRAH/f172de04b87566a9fe7f030c.json |
+| index-exact-copied-text | index-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/678 | -380.85% | 2.36/2.50 | 0.42/0.04 | warm-loaded | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-DDeXo1/dd4502aad39189e59f4c0d1c.json |
+| index-exact-copied-text | index-stale-refresh | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 162/677 | -317.90% | 2.28/2.50 | 0.77/0.03 | stale-refreshed | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-6fTIT9/36a8bb3d60c28d09e4dfd81f.json; refresh=1 added |
+| index-exact-copied-text | fts5-bm25-trigram | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 141/204 | -44.68% | 1.07/2.11 | 0.21/0.21 | - | SQLite FTS5 trigram MATCH with bm25 rank -0.000006 |
+| index-exact-copied-text | hybrid-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/169 | -19.86% | 1.88/1.91 | 0.32/0.42 | warm-loaded | selection=scanner-index-agree; scannerQueryMs=0.39; indexQueryMs=0.03 |
+| index-generated-artifact-decoy | scanner-default | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-9 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/1394 | 99.55% | 1.82/1.86 | 0.00/0.45 | - | Deterministic repo retrieval selected 1 candidate(s); top result docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-9 (BM25-style scored section chunk with 7/7 query-token coverage). |
+| index-generated-artifact-decoy | index-cold | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/441 | 99.86% | 2.06/2.09 | 0.60/0.09 | cold-built | BM25-style indexed window chunk with 5/5 query-token coverage; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-KqjOyP/28f108bffdc59e8c6fc367d1.json |
+| index-generated-artifact-decoy | index-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/441 | 99.86% | 2.28/2.30 | 0.40/0.05 | warm-loaded | BM25-style indexed window chunk with 5/5 query-token coverage; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-3yJwOY/2c1e76270adc1a02188256db.json |
+| index-generated-artifact-decoy | index-stale-refresh | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310241/441 | 99.86% | 2.77/3.02 | 0.60/0.04 | stale-refreshed | BM25-style indexed window chunk with 5/5 query-token coverage; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-vHpgJg/9242bdc233ce6203c5a6e6fe.json; refresh=1 added |
+| index-generated-artifact-decoy | fts5-bm25-trigram | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/315 | 99.90% | 1.82/1.89 | 0.22/0.22 | - | SQLite FTS5 trigram MATCH with bm25 rank -0.000007 |
+| index-generated-artifact-decoy | hybrid-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-9 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/236 | 99.92% | 2.69/2.82 | 0.41/0.49 | warm-loaded | selection=scanner-index-agree; scannerQueryMs=0.44; indexQueryMs=0.05 |
+| index-stale-refresh | scanner-default | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/873 | -1646.00% | 0.82/0.89 | 0.00/0.30 | - | Deterministic repo retrieval selected 1 candidate(s); top result target.md:1-3 (matched exact normalized query phrase in section chunk). |
+| index-stale-refresh | index-cold | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/322 | -544.00% | 0.97/1.09 | 0.38/0.02 | cold-built | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-IgHl0y/da51d69855603b1095f746c9.json |
+| index-stale-refresh | index-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/322 | -544.00% | 1.09/1.09 | 0.23/0.02 | warm-loaded | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-QNBb5t/04e813a08b0f962846486858.json |
+| index-stale-refresh | index-stale-refresh | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/322 | -544.00% | 1.29/1.94 | 0.54/0.03 | stale-refreshed | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-DvsbMu/ecaeda95e6acb158b7eca6cf.json; refresh=1 changed |
+| index-stale-refresh | fts5-bm25-trigram | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/180 | -260.00% | 0.89/1.00 | 0.21/0.21 | - | SQLite FTS5 trigram MATCH with bm25 rank -0.000004 |
+| index-stale-refresh | hybrid-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/169 | -238.00% | 1.38/1.42 | 0.23/0.26 | warm-loaded | selection=scanner-index-agree; scannerQueryMs=0.24; indexQueryMs=0.02 |
+
+## Adoption Decision
+
+Index adopted by default: no. This slice only records benchmark evidence; scanner behavior remains the default until a later adoption checkpoint explicitly approves otherwise.
+
+## Regression Status
+
+Warm experimental index, FTS5/BM25/trigram, and hybrid modes passed all gated fixtures without generated-artifact false positives.

package/evals/reports/runtime/output-router-regression-1-report.md

@@ -0,0 +1,81 @@
+# Output Router Regression Eval Report - Iteration 1
+
+Date: 2026-06-16
+
+## Scope
+
+Added deterministic Freeflow Router regression fixtures and tests for output-routing behavior. Refactored the host-facing guidance into a real `output-router` skill, with the safety policy stored as a skill reference.
+
+Fixtures live under:
+
+- `evals/fixtures/output-router/`
+
+Regression tests live in:
+
+- `router/tests/regression-fixtures.test.js`
+
+Runtime-facing skill files live in:
+
+- `skills/output-router/SKILL.md`
+- `skills/output-router/references/safety-policy.md`
+
+## Failure Preserved
+
+The router must not merely reduce output size. It must also preserve exact evidence and recovery paths.
+
+The preserved failure modes are:
+
+- broad repo queries dumping whole files instead of bounded evidence,
+- `preserve: full` becoming a lossy summary over cap,
+- noisy command output flooding context without raw recovery,
+- failed-command output losing exact assertion/stack evidence,
+- verification output paraphrasing completion evidence,
+- native post-tool safety-net output being shortened without a visible Freeflow label and exact recovery path,
+- ambiguous top-level `status` hiding the difference between tool success, command failure, and routing success.
+
+## Deterministic Check
+
+Added fixture-backed tests for:
+
+- large docs targeted query,
+- whole-artifact full-fidelity over-cap chunk behavior,
+- noisy successful command reduction and raw vault recovery,
+- failed command exact evidence and split statuses,
+- verification summary exactness,
+- Pi native safety-net labeling and exact raw tail recovery.
+
+The tests assert token/byte reduction where expected, exact evidence preservation, vault recovery, and split status semantics.
+
+## Result
+
+`npm run test:router`: pass, 42 tests.
+
+Actual Pi smoke also passed for both local-extension loading and installed package discovery using an isolated temporary `PI_CODING_AGENT_DIR` cache:
+
+- `freeflow_retrieve` registration and execution through Pi.
+- `freeflow_run` execution through Pi's command runner with configured routing thresholds.
+- Native `read` post-tool safety-net routing with explicit `postToolRouting: safety-net`.
+
+## Verification
+
+Commands:
+
+```sh
+npm run test:router
+pi --no-session --no-extensions --no-skills --no-context-files --approve -e "$PWD/pi-extension/index.js" --tools freeflow_retrieve --mode json -p "Use the freeflow_retrieve tool exactly once with action=query, source.kind=repo, source.path='evals/fixtures/output-router/large-router-manual.md', query='OUTPUT_ROUTER_SKILL_DECISION_ANCHOR safety net', preserve=important. Then report the output path and whether the anchor appears."
+PI_CODING_AGENT_DIR=/tmp/freeflow-pi-agent-cache-... pi install "$PWD" --approve
+PI_CODING_AGENT_DIR=/tmp/freeflow-pi-agent-cache-... pi --no-session --no-context-files --approve --tools freeflow_retrieve --mode json -p "Use freeflow_retrieve..."
+# plus temp-project Pi smoke for freeflow_run and native read safety-net routing through package discovery
+```
+
+Supporting checks from the same implementation pass:
+
+```sh
+node --check pi-extension/index.js
+npm pack --dry-run --json
+rg -n "child_process|node:child_process|\bspawn\b|\bexecFile\b|\bexecSync\b" router pi-extension || true
+```
+
+## Recommendation
+
+Keep these deterministic regression fixtures in the router test suite. Add model-based agent evals only after the deterministic runtime behavior is stable and a specific agent-selection failure is observed.

package/evals/reports/runtime/output-router-transform-eval-1-report.md

@@ -0,0 +1,28 @@
+# Output Router Transform Eval - Iteration 1
+
+Date: 2026-06-28
+
+## Scope
+
+Targeted deterministic eval for transformed-output routing. It compares direct/manual long-log inspection against explicit Freeflow transform operations.
+
+## Command
+
+```sh
+npm run build && node evals/scripts/run-output-router-transform-eval.js
+```
+
+## Summary
+
+- Fixtures: 1
+- Objective gates passed: 4/4
+
+## Results
+
+| fixture | direct/raw baseline | Freeflow routed behavior | status | gates |
+| --- | --- | --- | --- | --- |
+| long-log-manual-inspection-vs-freeflow-transform | 7729 raw log bytes for manual inspection | 193 filtered evidence bytes; countMatches=2; exact transformed recovery | pass | manualBaselineWouldReadWholeLog ✓; filteredTargetFacts ✓; countMatches ✓; lineage ✓ |
+
+## Result
+
+All targeted transform gates passed for these deterministic fixtures.

package/evals/reports/runtime/pi-observed-routing-eval-1-report.md

@@ -0,0 +1,44 @@
+# Pi Observed Routing Eval - Iteration 1
+
+Date: 2026-06-24
+
+## Scope
+
+Targeted deterministic eval for the Pi-only observed output routing slice. It exercises Pi's `tool_result` path for configured MCP, web, fetch, and code-search producers after direct host execution.
+
+This eval compares raw direct tool-output fixtures against the routed result returned to the agent. It verifies bounded output, exact critical fact preservation, persistence/recovery claims, metadata-only no-raw-recovery behavior, mutating MCP routing as metadata (not a gate), and Pi status capability reporting.
+
+It does not claim Context Mode or cross-host superiority. Claude and Codex observed-routing adapters remain out of scope.
+
+## Command
+
+```sh
+npm run build && node evals/scripts/run-pi-observed-routing-eval.js
+```
+
+## Summary
+
+- Fixtures: 7
+- Objective gates passed: 28/28
+- Host: Pi only.
+- Direct execution/permissions: owned by Pi; Freeflow routes only completed tool output.
+- Persistence modes covered: exact and metadata-only.
+- Overall byte reduction, excluding status-only fixtures: 82.2%.
+
+## Results
+
+Raw and routed columns are `bytes / lines / items-or-evidence-packets`. Reduction is calculated from raw bytes to routed-result bytes.
+
+| fixture | producer | raw direct output | routed output | byte reduction | recoverability | status | gates |
+| --- | --- | ---: | ---: | ---: | --- | --- | --- |
+| mcp-github-search-exact | mcp:github:search_issues | 8589 / 36 / 4 | 1201 / 40 / 1 | 86.0% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
+| mcp-github-create-mutating | mcp:github:create_issue | 2492 / 8 / 1 | 889 / 29 / 1 | 64.3% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
+| mcp-gmail-search-metadata-only | mcp:gmail:search | 12097 / 4 / 1 | 832 / 13 / 1 | 93.1% | metadata_only | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
+| pi-web-search-exact | web:web_search | 1752 / 29 / 4 | 1184 / 34 / 1 | 32.4% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
+| pi-fetch-content-exact | fetch:fetch_content | 2631 / 6 / 1 | 853 / 25 / 1 | 67.6% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
+| pi-code-search-exact | code_search:code_search | 8712 / 26 / 3 | 1490 / 36 / 1 | 82.9% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
+| pi-capability-status | status | 0 / 0 / 0 | 4970 / 190 / 0 | n/a | n/a | pass | hostCapabilityReported ✓; persistenceModesReported ✓; unsupportedRedactedReported ✓; configuredProducerReported ✓ |
+
+## Result
+
+All targeted Pi observed-routing gates passed for these deterministic fixtures. Pi observed routing materially reduced large/noisy fixture output while preserving critical facts and truthful recovery behavior.