@hassangameryt/freeflow 0.3.0 → 0.5.0

package/evals/reports/runtime/output-router-benchmark-1-report.md

@@ -1,81 +0,0 @@
-# Output Router Benchmark Report - Iteration 1
-
-Date: 2026-06-19
-
-## Scope
-
-Deterministic, CI-friendly tool benchmark for Freeflow Router retrieval behavior. The runner compares a native text-search proxy, a pre-hardening Freeflow-style proxy, and the improved Freeflow Router implementation. Optional external comparators are recorded as skipped rather than failed.
-
-Reduction percentages compare routed/context bytes and approximate tokens against the raw source or direct output for that mode. Negative reduction means structured routing overhead is larger than the tiny raw output; native exact-search proxies can still be smaller than the router for simple lookups.
-
-## Baseline Caveat
-
-The pre-hardening Freeflow mode is a deterministic proxy for the old line-scoring behavior, not a checkout of an older runtime. It is useful for stable regression pressure, not historical performance archaeology.
-
-## Command
-
-```sh
-npm run bench:router
-```
-
-The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
-
-## Summary
-
-- Iterations per mode: 3
-- Fixtures: 7
-- Improved Freeflow Router gated pass: 7/7
-- Native baseline proxy pass: 3/7
-- Pre-hardening Freeflow proxy pass: 2/7
-- Generated false positives observed: 4/21 mode results
-- Improved generated false positives: 0/7
-- Improved weighted byte/token reduction: 98.54% / 98.54% (511618/127907 raw to 7473/1870 routed)
-- Improved average byte/token reduction: -235.00% / -231.39%
-- Improved median byte/token reduction: -105.94% / -105.77%
-- Improved path/span/excerpt checks: 7/7 path, 7/7 span, 7/7 excerpt
-- Sandbox failure fixed: yes
-
-## Results
-
-| fixture | mode | correctness | checks | path | lines | raw bytes/tokens | routed bytes/tokens | byte/token reduction | latency p50/p95 ms | recovery | notes |
-| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
-| exact-copied-text-block | native-baseline-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 138/35 | 78/20 | 43.48%/42.86% | 0.28/5.15 | not-applicable | score=28.00 |
-| exact-copied-text-block | pre-hardening-freeflow-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 138/35 | 78/20 | 43.48%/42.86% | 0.24/0.26 | failed | score=28.00 |
-| exact-copied-text-block | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 138/35 | 903/226 | -554.35%/-545.71% | 0.80/2.55 | passed | Deterministic repo retrieval selected 1 candidate(s); top result target.md:1-3 (matched exact normalized query phrase in section chunk).; recovery=passed: Verified repo retrieve target.md:1-3. |
-| markdown-heading-nearby-body | native-baseline-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 169/43 | 44/11 | 73.96%/74.42% | 0.18/0.22 | not-applicable | score=20.00 |
-| markdown-heading-nearby-body | pre-hardening-freeflow-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 169/43 | 44/11 | 73.96%/74.42% | 0.17/0.27 | failed | score=20.00 |
-| markdown-heading-nearby-body | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 3-6 | 169/43 | 954/239 | -464.50%/-455.81% | 0.67/0.68 | passed | Deterministic repo retrieval selected 1 candidate(s); top result target.md:3-6 (BM25-style scored section chunk with 6/6 query-token coverage).; recovery=passed: Verified repo retrieve target.md:3-6. |
-| generated-artifact-decoy | native-baseline-proxy | fail | path ✗ span ✓ excerpt ✓ gen-fp ✗ | graphify-out/graph.html | 1-3 | 310220/77555 | 310058/77515 | 0.05%/0.05% | 1.42/1.57 | not-applicable | score=140000.00 |
-| generated-artifact-decoy | pre-hardening-freeflow-proxy | fail | path ✗ span ✓ excerpt ✓ gen-fp ✗ | graphify-out/graph.html | 1-3 | 310220/77555 | 310058/77515 | 0.05%/0.05% | 1.37/1.43 | failed | score=140000.00 |
-| generated-artifact-decoy | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-9 | 310220/77555 | 1396/349 | 99.55%/99.55% | 1.09/1.24 | passed | Deterministic repo retrieval selected 1 candidate(s); top result docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-9 (BM25-style scored section chunk with 7/7 query-token coverage).; recovery=passed: Verified repo retrieve docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-9. |
-| huge-single-line-decoy | native-baseline-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✗ | debug.log | 1-1 | 192091/48023 | 192013/48004 | 0.04%/0.04% | 0.41/0.46 | not-applicable | score=24000.00 |
-| huge-single-line-decoy | pre-hardening-freeflow-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✗ | debug.log | 1-1 | 192091/48023 | 192013/48004 | 0.04%/0.04% | 0.43/0.57 | failed | score=24000.00 |
-| huge-single-line-decoy | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | docs/router.md | 1-3 | 192091/48023 | 946/237 | 99.51%/99.51% | 0.73/0.81 | passed | Deterministic repo retrieval selected 1 candidate(s); top result docs/router.md:1-3 (BM25-style scored section chunk with 6/6 query-token coverage).; recovery=passed: Verified repo retrieve docs/router.md:1-3. |
-| ambiguous-multi-file-query | native-baseline-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 8262/2066 | 8100/2025 | 1.96%/1.98% | 0.26/0.28 | not-applicable | score=3600.00 |
-| ambiguous-multi-file-query | pre-hardening-freeflow-proxy | fail | path ✗ span ✗ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 8262/2066 | 8100/2025 | 1.96%/1.98% | 0.26/0.30 | failed | score=3600.00 |
-| ambiguous-multi-file-query | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-3 | 8262/2066 | 947/237 | 88.54%/88.53% | 1.03/1.05 | passed | Deterministic repo retrieval selected 1 candidate(s); top result target.md:1-3 (BM25-style scored section chunk with 6/6 query-token coverage).; recovery=passed: Verified repo retrieve target.md:1-3. |
-| vaulted-output-query | native-baseline-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | ffout_91e882b51880dd5b9bbbc3ac:stderr | 1-3 | 115/29 | 115/29 | 0.00%/0.00% | 0.01/0.07 | not-applicable | score=12.00 |
-| vaulted-output-query | pre-hardening-freeflow-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | ffout_91e882b51880dd5b9bbbc3ac:stderr | 1-3 | 115/29 | 115/29 | 0.00%/0.00% | 0.00/0.01 | failed | score=12.00 |
-| vaulted-output-query | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | ffout_91e882b51880dd5b9bbbc3ac:stderr | 1-3 | 115/29 | 1044/261 | -807.83%/-800.00% | 0.39/0.64 | passed | Deterministic lexical retrieval selected vaulted outputId=ffout_91e882b51880dd5b9bbbc3ac stream=stderr lines=1-3.; recovery=passed: Verified vault retrieve ffout_91e882b51880dd5b9bbbc3ac:stderr:1-3. |
-| expand-narrow-evidence | native-baseline-proxy | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-24 | 623/156 | 623/156 | 0.00%/0.00% | 0.06/0.08 | not-applicable |  |
-| expand-narrow-evidence | pre-hardening-freeflow-proxy | fail | path ✗ span ✓ excerpt ✗ gen-fp ✓ | decoy.md | 1-1 | 3824/956 | 3200/800 | 16.32%/16.32% | 0.17/0.18 | failed | score=800.00 |
-| expand-narrow-evidence | improved-freeflow-router | pass | path ✓ span ✓ excerpt ✓ gen-fp ✓ | target.md | 1-24 | 623/156 | 1283/321 | -105.94%/-105.77% | 0.87/0.93 | passed | Expanded repo evidence ev_0ba9afec918a3334 to target.md:1-24.; recovery=passed: Verified repo retrieve target.md:1-24. |
-
-## Not Yet Measured
-
-- recall@3 / alternate candidates: not measured in this first deterministic runner.
-- explanation quality: not measured beyond route reason capture.
-- command-output parser benchmarks: deferred to the command benchmark track.
-
-## Skipped External Comparators
-
-- Graphify: Optional external comparator is not required for CI-friendly router benchmarks.
-- Claude Context: Optional semantic/hybrid search comparator is skipped unless configured separately.
-- RTK: Command-output comparator belongs to the later command benchmark track.
-- Squeez: Session-efficiency comparator belongs to the later command/session benchmark track.
-
-## Regression Status
-
-Improved Freeflow Router passed all gated benchmark fixtures.
-
-The generated-artifact decoy benchmark preserves the original Sandbox Permissions false-positive shape and records it as fixed when the improved router selects the docs target instead of `graphify-out/graph.html`.

package/evals/reports/runtime/output-router-capture-derive-provider-eval-1-report.md

@@ -1,35 +0,0 @@
-# Output Router Capture/Derive/Provider Eval - Iteration 1
-
-Date: 2026-06-23
-
-## Scope
-
-Targeted deterministic eval for Slice 9 of the universal output capture/derive work. It compares direct/raw evidence shapes against explicit Freeflow routing for read-only provider capture, web-shaped capture, deterministic derive, and provider-summary tool choice.
-
-This eval does not claim broad superiority. It verifies bounded evidence, exact recovery where promised, lineage, category-scoped provider summaries, and non-injection of invalid custom manifest text for these fixtures.
-
-## Command
-
-```sh
-npm run build && node evals/scripts/run-output-router-capture-derive-eval.js
-```
-
-## Summary
-
-- Fixtures: 4
-- Objective gates passed: 14/14
-- Direct host-tool capture: not evaluated and remains off by default.
-- Mutating provider tools: not mediated by `freeflow_capture`.
-
-## Results
-
-| fixture | direct/raw baseline | Freeflow routed behavior | status | gates |
-| --- | --- | --- | --- | --- |
-| direct-readonly-provider-vs-freeflow-capture | 1268 raw bytes would enter context from direct provider output | 233 evidence bytes entered context; raw recovery=exact | pass | readOnlyCaptured ✓; boundedEvidence ✓; exactRecovery ✓ |
-| web-shaped-capture-and-recovery | 1682 raw bytes from web-shaped output | 301 evidence bytes plus exact recovery | pass | webProducerCaptured ✓; boundedEvidence ✓; exactRecovery ✓ |
-| long-log-manual-inspection-vs-freeflow-derive | 7729 raw log bytes for manual inspection | 176 filtered evidence bytes; countMatches=2; exact derived recovery | pass | manualBaselineWouldReadWholeLog ✓; filteredTargetFacts ✓; countMatches ✓; lineage ✓ |
-| provider-summary-tool-choice-accuracy | Raw provider docs/manuals are not injected | 458 summary bytes; diagnostics category only; invalid custom guidance omitted | pass | compactSummary ✓; categoryScoped ✓; invalidCustomNotInjected ✓; mutationBoundaryPresent ✓ |
-
-## Result
-
-All targeted Slice 9 capture/derive/provider-summary gates passed for these deterministic fixtures.

package/evals/reports/runtime/output-router-codex-large-index-sqlite-adhoc-report.md

@@ -1,150 +0,0 @@
-# Output Router Codex Large Repo Index/SQLite Ad Hoc Benchmark
-
-Date: 2026-06-18
-
-## Scope
-
-Ad hoc large-repo benchmark requested during Slice 8 follow-up. This is not an adoption decision and did not change product defaults.
-
-Repository:
-
-- URL: `https://github.com/openai/codex`
-- Local path: `/tmp/freeflow-codex-large-benchmark/codex`
-- Commit: `e12dd73`
-- Git-tracked files: 5,118
-- Working tree size after shallow clone: ~70 MB
-
-Modes measured:
-
-1. Existing `freeflow_retrieve` scanner default.
-2. Slice 8 no-dependency experimental local index.
-3. Improved no-dependency prototype with slimmer symbol/section index.
-4. Python stdlib SQLite FTS5 ad hoc comparator.
-5. SQLite FTS5 with deterministic reranking and tuned candidate limit.
-6. Scanner-hardening changes derived from this benchmark.
-
-SQLite/FTS used only Python standard-library `sqlite3` with FTS5 available. No npm/package dependency was added.
-
-## Baseline Result
-
-| mode | build/load | query p50 | query p95 | strict path pass | avg context bytes | notes |
-| --- | ---: | ---: | ---: | ---: | ---: | --- |
-| scanner default | none | 5,663 ms | 8,929 ms | 3/8 | 1,230 | live scan each query |
-| Slice 8 no-dep local index | failed after 10,013 ms | n/a | n/a | n/a | n/a | `RangeError: Invalid string length` while serializing cache |
-| SQLite FTS5 AND | 3,934 ms | 0.18 ms | 2.24 ms | 2/8 | 135 | very fast but low recall from strict AND query construction |
-| SQLite FTS5 OR + camel split | 8,145 ms | 42.97 ms | 50.94 ms | 3/8 | 322 | same strict pass count as scanner, much faster query-only |
-
-## Improvement Attempts
-
-Changes tried in temp prototypes:
-
-- no-dep index: remove duplicated line-window text; use symbol/Markdown chunks plus coarse blocks; store token counts and metadata only.
-- no-dep ranking: camel/Pascal split, exact compound boosts, source/test path priors, symbol-definition boosts, path-intent boosts, phrase-sequence boosts, candidate prefiltering.
-- SQLite: OR query for recall, ordered FTS candidate set, deterministic reranking with the same scoring features, candidate-limit tuning.
-
-Best observed variants:
-
-| mode | build/load | query p50 | query p95 | strict path pass | avg context bytes | cache/size | notes |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
-| no-dep V1 slim | 42,496 ms | 23,814 ms | 42,197 ms | 5/8 | 1,397 | 169.5 MB JSON estimate | accuracy improved, speed unacceptable |
-| no-dep V2 candidate prefilter | 73,755 ms | 1,973 ms | 2,933 ms | 5/8 | 1,542 | 156.4 MB JSON estimate | query faster, build/cache unacceptable |
-| no-dep V3 symbol/section only | 41,048 ms | 1,484 ms | 2,720 ms | 5/8 | 1,413 | 97.5 MB JSON estimate | best no-dep profile, still heavy |
-| SQLite FTS rerank limit=1000 | 1,936 ms | 137 ms | 287 ms | 5/8 | 1,128 | in-memory FTS | best overall prototype |
-
-## Scanner Hardening Applied
-
-Changes applied to the default scanner from this evidence:
-
-- skip broad traversal of `generated/` directories while preserving explicit generated-path retrieval,
-- add code symbol chunks for enum/struct/function/impl-style matches,
-- split CamelCase/PascalCase/snake/path identifiers for scoring,
-- preserve stopwords for exact phrase matching while keeping scoring stopword-filtered,
-- add bounded symbol chunk evidence instead of narrow ±2-line snippets,
-- add source/test path priors,
-- add definition, path-intent, ordered-phrase, and complete-query-coverage boosts,
-- use structural chunks first and line-window chunks only as a fallback for unstructured files.
-
-Follow-up speed pass kept the same scanner result while reducing overhead:
-
-- build BM25 stats only for chunks that pass the cheap `chunkMightMatch` prefilter,
-- memoize token and phrase-sequence normalization per chunk,
-- reuse tokenized chunk content for ordered-phrase scoring,
-- read eligible repo text files with bounded concurrency while preserving deterministic traversal order,
-- skip expensive identifier-splitting regexes for plain tokens without separators or case boundaries.
-
-Measured scanner-hardening result on the same Codex repo/query set:
-
-| mode | query p50 | query p95 | strict path+excerpt pass | avg context bytes | notes |
-| --- | ---: | ---: | ---: | ---: | --- |
-| scanner before hardening | 5,663 ms | 8,929 ms | 3/8 | 1,230 | baseline live scan |
-| scanner after hardening | 8,776 ms | 12,668 ms | 5/8 | 2,178 | more accurate but slower and larger context |
-| scanner after speed pass | 5,112 ms | 8,269 ms | 5/8 | 2,307 | accuracy kept; latency below baseline scanner p50/p95 |
-| scanner after review-fix pass | 5,473 ms | 8,611 ms | 6/8 | 2,082 | restored regressed source-definition queries and kept MCP test-query improvement |
-| scanner after deep-module cleanup rerun | 4,305 ms | 6,773 ms | 6/8 | 2,082 | final verification after `RepoTraversalPolicy`, exact line ranges, `EvidenceRangeSelector`, vault locking, command benchmark observation changes, and the post-review exact-phrase fixes |
-| scanner after architecture-deepening report refresh | 4,272 ms | 6,679 ms | 6/8 | 2,082 | report-refresh verification after bounded evidence, BenchmarkHarness, EvidenceSearch, RouterContract, and experimental-index capsule slices |
-
-Interpretation: scanner hardening produced a real accuracy gain (**+2/8 strict fixtures**) but initially regressed latency/context. The speed pass recovered the latency regression and kept the accuracy gain, although context remained larger because symbol chunks return fuller evidence. The later review-fix pass improved strict accuracy to **6/8** by constraining symbol complete-coverage boosts, preserving exact-phrase evidence ranges, adding Markdown preamble chunks, and tightening generated/media-path filtering while keeping scanner latency near the speed-pass result.
-
-## Before/After Summary
-
-Against scanner default:
-
-- Scanner hardening plus the review-fix pass improved accuracy from **3/8** to **6/8**; the latest report-refresh rerun kept **6/8** and kept query p50 below the original scanner baseline (**5,663 ms → 4,272 ms**).
-- Accuracy improved from **3/8** strict expected-path matches to **5/8** for no-dep V3, SQLite FTS rerank, and scanner hardening/speed pass, then to **6/8** for the default scanner after review fixes.
-- SQLite FTS rerank query p50 improved from **5,663 ms** to **137 ms**: about **41× faster query-only**.
-- SQLite FTS rerank query p95 improved from **8,929 ms** to **287 ms**: about **31× faster query-only**.
-- Scanner total for eight baseline queries was about **47.6 s**; initial scanner-hardening total was about **78.1 s**; after the speed pass it was about **45.5 s**; after review fixes it was about **53.5 s** in the observed run; the latest report-refresh rerun was about **38.6 s** for the scanner rows.
-- SQLite FTS rerank total was about **3.3 s including build**: about **14× faster end-to-end for this query batch** versus baseline scanner.
-- SQLite FTS rerank context was slightly smaller than baseline scanner: **1,230 avg bytes → 1,128 avg bytes** (~8% lower). Scanner hardening increased context to **2,178-2,307 avg bytes** because symbol chunks return fuller evidence.
-- No-dep V3 warm queries were faster than baseline scanner, but cold build/cache costs made it unattractive on this repo.
-
-## Misses Remaining
-
-The best strict result is now **6/8**. The repeated misses indicate ranking/query-intent problems, not just speed problems:
-
-- Some broad queries legitimately have multiple plausible targets.
-- Test/source intent is hard to infer from query text alone.
-- Path-intent queries like `apply_patch prompt codex prompts lib` remain sensitive: current scanner evidence includes the expected excerpt, but strict path grading still fails because it selects the deeper implementation handler instead of the expected prompt source.
-- FTS candidate generation and deterministic reranking need a more principled query classifier before automatic routing is safe.
-
-## Interpretation
-
-- The current no-dependency local index is **not ready for large repos**. It needs a more compact cache format and fewer or more targeted chunks before it is worth productizing.
-- SQLite/FTS is **promising enough to keep as an optional benchmark/comparator**. It produced better strict accuracy than scanner and much better speed in this ad hoc run.
-- SQLite/FTS is **not ready to become product behavior**. Accuracy is still mid, and adding SQLite/FTS has packaging/portability implications that need an explicit dependency decision.
-- A future `hybrid:auto` could be valuable only after a proper benchmark shows reliable query classification, fallback behavior, and no accuracy regression.
-
-## Decision
-
-- Keep scanner as default for now, with the applied hardening guarded by tests.
-- Keep no-dependency local index experimental; redesign its large-repo chunk/cache strategy before further adoption discussion.
-- Do not add SQLite/FTS as a product dependency yet.
-- Keep SQLite/FTS as an optional large-repo benchmark comparator candidate.
-- Do not implement `hybrid:auto` yet; the evidence supports future research, not default behavior.
-- Before further accuracy hardening, keep the speed-pass benchmark in the loop so ranking gains do not silently become latency regressions.
-
-## Raw Evidence Pointers
-
-Temporary files from this ad hoc run:
-
-- `/tmp/freeflow-codex-large-benchmark/scanner-sqlite-results.json`
-- `/tmp/freeflow-codex-large-benchmark/sqlite-results.json`
-- `/tmp/freeflow-codex-large-benchmark/sqlite-or-results.json`
-- `/tmp/freeflow-codex-large-benchmark/improved-index-results.json`
-- `/tmp/freeflow-codex-large-benchmark/improved-index-v2-results.json`
-- `/tmp/freeflow-codex-large-benchmark/improved-index-v3-results.json`
-- `/tmp/freeflow-codex-large-benchmark/results-symbols-only.json`
-- `/tmp/freeflow-codex-large-benchmark/scanner-sqlite-final-green-output.json`
-- `/tmp/freeflow-codex-large-benchmark/scanner-sqlite-speed-pass-output.json`
-- Post-review-fix rerun command: `node /tmp/freeflow-codex-large-benchmark/bench-scanner-sqlite.mjs` on 2026-06-18/19 session; observed scanner summary `pass=6/8`, `p50=5,473 ms`, `p95=8,611 ms`, `avgContextBytes=2,082`.
-- Deep-module cleanup final verification command: `node /tmp/freeflow-codex-large-benchmark/bench-scanner-sqlite.mjs`; observed scanner summary `pass=6/8`, `p50=4,305 ms`, `p95=6,773 ms`, `avgContextBytes=2,082`.
-- Architecture-deepening report-refresh command: `node /tmp/freeflow-codex-large-benchmark/bench-scanner-sqlite.mjs`; observed scanner summary `pass=6/8`, `p50=4,272 ms`, `p95=6,679 ms`, `avgContextBytes=2,082`, scanner row total about `38.6 s`.
-- Latest report-refresh scanner rows:
-  - `sandbox-permissions-enum`: pass, `codex-rs/protocol/src/models.rs`, lines `47-66`, 1,759 context bytes.
-  - `mcp-tool-call-event`: pass, `codex-rs/analytics/src/analytics_client_tests.rs`, lines `4207-4211`, 1,409 context bytes.
-  - `approval-mode-cli-arg`: pass, `codex-rs/utils/cli/src/approval_mode_cli_arg.rs`, lines `9-28`, 1,845 context bytes.
-  - `turn-diff-tracker`: fail strict path, selected `codex-rs/core/src/tools/handlers/unified_exec_tests.rs`, lines `3-35`, excerpt complete, 2,350 context bytes.
-  - `model-provider-info`: pass, `codex-rs/ollama/src/lib.rs`, lines `63-79`, 1,467 context bytes.
-  - `network-proxy-spec`: pass, `codex-rs/core/src/config/network_proxy_spec.rs`, lines `24-31`, 1,296 context bytes.
-  - `codex-http-client`: pass, `codex-rs/codex-client/src/default_client.rs`, lines `21-50`, 1,794 context bytes.
-  - `apply-patch-prompt`: fail strict path, selected `codex-rs/core/src/tools/handlers/apply_patch.rs`, lines `330-409`, excerpt complete, 4,737 context bytes.

package/evals/reports/runtime/output-router-codex-qa-benchmark-1-report.md

@@ -1,46 +0,0 @@
-# Codex Structured Q&A Macro Benchmark Report - Iteration 1
-
-Date: 2026-06-19
-
-## Scope
-
-Fast, deterministic Stage 1 macro benchmark for the Codex CLI agent-harness research corpus. The benchmark asks one structured Q&A question derived from the Sandbox Permissions pass and grades retrieval-backed answer quality, citation quality, evidence quality, context size, and latency.
-
-The fixture uses existing Freeflow research docs as oracle scaffolding and includes a generated `graphify-out` decoy to preserve the original broad-retrieval failure shape. Upstream Codex source citation comparison is marked skipped until a source snapshot is explicitly supplied.
-
-## Command
-
-```sh
-npm run bench:router:codex-qa
-```
-
-The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
-
-## Summary
-
-- Iterations per mode: 1
-- Fixtures: 1
-- Improved Freeflow Router gated pass: 1/1
-- Native broad-search proxy pass: 0/1
-- Sandbox failure fixed: yes
-- Generated false positives observed: 1/2 mode results
-- Improved answer/citation/evidence: 1/1 answer, 1/1 citation, 1/1 evidence
-- Improved weighted byte/token reduction: 99.50% / 99.50% (580499/145125 raw to 2892/723 context)
-
-## Results
-
-| fixture | mode | correctness | checks | path | lines | raw bytes/tokens | context bytes/tokens | byte/token reduction | latency p50/p95 ms | proxy calls | answer | notes |
-| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | ---: | --- | --- |
-|sandbox-permissions-structured-qa | native-broad-search-proxy | fail | answer ✗ citation ✗ evidence ✗ gen-fp ✗ | graphify-out/graph.html | 1-3 | 580499/145125 | 580058/145015 | 0.08%/0.08% | 11.67/11.67 | 1 | Sandbox Permissions evidence from graphify-out/graph.html:1-3 did not include enough plain-language definitions to answer. | lexical score=40000 |
-|sandbox-permissions-structured-qa | improved-freeflow-router | pass | answer ✓ citation ✓ evidence ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 1-28 | 580499/145125 | 2892/723 | 99.50%/99.50% | 3.66/3.66 | 2 | Sandbox Permissions are documented at docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:1-28. UseDefault: run with the turn's normal sandbox. RequireEscalated: request unsandboxed execution. WithAdditionalPermissions: stay sandboxed but widen permissions for this one command. | Deterministic repo retrieval selected 1 candidate(s); top result docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-17 (BM25-style scored section chunk with 14/14 query-token coverage).; Expanded repo evidence ev_178fc507730e20fc to docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:1-28. |
-
-## Fixture Questions
-
-- sandbox-permissions-structured-qa: Find the Sandbox Permissions section, report file/lines, and explain UseDefault, RequireEscalated, and WithAdditionalPermissions.
-  - expected doc: `docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md`
-  - upstream source citation comparison: skipped-unavailable
-
-## Skipped External Comparators
-
-- Graphify: Skipped unless a fresh graph for the benchmark fixture is explicitly supplied.
-- Claude Context: Skipped unless an index is configured and fresh for the benchmark fixture.

package/evals/reports/runtime/output-router-command-benchmark-1-report.md

@@ -1,62 +0,0 @@
-# Output Router Command Benchmark Report - Iteration 1
-
-Date: 2026-06-19
-
-## Scope
-
-Deterministic, CI-friendly command-output benchmark for `freeflow_run`. The runner compares native direct output against the improved Freeflow command router and records optional RTK/Squeez comparators as skipped unless a caller supplies configured comparator hooks.
-
-Reduction percentages compare routed/context bytes and approximate tokens against raw command output bytes for each fixture. Correctness is gated on execution status, exit code, exact key fact preservation, and Freeflow raw vault recovery.
-
-## Command
-
-```sh
-npm run bench:router:commands
-```
-
-The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
-
-## Summary
-
-- Iterations per mode: 3
-- Fixtures: 8
-- Improved Freeflow run gated pass: 8/8
-- Native baseline proxy pass: 8/8
-- Improved exact fact preservation: 8/8
-- Improved recovery pass: 8/8
-- Improved weighted byte/token reduction: 85.03% / 85.02% (71893/17977 raw to 10760/2693 routed)
-- Improved average byte/token reduction: -291.55% / -288.41%
-- Improved median byte/token reduction: -451.81% / -444.90%
-- Failed command facts preserved: yes
-
-## Command Output Fixtures
-
-| fixture | mode | correctness | checks | status/code | parser | raw bytes/tokens | routed bytes/tokens | byte/token reduction | latency p50/p95 ms | recovery | outputId | notes |
-| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- | --- |
-| noisy-success | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 2439/610 | 2439/610 | 0.00%/0.00% | 0.00/0.03 | not-applicable | - |  |
-| noisy-success | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | generic 0.35 exact | 2439/610 | 1095/274 | 55.10%/55.08% | 1.60/5.45 | passed | ffout_e3bdc8fdb3c6bda34cfdb163 | Large successful command output (2439 bytes, 81 lines) was vaulted; bounded important lines were returned instead of full output (parser=generic confidence=0.35 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_e3bdc8fdb3c6bda34cfdb163. |
-| failed-stack-trace | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | - | 120/30 | 120/30 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
-| failed-stack-trace | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | generic 0.35 exact | 120/30 | 956/239 | -696.67%/-696.67% | 1.29/1.69 | passed | ffout_7070abadf7eb9e9d6671c150 | Command failed or did not complete (executionStatus=failed exitCode=1); selected failure evidence was returned and raw output was vaulted before routing (parser=generic confidence=0.35 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_7070abadf7eb9e9d6671c150. |
-| test-summary | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | - | 127/32 | 127/32 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
-| test-summary | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/1 | test-runner 0.92 exact | 127/32 | 1110/278 | -774.02%/-768.75% | 1.27/1.91 | passed | ffout_78debcd7489ddf0ac2d17210 | Command failed or did not complete (executionStatus=failed exitCode=1); selected failure evidence was returned and raw output was vaulted before routing (parser=test-runner confidence=0.92 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_78debcd7489ddf0ac2d17210. |
-| diagnostics | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/2 | - | 169/43 | 169/43 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
-| diagnostics | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | failed/2 | typescript-lint 0.88 exact | 169/43 | 1366/342 | -708.28%/-695.35% | 1.12/1.13 | passed | ffout_694685d2e031abd0deca0b5c | Command failed or did not complete (executionStatus=failed exitCode=2); selected failure evidence was returned and raw output was vaulted before routing (parser=typescript-lint confidence=0.88 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_694685d2e031abd0deca0b5c. |
-| git-output | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 193/49 | 193/49 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
-| git-output | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | git-status-diffstat 0.76 exact | 193/49 | 1065/267 | -451.81%/-444.90% | 1.04/1.34 | passed | ffout_b3241210e6a212f818865c01 | Large successful command output (193 bytes, 4 lines) was vaulted; bounded important lines were returned instead of full output (parser=git-status-diffstat confidence=0.76 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_b3241210e6a212f818865c01. |
-| repetitive-log | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 4365/1092 | 4365/1092 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - |  |
-| repetitive-log | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | generic 0.35 exact | 4365/1092 | 1144/286 | 73.79%/73.81% | 1.23/1.24 | passed | ffout_a68393c538a09a60bb271f44 | Large successful command output (4365 bytes, 121 lines) was vaulted; bounded important lines were returned instead of full output (parser=generic confidence=0.35 fidelity=exact).; recovery=passed: Verified exact combined output recovery for ffout_a68393c538a09a60bb271f44. |
-| huge-json-table | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 60090/15023 | 60090/15023 | 0.00%/0.00% | 0.00/0.01 | not-applicable | - |  |
-| huge-json-table | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | generic 0.35 lossy | 60090/15023 | 2897/725 | 95.18%/95.17% | 1.62/1.80 | passed | ffout_264e9a7aceed6ad38635bc21 | Large successful command output (60090 bytes, 1 lines) was vaulted; bounded important lines were returned instead of full output (parser=generic confidence=0.35 fidelity=lossy).; recovery=passed: Verified exact combined output recovery for ffout_264e9a7aceed6ad38635bc21. |
-| repeated-command-output | native-baseline-proxy | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | - | 4390/1098 | 4390/1098 | 0.00%/0.00% | 0.00/0.00 | not-applicable | - | Exact duplicate detection returns a compact note while current and prior raw outputs remain recoverable. |
-| repeated-command-output | improved-freeflow-run | pass | status ✓ exit ✓ facts ✓ failed-exact ✓ | success/0 | duplicate-output 1.00 exact | 4390/1098 | 1127/282 | 74.33%/74.32% | 2.21/2.31 | passed | ffout_3142c5e358f7098c9be2225c | Exact duplicate command output matched previous outputId=ffout_857f56abd6351518d5393719 by exact output hash and command/cwd/status fingerprint (executionStatus=success exitCode=0); returning a compact duplicate note instead of re-injecting repeated output. Current raw output was vaulted as outputId=ffout_3142c5e358f7098c9be2225c.; Exact duplicate detection returns a compact note while current and prior raw outputs remain recoverable.; repeatedRuns=2; outputIds=ffout_ddd734b516d927ca72811bb3,ffout_3142c5e358f7098c9be2225c; recovery=passed: Verified exact combined output recovery for ffout_3142c5e358f7098c9be2225c. |
-
-## Skipped Optional Command Compressors
-
-- RTK: Optional command compressor comparator is skipped unless a caller supplies a configured comparator hook.
-- Squeez: Optional session/output compressor comparator is skipped unless a caller supplies a configured comparator hook.
-
-## Regression Status
-
-Improved Freeflow command routing passed all gated command-output benchmark fixtures.
-
-Failed command facts preserved: yes.

package/evals/reports/runtime/output-router-index-benchmark-1-report.md

@@ -1,64 +0,0 @@
-# Output Router Index Benchmark Report - Iteration 1
-
-Date: 2026-06-28
-
-## Scope
-
-Repo Search Backend Benchmark for `freeflow_search`. The scanner remains the product default; this benchmark compares scanner-only retrieval, the no-dependency local lexical index, a conservative hybrid scanner+index path, and records whether an FTS5/BM25/trigram candidate is available.
-
-The index cache is keyed by repo root and stores outside the repo by default. No external service, vector DB, or native dependency is required for the local lexical index.
-
-## Command
-
-```sh
-npm run bench:router:index
-```
-
-The CLI writes machine-readable JSON under `evals/runs/output-router/` by default. That JSON is generated run data; this Markdown file is the durable runtime report.
-
-## Summary
-
-- Fixtures: 3
-- Scanner remains default: yes
-- Index adopted by default: no
-- Scanner pass: 3/3 (recall@3: 3/3)
-- Index warm pass: 3/3 (recall@3: 3/3)
-- FTS5/BM25/trigram pass: 3/3 (skipped: 0; recall@3: 3/3)
-- Hybrid warm pass: 3/3 (recall@3: 3/3)
-- FTS5/BM25/trigram candidate: available — Node node:sqlite is available and supports SQLite FTS5 trigram tokenization with bm25 ranking.
-- Generated false positives: 0/18
-- Cold build p50/p95: 0.56/0.60 ms
-- Warm query p50/p95: 0.04/0.05 ms
-- Stale refresh p50/p95: 0.60/0.77 ms
-- Index warm context reduction: 99.54% (310411 raw bytes to 1441 context bytes)
-
-## Results
-
-| fixture | mode | correctness | checks | path | lines | candidates | raw/context bytes | context reduction | latency p50/p95 ms | build/query ms | index mode | notes |
-| --- | --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- |
-| index-exact-copied-text | scanner-default | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/1201 | -751.77% | 1.84/8.82 | 0.00/0.49 | - | Deterministic repo retrieval selected 2 candidate(s); top result target.md:1-3 (matched exact normalized query phrase in section chunk). |
-| index-exact-copied-text | index-cold | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/678 | -380.85% | 1.45/2.77 | 0.56/0.05 | cold-built | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-z1mRAH/f172de04b87566a9fe7f030c.json |
-| index-exact-copied-text | index-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/678 | -380.85% | 2.36/2.50 | 0.42/0.04 | warm-loaded | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-DDeXo1/dd4502aad39189e59f4c0d1c.json |
-| index-exact-copied-text | index-stale-refresh | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 162/677 | -317.90% | 2.28/2.50 | 0.77/0.03 | stale-refreshed | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-6fTIT9/36a8bb3d60c28d09e4dfd81f.json; refresh=1 added |
-| index-exact-copied-text | fts5-bm25-trigram | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 141/204 | -44.68% | 1.07/2.11 | 0.21/0.21 | - | SQLite FTS5 trigram MATCH with bm25 rank -0.000006 |
-| index-exact-copied-text | hybrid-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md, decoy.md | 141/169 | -19.86% | 1.88/1.91 | 0.32/0.42 | warm-loaded | selection=scanner-index-agree; scannerQueryMs=0.39; indexQueryMs=0.03 |
-| index-generated-artifact-decoy | scanner-default | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-9 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/1394 | 99.55% | 1.82/1.86 | 0.00/0.45 | - | Deterministic repo retrieval selected 1 candidate(s); top result docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md:3-9 (BM25-style scored section chunk with 7/7 query-token coverage). |
-| index-generated-artifact-decoy | index-cold | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/441 | 99.86% | 2.06/2.09 | 0.60/0.09 | cold-built | BM25-style indexed window chunk with 5/5 query-token coverage; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-KqjOyP/28f108bffdc59e8c6fc367d1.json |
-| index-generated-artifact-decoy | index-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/441 | 99.86% | 2.28/2.30 | 0.40/0.05 | warm-loaded | BM25-style indexed window chunk with 5/5 query-token coverage; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-3yJwOY/2c1e76270adc1a02188256db.json |
-| index-generated-artifact-decoy | index-stale-refresh | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310241/441 | 99.86% | 2.77/3.02 | 0.60/0.04 | stale-refreshed | BM25-style indexed window chunk with 5/5 query-token coverage; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-vHpgJg/9242bdc233ce6203c5a6e6fe.json; refresh=1 added |
-| index-generated-artifact-decoy | fts5-bm25-trigram | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-7 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/315 | 99.90% | 1.82/1.89 | 0.22/0.22 | - | SQLite FTS5 trigram MATCH with bm25 rank -0.000007 |
-| index-generated-artifact-decoy | hybrid-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 3-9 | docs/codex-cli-agent-harness/2026-06-12-pass-3-sandboxing-and-permissions.md | 310220/236 | 99.92% | 2.69/2.82 | 0.41/0.49 | warm-loaded | selection=scanner-index-agree; scannerQueryMs=0.44; indexQueryMs=0.05 |
-| index-stale-refresh | scanner-default | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/873 | -1646.00% | 0.82/0.89 | 0.00/0.30 | - | Deterministic repo retrieval selected 1 candidate(s); top result target.md:1-3 (matched exact normalized query phrase in section chunk). |
-| index-stale-refresh | index-cold | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/322 | -544.00% | 0.97/1.09 | 0.38/0.02 | cold-built | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-IgHl0y/da51d69855603b1095f746c9.json |
-| index-stale-refresh | index-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/322 | -544.00% | 1.09/1.09 | 0.23/0.02 | warm-loaded | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-QNBb5t/04e813a08b0f962846486858.json |
-| index-stale-refresh | index-stale-refresh | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/322 | -544.00% | 1.29/1.94 | 0.54/0.03 | stale-refreshed | matched exact normalized query phrase in indexed section chunk; cachePath=/var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-router-index-benchmark-cache-DvsbMu/ecaeda95e6acb158b7eca6cf.json; refresh=1 changed |
-| index-stale-refresh | fts5-bm25-trigram | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/180 | -260.00% | 0.89/1.00 | 0.21/0.21 | - | SQLite FTS5 trigram MATCH with bm25 rank -0.000004 |
-| index-stale-refresh | hybrid-warm | pass | path ✓ span ✓ excerpt ✓ recall@3 ✓ gen-fp ✓ | target.md | 1-3 | target.md | 50/169 | -238.00% | 1.38/1.42 | 0.23/0.26 | warm-loaded | selection=scanner-index-agree; scannerQueryMs=0.24; indexQueryMs=0.02 |
-
-## Adoption Decision
-
-Index adopted by default: no. This slice only records benchmark evidence; scanner behavior remains the default until a later adoption checkpoint explicitly approves otherwise.
-
-## Regression Status
-
-Warm experimental index, FTS5/BM25/trigram, and hybrid modes passed all gated fixtures without generated-artifact false positives.

package/evals/reports/runtime/output-router-regression-1-report.md

@@ -1,81 +0,0 @@
-# Output Router Regression Eval Report - Iteration 1
-
-Date: 2026-06-16
-
-## Scope
-
-Added deterministic Freeflow Router regression fixtures and tests for output-routing behavior. Refactored the host-facing guidance into a real `output-router` skill, with the safety policy stored as a skill reference.
-
-Fixtures live under:
-
-- `evals/fixtures/output-router/`
-
-Regression tests live in:
-
-- `router/tests/regression-fixtures.test.js`
-
-Runtime-facing skill files live in:
-
-- `skills/output-router/SKILL.md`
-- `skills/output-router/references/safety-policy.md`
-
-## Failure Preserved
-
-The router must not merely reduce output size. It must also preserve exact evidence and recovery paths.
-
-The preserved failure modes are:
-
-- broad repo queries dumping whole files instead of bounded evidence,
-- `preserve: full` becoming a lossy summary over cap,
-- noisy command output flooding context without raw recovery,
-- failed-command output losing exact assertion/stack evidence,
-- verification output paraphrasing completion evidence,
-- native post-tool safety-net output being shortened without a visible Freeflow label and exact recovery path,
-- ambiguous top-level `status` hiding the difference between tool success, command failure, and routing success.
-
-## Deterministic Check
-
-Added fixture-backed tests for:
-
-- large docs targeted query,
-- whole-artifact full-fidelity over-cap chunk behavior,
-- noisy successful command reduction and raw vault recovery,
-- failed command exact evidence and split statuses,
-- verification summary exactness,
-- Pi native safety-net labeling and exact raw tail recovery.
-
-The tests assert token/byte reduction where expected, exact evidence preservation, vault recovery, and split status semantics.
-
-## Result
-
-`npm run test:router`: pass, 42 tests.
-
-Actual Pi smoke also passed for both local-extension loading and installed package discovery using an isolated temporary `PI_CODING_AGENT_DIR` cache:
-
-- `freeflow_retrieve` registration and execution through Pi.
-- `freeflow_run` execution through Pi's command runner with configured routing thresholds.
-- Native `read` post-tool safety-net routing with explicit `postToolRouting: safety-net`.
-
-## Verification
-
-Commands:
-
-```sh
-npm run test:router
-pi --no-session --no-extensions --no-skills --no-context-files --approve -e "$PWD/pi-extension/index.js" --tools freeflow_retrieve --mode json -p "Use the freeflow_retrieve tool exactly once with action=query, source.kind=repo, source.path='evals/fixtures/output-router/large-router-manual.md', query='OUTPUT_ROUTER_SKILL_DECISION_ANCHOR safety net', preserve=important. Then report the output path and whether the anchor appears."
-PI_CODING_AGENT_DIR=/tmp/freeflow-pi-agent-cache-... pi install "$PWD" --approve
-PI_CODING_AGENT_DIR=/tmp/freeflow-pi-agent-cache-... pi --no-session --no-context-files --approve --tools freeflow_retrieve --mode json -p "Use freeflow_retrieve..."
-# plus temp-project Pi smoke for freeflow_run and native read safety-net routing through package discovery
-```
-
-Supporting checks from the same implementation pass:
-
-```sh
-node --check pi-extension/index.js
-npm pack --dry-run --json
-rg -n "child_process|node:child_process|\bspawn\b|\bexecFile\b|\bexecSync\b" router pi-extension || true
-```
-
-## Recommendation
-
-Keep these deterministic regression fixtures in the router test suite. Add model-based agent evals only after the deterministic runtime behavior is stable and a specific agent-selection failure is observed.

package/evals/reports/runtime/output-router-transform-eval-1-report.md

@@ -1,28 +0,0 @@
-# Output Router Transform Eval - Iteration 1
-
-Date: 2026-06-28
-
-## Scope
-
-Targeted deterministic eval for transformed-output routing. It compares direct/manual long-log inspection against explicit Freeflow transform operations.
-
-## Command
-
-```sh
-npm run build && node evals/scripts/run-output-router-transform-eval.js
-```
-
-## Summary
-
-- Fixtures: 1
-- Objective gates passed: 4/4
-
-## Results
-
-| fixture | direct/raw baseline | Freeflow routed behavior | status | gates |
-| --- | --- | --- | --- | --- |
-| long-log-manual-inspection-vs-freeflow-transform | 7729 raw log bytes for manual inspection | 193 filtered evidence bytes; countMatches=2; exact transformed recovery | pass | manualBaselineWouldReadWholeLog ✓; filteredTargetFacts ✓; countMatches ✓; lineage ✓ |
-
-## Result
-
-All targeted transform gates passed for these deterministic fixtures.

package/evals/reports/runtime/pi-observed-routing-eval-1-report.md

@@ -1,44 +0,0 @@
-# Pi Observed Routing Eval - Iteration 1
-
-Date: 2026-06-24
-
-## Scope
-
-Targeted deterministic eval for the Pi-only observed output routing slice. It exercises Pi's `tool_result` path for configured MCP, web, fetch, and code-search producers after direct host execution.
-
-This eval compares raw direct tool-output fixtures against the routed result returned to the agent. It verifies bounded output, exact critical fact preservation, persistence/recovery claims, metadata-only no-raw-recovery behavior, mutating MCP routing as metadata (not a gate), and Pi status capability reporting.
-
-It does not claim Context Mode or cross-host superiority. Claude and Codex observed-routing adapters remain out of scope.
-
-## Command
-
-```sh
-npm run build && node evals/scripts/run-pi-observed-routing-eval.js
-```
-
-## Summary
-
-- Fixtures: 7
-- Objective gates passed: 28/28
-- Host: Pi only.
-- Direct execution/permissions: owned by Pi; Freeflow routes only completed tool output.
-- Persistence modes covered: exact and metadata-only.
-- Overall byte reduction, excluding status-only fixtures: 82.2%.
-
-## Results
-
-Raw and routed columns are `bytes / lines / items-or-evidence-packets`. Reduction is calculated from raw bytes to routed-result bytes.
-
-| fixture | producer | raw direct output | routed output | byte reduction | recoverability | status | gates |
-| --- | --- | ---: | ---: | ---: | --- | --- | --- |
-| mcp-github-search-exact | mcp:github:search_issues | 8589 / 36 / 4 | 1201 / 40 / 1 | 86.0% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
-| mcp-github-create-mutating | mcp:github:create_issue | 2492 / 8 / 1 | 889 / 29 / 1 | 64.3% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
-| mcp-gmail-search-metadata-only | mcp:gmail:search | 12097 / 4 / 1 | 832 / 13 / 1 | 93.1% | metadata_only | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
-| pi-web-search-exact | web:web_search | 1752 / 29 / 4 | 1184 / 34 / 1 | 32.4% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
-| pi-fetch-content-exact | fetch:fetch_content | 2631 / 6 / 1 | 853 / 25 / 1 | 67.6% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
-| pi-code-search-exact | code_search:code_search | 8712 / 26 / 3 | 1490 / 36 / 1 | 82.9% | exact | pass | routedByPiHook ✓; boundedOutput ✓; criticalFactsPreserved ✓; recoverabilityAccurate ✓ |
-| pi-capability-status | status | 0 / 0 / 0 | 4970 / 190 / 0 | n/a | n/a | pass | hostCapabilityReported ✓; persistenceModesReported ✓; unsupportedRedactedReported ✓; configuredProducerReported ✓ |
-
-## Result
-
-All targeted Pi observed-routing gates passed for these deterministic fixtures. Pi observed routing materially reduced large/noisy fixture output while preserving critical facts and truthful recovery behavior.