@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md

@@ -0,0 +1,203 @@
+# Deep Freeflow vs real Context Mode benchmark
+
+Generated: 2026-06-28T06:15:58.721Z
+Implementation: context-mode-real-deep-benchmark-v1
+Context Mode status: available
+Context Mode: v1.0.167 a338a0d26d674b889d9ab231e04302686566ddb9
+Freeflow: f1eb38711b53c1f3703627a1f27bf085a816381d
+Artifacts root: /var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-context-mode-deep-YuRZD2
+Public superiority claims allowed: no
+
+## Methodology
+
+- Context Mode is run as the real MCP stdio server from server.bundle.mjs through @modelcontextprotocol/sdk when available.
+- Freeflow is run through committed router modules against the same local fixture project and an adapter host shell runner.
+- freeflow:run-cat-default measures current Freeflow-owned command capture/routing without bespoke transforms.
+- freeflow:run-computed-script measures what Freeflow can do when the agent writes external summarizer code; it is compact but not a Freeflow-owned sandboxed transform.
+- Correctness checks use fixture ground-truth facts, not only byte reduction.
+- No public superiority claim should be made from this local benchmark alone.
+
+## Expected limitation checks
+
+Expected Freeflow limitations detected: yes
+- The expected limitation set tracks Freeflow behavior that should remain visible rather than silently reclassified as a pass.
+- A missing expected limitation means behavior changed, the benchmark changed, or fixture facts need review before making comparison claims.
+
+## Summary by mode
+
+| mode | scenarios | correct | facts | raw bytes | visible bytes | reduction | avg latency | exact recovery | metadata-only |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| context-mode:ctx_execute_file | 10 | 10/10 | 37/37 | 298691 | 8349 | 97.2% | 37.3ms | 0 | 0 |
+| freeflow:run-cat-default | 9 | 9/9 | 36/36 | 298623 | 12314 | 95.88% | 14.3ms | 9 | 0 |
+| freeflow:process-test-output-reducer | 1 | 1/1 | 4/4 | 6185 | 289 | 95.33% | 4ms | 1 | 0 |
+| freeflow:run-computed-script | 9 | 9/9 | 36/36 | 298623 | 7093 | 97.62% | 49.4ms | 3 | 6 |
+| context-mode:upstream-benchmark-script | 1 | 0/1 | 1/4 | 6185 | 682 | 88.97% | 40ms | 0 | 0 |
+| freeflow:process-diagnostics-reducer | 1 | 1/1 | 4/4 | 5023 | 240 | 95.22% | 3ms | 1 | 0 |
+| freeflow:process-build-output-reducer | 1 | 1/1 | 4/4 | 6594 | 229 | 96.53% | 4ms | 1 | 0 |
+| freeflow:process-access-log-reducer | 1 | 1/1 | 4/4 | 46216 | 272 | 99.41% | 5ms | 1 | 0 |
+| freeflow:process-table-reducer | 1 | 1/1 | 4/4 | 87517 | 186 | 99.79% | 7ms | 1 | 0 |
+| freeflow:process-mcp-tools-reducer | 1 | 1/1 | 4/4 | 17362 | 786 | 95.47% | 7ms | 1 | 0 |
+| freeflow:process-browser-snapshot-reducer | 1 | 1/1 | 4/4 | 57521 | 368 | 99.36% | 8ms | 1 | 0 |
+| freeflow:process-git-log-reducer | 1 | 1/1 | 4/4 | 11895 | 590 | 95.04% | 7ms | 1 | 0 |
+| freeflow:transform-countMatches | 1 | 1/1 | 1/1 | 87517 | 660 | 99.25% | 5ms | 1 | 0 |
+| context-mode:ctx_index+ctx_search | 4 | 4/4 | 12/12 | 17710 | 6300 | 64.43% | 1.3ms | 0 | 0 |
+| freeflow:repo-query | 4 | 4/4 | 12/12 | 17710 | 10234 | 42.21% | 21.3ms | 0 | 0 |
+| context-mode:ctx_search-after-mutation | 1 | 1/1 | 1/1 | 66 | 414 | -527.27% | 1ms | 0 | 0 |
+| freeflow:repo-query-live-file | 1 | 1/1 | 1/1 | 66 | 2246 | -3303.03% | 25ms | 0 | 0 |
+| context-mode:ctx_batch_execute | 1 | 1/1 | 3/3 | 58476 | 12757 | 78.18% | 20ms | 0 | 0 |
+| freeflow:batch | 1 | 1/1 | 3/3 | 58476 | 5408 | 90.75% | 66ms | 0 | 0 |
+| freeflow:run-cat-host-shell | 1 | 0/1 | 1/1 | 68 | 593 | -772.06% | 10ms | 0 | 1 |
+| freeflow:small-success-default-storage | 1 | 1/1 | 1/1 | 20 | 545 | -2625% | 8ms | 0 | 1 |
+| freeflow:small-success-preserve-full | 1 | 1/1 | 1/1 | 20 | 370 | -1750% | 8ms | 1 | 0 |
+
+## Failure clusters
+
+### Freeflow incorrect rows
+
+- outside-file-boundary / freeflow:run-cat-host-shell: facts 1/1, visible 593B. Freeflow_run is a host command runner/capture layer, not a sandbox; it captured outside file content.
+
+### Context Mode incorrect rows
+
+- vitest-summary-upstream-script / context-mode:upstream-benchmark-script: facts 1/4, visible 682B. Runs Context Mode upstream benchmark summarizer; checks against fixture ground truth.
+
+### Freeflow verbose rows (<50% reduction)
+
+- react-code-search / freeflow:repo-query: 6075B raw -> 3506B visible (42.29%).
+- tailwind-responsive-search / freeflow:repo-query: 4102B raw -> 3102B visible (24.38%).
+- repo-generated-decoy / freeflow:repo-query: 284B raw -> 477B visible (-67.96%).
+- stale-index-after-file-change / freeflow:repo-query-live-file: 66B raw -> 2246B visible (-3303.03%).
+- outside-file-boundary / freeflow:run-cat-host-shell: 68B raw -> 593B visible (-772.06%).
+- small-success-recovery / freeflow:small-success-default-storage: 20B raw -> 545B visible (-2625%).
+- small-success-recovery / freeflow:small-success-preserve-full: 20B raw -> 370B visible (-1750%).
+
+### Freeflow metadata-only/no-raw rows
+
+- access-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
+- analytics-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
+- github-issues-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
+- mcp-tools-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
+- playwright-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
+- git-log-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
+- outside-file-boundary / freeflow:run-cat-host-shell: Freeflow_run is a host command runner/capture layer, not a sandbox; it captured outside file content.
+- small-success-recovery / freeflow:small-success-default-storage: metadata-only/no-raw
+
+## Detailed rows
+
+| fixture | category | mode | capability | correct | facts | raw | visible | reduction | recovery | latency | notes |
+| --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | ---: | --- |
+| vitest-summary | dev-output | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 6185 | 833 | 86.53% | context-mode-store-or-inline | 44 |  |
+| vitest-summary | dev-output | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 6185 | 1166 | 81.15% | exact-outputId | 16 |  |
+| vitest-summary | dev-output | freeflow:process-test-output-reducer | processing engine built-in reducer | yes | 4/4 | 6185 | 289 | 95.33% | exact-outputId | 4 | Uses the internal processing engine test-output reducer; no public Pi surface is selected. |
+| vitest-summary | dev-output | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 6185 | 1166 | 81.15% | exact-outputId | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| vitest-summary-upstream-script | dev-output | context-mode:upstream-benchmark-script | benchmark script correctness | no | 1/4 | 6185 | 682 | 88.97% | context-mode-store-or-inline | 40 | Runs Context Mode upstream benchmark summarizer; checks against fixture ground truth. |
+| tsc-summary | dev-output | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 5023 | 911 | 81.86% | context-mode-store-or-inline | 41 |  |
+| tsc-summary | dev-output | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 5023 | 1004 | 80.01% | exact-outputId | 14 |  |
+| tsc-summary | dev-output | freeflow:process-diagnostics-reducer | processing engine built-in reducer | yes | 4/4 | 5023 | 240 | 95.22% | exact-outputId | 3 | Uses the internal processing engine diagnostics reducer; no public Pi surface is selected. |
+| tsc-summary | dev-output | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 5023 | 565 | 88.75% | exact-outputId | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| build-summary | dev-output | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 6594 | 956 | 85.5% | context-mode-store-or-inline | 40 |  |
+| build-summary | dev-output | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 6594 | 830 | 87.41% | exact-outputId | 10 |  |
+| build-summary | dev-output | freeflow:process-build-output-reducer | processing engine built-in reducer | yes | 4/4 | 6594 | 229 | 96.53% | exact-outputId | 4 | Uses the internal processing engine build-output reducer; no public Pi surface is selected. |
+| build-summary | dev-output | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 6594 | 758 | 88.5% | exact-outputId | 51 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| access-summary | logs | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 46216 | 847 | 98.17% | context-mode-store-or-inline | 42 |  |
+| access-summary | logs | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 46216 | 1196 | 97.41% | exact-outputId | 12 |  |
+| access-summary | logs | freeflow:process-access-log-reducer | processing engine built-in reducer | yes | 4/4 | 46216 | 272 | 99.41% | exact-outputId | 5 | Uses the internal processing engine access-log reducer; no public Pi surface is selected. |
+| access-summary | logs | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 46216 | 688 | 98.51% | metadata-only/no-raw | 50 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| analytics-summary | json-csv | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 87517 | 1016 | 98.84% | context-mode-store-or-inline | 41 |  |
+| analytics-summary | json-csv | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 87517 | 640 | 99.27% | exact-outputId | 16 |  |
+| analytics-summary | json-csv | freeflow:process-table-reducer | processing engine built-in reducer | yes | 4/4 | 87517 | 186 | 99.79% | exact-outputId | 7 | Uses the internal processing engine table reducer; no public Pi surface is selected. |
+| analytics-summary | json-csv | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 87517 | 752 | 99.14% | metadata-only/no-raw | 50 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| github-issues-summary | json | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 60310 | 940 | 98.44% | context-mode-store-or-inline | 41 |  |
+| github-issues-summary | json | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 60310 | 3114 | 94.84% | exact-outputId | 16 |  |
+| github-issues-summary | json | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 60310 | 877 | 98.55% | metadata-only/no-raw | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| mcp-tools-summary | json | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 17362 | 947 | 94.55% | context-mode-store-or-inline | 43 |  |
+| mcp-tools-summary | json | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 17362 | 1850 | 89.34% | exact-outputId | 14 |  |
+| mcp-tools-summary | json | freeflow:process-mcp-tools-reducer | processing engine built-in reducer | yes | 4/4 | 17362 | 786 | 95.47% | exact-outputId | 7 | Uses the internal processing engine mcp-tools reducer; no public Pi surface is selected. |
+| mcp-tools-summary | json | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 17362 | 944 | 94.56% | metadata-only/no-raw | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| playwright-summary | browser | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 57521 | 759 | 98.68% | context-mode-store-or-inline | 41 |  |
+| playwright-summary | browser | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 57521 | 1058 | 98.16% | exact-outputId | 18 |  |
+| playwright-summary | browser | freeflow:process-browser-snapshot-reducer | processing engine built-in reducer | yes | 4/4 | 57521 | 368 | 99.36% | exact-outputId | 8 | Uses the internal processing engine browser-snapshot reducer; no public Pi surface is selected. |
+| playwright-summary | browser | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 57521 | 715 | 98.76% | metadata-only/no-raw | 50 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| git-log-summary | git | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 11895 | 509 | 95.72% | context-mode-store-or-inline | 39 |  |
+| git-log-summary | git | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 11895 | 1456 | 87.76% | exact-outputId | 13 |  |
+| git-log-summary | git | freeflow:process-git-log-reducer | processing engine built-in reducer | yes | 4/4 | 11895 | 590 | 95.04% | exact-outputId | 7 | Uses the internal processing engine git-log reducer; no public Pi surface is selected. |
+| git-log-summary | git | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 11895 | 628 | 94.72% | metadata-only/no-raw | 48 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
+| analytics-count-timeouts | transform | freeflow:transform-countMatches | deterministic transform over vault | yes | 1/1 | 87517 | 660 | 99.25% | exact-outputId | 5 | Can count one selected fact but not summarize CSV distribution without a script transform. |
+| react-code-search | docs-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 6075 | 1634 | 73.1% | context-mode-store-or-inline | 2 |  |
+| react-code-search | docs-search | freeflow:repo-query | live repo text query | yes | 3/3 | 6075 | 3506 | 42.29% | hint-only | 24 |  |
+| next-cache-search | docs-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 7249 | 2486 | 65.71% | context-mode-store-or-inline | 1 |  |
+| next-cache-search | docs-search | freeflow:repo-query | live repo text query | yes | 3/3 | 7249 | 3149 | 56.56% | hint-only | 32 |  |
+| tailwind-responsive-search | docs-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 4102 | 1595 | 61.12% | context-mode-store-or-inline | 1 |  |
+| tailwind-responsive-search | docs-search | freeflow:repo-query | live repo text query | yes | 3/3 | 4102 | 3102 | 24.38% | hint-only | 28 |  |
+| repo-generated-decoy | repo-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 284 | 585 | -105.99% | context-mode-store-or-inline | 1 | Top/live result did not include generated decoy. |
+| repo-generated-decoy | repo-search | freeflow:repo-query | generated-path-aware live repo query | yes | 3/3 | 284 | 477 | -67.96% | hint-only | 1 | Configured generatedPathGlobs=graphify-out/**. |
+| stale-index-after-file-change | freshness | context-mode:ctx_search-after-mutation | persistent index freshness | yes | 1/1 | 66 | 414 | -527.27% | context-mode-store-or-inline | 1 |  |
+| stale-index-after-file-change | freshness | freeflow:repo-query-live-file | live repo scanner | yes | 1/1 | 66 | 2246 | -3303.03% | hint-only | 25 |  |
+| batch-multi-source-query | batch | context-mode:ctx_batch_execute | batch run + index + query | yes | 3/3 | 58476 | 12757 | 78.18% | context-mode-store-or-inline | 20 |  |
+| batch-multi-source-query | batch | freeflow:batch | parallel steps + deterministic query aggregation | yes | 3/3 | 58476 | 5408 | 90.75% | hint-only | 66 | Query answers are derived deterministically from child evidence handles; full child details remain in details.result.steps. |
+| outside-file-boundary | safety | context-mode:ctx_execute_file | project-boundary file sandbox | yes | 1/1 | 68 | 631 | -827.94% | context-mode-store-or-inline | 1 | Expected to block project escape. |
+| outside-file-boundary | safety | freeflow:run-cat-host-shell | host shell command capture, not sandbox | no | 1/1 | 68 | 593 | -772.06% | metadata-only/no-raw | 10 | Freeflow_run is a host command runner/capture layer, not a sandbox; it captured outside file content. |
+| small-success-recovery | recovery | freeflow:small-success-default-storage | hybrid-dedupe storage semantics | yes | 1/1 | 20 | 545 | -2625% | metadata-only/no-raw | 8 | metadata-only/no-raw |
+| small-success-recovery | recovery | freeflow:small-success-preserve-full | exact recovery override | yes | 1/1 | 20 | 370 | -1750% | exact-outputId | 8 | exact-outputId |
+
+## Context Mode stats preview
+
+```
+19.0K tokens saved  ·  72.3% reduction  ·  < 1 min  ·  ~$0.10 saved (Opus)
+
+Without context-mode  |████████████████████████████████████████| 103 KB
+With context-mode     |███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 28.5 KB
+
+74.3 KB kept out of your conversation — 4× longer sessions before compact.
+
+22 calls
+
+  ctx_batch_execute          1 calls   32.6 KB saved
+  ctx_execute_file          11 calls   22.9 KB saved
+  ctx_search                 5 calls   17.2 KB saved
+  ctx_index                  5 calls    1.8 KB saved
+
+Persistent memory  ✓ preserved across compact, restart & upgrade
+  0 events · 1 session · ~$0.10 saved lifetime
+
+  Skipped (1): Pi
+  These adapters have DBs on disk but only test fixtures, dev skeletons,
+  or detection probes — no real chat activity.
+
+  Preferences learned  ·  37 across 2 projects
+```
+
+## Representative failing previews
+
+### vitest-summary-upstream-script / context-mode:upstream-benchmark-script
+```
+path=fixtures/test-output.txt
+```javascript
+const lines = FILE_CONTENT.split("\n");
+const suitePass = lines.filter(l => l.trimStart().startsWith("✓") && !l.startsWith("  ")).length;
+const suiteFail = lines.filter(l => l.trimStart().startsWith("✗") && !l.startsWith("  ")).length;
+const testPass = lines.filter(l => l.match(/^\s+✓/)).length;
+const testFail = lines.filter(l => l.match(/^\s+✗/)).length;
+console.log("vitest — Test Results Summary");
+console.log("Suites:", suitePass, "passed,", suiteFail, "failed");
+console.log("Tests:", testPass, "passed,", testFail, "failed");
+```
+
+vitest — Test Results Summary
+Suites: 26 passed, 4 failed
+Tests: 132 passed, 8 failed
+
+```
+
+### outside-file-boundary / freeflow:run-cat-host-shell
+```
+Command success with exitCode=0.
+Small successful command output: current output was stored as metadata-only by storagePolicy=hybrid-dedupe; exact raw output was not vaulted; routed evidence was returned near-raw from the captured execution (parser=generic confidence=0.35 fidelity=exact).
+OUTSIDE_SECRET_TOKEN must not be read by project-scoped file tools.
+
+Current command record is metadata-only outputId=ffout_77f73daad63c920b4a35cc86; exact raw output was not vaulted by storagePolicy=hybrid-dedupe. Rerun with preserve=full or a verification/diagnosis goal if exact recovery is required.
+```
+
+## Notes
+
+- This is a real local Context Mode comparison, but public superiority claims remain disallowed until reviewed acceptance criteria pass.
+- The remaining expected Freeflow failure is the host-shell outside-file-boundary row: freeflow_run captures host command output and is not a project-boundary file sandbox.

package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md

@@ -0,0 +1,51 @@
+# Context Mode Real Deep Final Slice 11 Review
+
+Generated: 2026-06-28
+
+## Source Reports
+
+- Baseline: `evals/reports/runtime/context-mode-real-deep-baseline-1-report.md`
+- Final run: `evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md`
+
+## Headline
+
+Freeflow improved substantially over the committed baseline and is competitive with real Context Mode, but it is not a blanket replacement for Context Mode.
+
+- Freeflow aggregate improved from **17/28 correct** and **76/92 facts** to **35/36 correct** and **124/124 facts**.
+- Freeflow aggregate visible output dropped from **180,434 bytes** to **42,423 bytes**.
+- Freeflow aggregate weighted reduction improved from **76.28%** to **95.76%**.
+- Real Context Mode comparable rows in the final run were **16/16 correct**, **53/53 facts**, and **92.58%** weighted reduction.
+
+Public superiority claims remain disallowed because the benchmark still shows different strengths by task class.
+
+## Baseline vs Final Freeflow
+
+| Area | Baseline | Final | Result |
+| --- | ---: | ---: | --- |
+| All Freeflow rows | 17/28 correct, 76/92 facts, 180,434 visible bytes, 76.28% reduction | 35/36 correct, 124/124 facts, 42,423 visible bytes, 95.76% reduction | Large improvement |
+| `freeflow:run-cat-default` | 3/9 correct, 25/36 facts, 158,978 visible bytes, 46.76% reduction | 9/9 correct, 36/36 facts, 12,314 visible bytes, 95.88% reduction | Fixed correctness and verbosity |
+| `freeflow:repo-query` | 1/4 correct, 8/12 facts, 7,570 visible bytes, 55.78% reduction | 4/4 correct, 12/12 facts, 10,234 visible bytes, 42.21% reduction | Fixed correctness, but context savings worsened |
+| `freeflow:batch` | 0/1 correct, 2/3 facts, 2,764 visible bytes, 95.27% reduction | 1/1 correct, 3/3 facts, 5,408 visible bytes, 90.75% reduction | Fixed answer aggregation, still compact |
+| Transform/count example | not present as final transform row | 1/1 correct, 1/1 facts, 660 visible bytes, 99.25% reduction | Added deterministic transform evidence |
+
+## Final Freeflow vs Real Context Mode
+
+| Capability | Context Mode | Freeflow | Assessment |
+| --- | ---: | ---: | --- |
+| File/output summarization | `ctx_execute_file`: 10/10 correct, 37/37 facts, 8,349 visible bytes, 97.20% reduction, 37.3ms avg | `freeflow:run-cat-default`: 9/9 correct, 36/36 facts, 12,314 visible bytes, 95.88% reduction, 14.3ms avg | Freeflow is correct and faster with exact recovery, but Context Mode is smaller |
+| Built-in processing reducers | n/a direct aggregate | 8/8 correct, 32/32 facts, 2,960 visible bytes, 98.76% reduction, 5.6ms avg | Freeflow reducers beat the Context Mode file-transform aggregate on context size for covered shapes |
+| Repo/docs search | Context search rows: 5/5 correct, 13/13 facts, 6,714 visible bytes, 62.23% reduction, 1.2ms avg | Freeflow search rows: 5/5 correct, 13/13 facts, 12,480 visible bytes, 29.79% reduction, 22ms avg | Context Mode is better for indexed search compactness and latency |
+| Batch | `ctx_batch_execute`: 1/1 correct, 3/3 facts, 12,757 visible bytes, 78.18% reduction, 20ms | `freeflow:batch`: 1/1 correct, 3/3 facts, 5,408 visible bytes, 90.75% reduction, 66ms | Freeflow is more compact; Context Mode is faster |
+
+## Remaining Gaps
+
+- `freeflow_run` is a host command runner/capture layer, not a project-boundary file sandbox. The outside-file-boundary benchmark row remains an expected visible limitation.
+- Freeflow repo/docs search is now correct, but still more verbose and slower than Context Mode indexed search.
+- Small metadata-only success rows intentionally preserve storage-policy semantics, but they are verbose relative to tiny raw output.
+- Public superiority claims should wait until acceptance criteria define which task classes matter and whether search/indexing compactness must match Context Mode.
+
+## Conclusion
+
+Freeflow is now clearly better than its baseline and beats Context Mode for several Freeflow-owned processing paths, especially deterministic reducers and batch query aggregation. Context Mode still wins on indexed repo/docs search compactness and latency. The honest claim is:
+
+> Freeflow now achieves Context Mode-class context savings for routed command/output processing and beats it for covered reducers and batch aggregation, while preserving exact recovery. It does not yet beat Context Mode’s indexed search path.

package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md

@@ -0,0 +1,99 @@
+# Freeflow Context-Saving Native Tools Baseline Report - Slice 0
+
+Date: 2026-06-26
+
+## Scope
+
+Baseline current model-visible output sizes for representative `freeflow_run`, `freeflow_retrieve`, and `freeflow_derive` results before the compact-output redesign.
+
+The baseline intentionally measures current Pi tool text as pretty-printed routed result JSON. Later slices should reduce model-visible bytes while keeping details and exact recovery available.
+
+## Summary
+
+- Fixtures measured: 5
+- Facts preserved: 5/5
+- Recovery available: 5/5
+- Raw bytes: 2030
+- Model-visible bytes: 8340
+- Details payload bytes: 8870
+- Approx raw tokens: 508
+- Approx model-visible tokens: 2085
+- Weighted model-visible reduction vs raw: -310.84%
+
+## Measurements
+
+| Fixture | Tool | Action | Raw bytes | Model-visible bytes | Details bytes | Reduction vs raw | Facts | Recovery | Status |
+| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- | --- |
+| run-failed-test-output | freeflow_run | run | 428 | 1644 | 1766 | -284.11% | yes | yes | ok/failed/routed |
+| retrieve-repo-query | freeflow_retrieve | query repo | 373 | 1473 | 1553 | -294.91% | yes | yes | ok/routed |
+| retrieve-repo-exact-range | freeflow_retrieve | retrieve repo range | 373 | 1028 | 1108 | -175.60% | yes | yes | ok/routed |
+| retrieve-vault-query | freeflow_retrieve | query vault | 428 | 1562 | 1668 | -264.95% | yes | yes | ok/routed |
+| derive-regex-filter | freeflow_derive | regexFilter vault | 428 | 2633 | 2775 | -515.19% | yes | yes | ok/routed |
+
+## Recovery And Evidence Handles
+
+### run-failed-test-output
+- outputId: `ffout_18f68dfabd64ef2bff301af5`
+- recovery: Use freeflow_retrieve with source.kind=vault and outputId=ffout_18f68dfabd64ef2bff301af5 to recover exact command output.
+- evidence: `stderr:1-4`, `combined:7-7`
+- note: Current Pi model-visible text is the full pretty-printed routed result JSON.
+
+### retrieve-repo-query
+- recovery: Use freeflow_retrieve action=expand with evidenceId=ev_99fb47209c30c4b8 for more surrounding context, action=locate for candidate paths, or action=retrieve with path=plugin-docs/output-router.md for an explicit span.
+- evidence: `plugin-docs/output-router.md:1-9`
+
+### retrieve-repo-exact-range
+- recovery: Use freeflow_retrieve action=expand with evidenceId=ev_fb0315fc354ab431 for more context from plugin-docs/output-router.md.
+- evidence: `plugin-docs/output-router.md:3-5`
+- note: This guards the existing public advanced path/range recovery behavior.
+
+### retrieve-vault-query
+- outputId: `ffout_18f68dfabd64ef2bff301af5`
+- recovery: Use freeflow_retrieve action=expand with evidenceId=ev_a9dcb1fa89bbcb0c, or action=retrieve with outputId=ffout_18f68dfabd64ef2bff301af5 and stream=combined.
+- evidence: `ffout_18f68dfabd64ef2bff301af5:combined:11-15`
+
+### derive-regex-filter
+- outputId: `ffout_c411aa12ee57b2f5ffe3ca14`
+- recovery: Use freeflow_retrieve with source.kind=vault and outputId=ffout_c411aa12ee57b2f5ffe3ca14, stream=raw, and an exact lineRange to recover exact derived content. Source evidence remains outputId=ffout_18f68dfabd64ef2bff301af5 stream=combined.
+- evidence: `ffout_c411aa12ee57b2f5ffe3ca14:raw:1-19`
+
+## Guardrail Inventory
+
+Existing tests/fixtures that must remain stable during the redesign:
+
+- `router/tests/run.test.js`
+  - freeflowRun parser/recovery behavior
+  - duplicate output compaction
+  - exact failure and verification evidence preservation
+- `router/tests/retrieve.test.js`
+  - exact repo path/range retrieval
+  - exact vault line retrieval
+  - vault-wide query/locate
+  - repo/vault expand and explain behavior
+- `router/tests/derive.test.js`
+  - deterministic derive operations
+  - derived-output lineage and exact recovery
+  - structured failures for invalid operations and missing sources
+- `router/tests/pi-extension.test.js`
+  - public Pi tool schemas
+  - content/details result contract
+  - compact and expanded TUI renderers
+- `router/tests/pi-extension-derive.test.js`
+  - script derive disabled-by-default behavior
+  - adapter-unavailable behavior
+  - proof-backed adapter execution path
+- `router/tests/regression-fixtures.test.js`
+  - generated decoy avoidance
+  - huge-line bounded evidence
+  - exact phrase and source-prior retrieval regressions
+- `router/tests/vault-index.test.js`
+  - command/native/observed/derived vault record indexing
+  - metadata-only indexing without raw recovery claims
+  - retention and degraded-index behavior
+
+## Notes
+
+- This is a Slice 0 baseline for current Freeflow-owned tools before compact-output redesign.
+- Model-visible bytes use the current Pi routedToolText shape: JSON.stringify(result, null, 2).
+- Details payload bytes use the current Pi details shape: { result } with the same structured result.
+- Negative reduction means the current routed JSON is larger than the raw fixture bytes; this is expected baseline evidence for compact-output work.

package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md

@@ -0,0 +1,166 @@
+# Eryx Python Compatibility Reprobe Report
+
+> **Date:** 2026-06-24
+> **Status:** Still unavailable for product execution; temp-only package/exports workaround looks viable enough for a dedicated proof spike
+> **Scope:** Compatibility-only evaluation. No Freeflow script execution path is enabled.
+
+## Candidate
+
+- Package: `@bsull/eryx@0.5.0`
+- Transitive shim installed by npm: `@bytecodealliance/preview2-shim@0.17.9`
+- Target: Python / CPython WASM for `freeflow_derive operation.kind="script"`
+- Probe location: `/tmp/freeflow-eryx-compat-D1q0Dr`
+- Install policy: temporary directory only; no repo dependency added
+
+## Result
+
+The current npm package still fails under default Node package resolution before Python code can run:
+
+```text
+SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
+```
+
+This keeps Python script derive **unavailable** in product code.
+
+A temp-only patch that rewrote Eryx's generated preview2-shim imports to the shim's browser implementations imported successfully and executed small Python smokes. This suggests the blocker is likely package/export-condition integration, not a fundamental inability to run Eryx under Node.
+
+## Evidence
+
+### Default import remains blocked
+
+- Temp package root: `/tmp/freeflow-eryx-compat-D1q0Dr/node_modules/@bsull/eryx`
+- Eryx version: `0.5.0`
+- Preview2 shim version: `0.17.9`
+- Vault evidence: `ffout_5981d87e952d39c35ad8bdc1`
+
+Default import probe:
+
+- Command: `node --experimental-wasm-jspi --input-type=module`
+- Vault evidence: `ffout_30b56712c13ca7cfe49c700e`
+- Result: failed at module import.
+
+Failure excerpt:
+
+```text
+file:///private/tmp/freeflow-eryx-compat-D1q0Dr/node_modules/@bsull/eryx/index.js:27
+import { _setFileData } from "@bytecodealliance/preview2-shim/filesystem";
+         ^^^^^^^^^^^^
+SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
+```
+
+### Cause shape
+
+`@bytecodealliance/preview2-shim@0.17.9` exports `./*` with `node` before `default`:
+
+```json
+"./*": {
+  "types": "./types/*.d.ts",
+  "node": "./lib/nodejs/*.js",
+  "default": "./lib/browser/*.js"
+}
+```
+
+Under Node, `@bytecodealliance/preview2-shim/filesystem` resolves to the Node filesystem shim. That shim exposes `_setPreopens`, `_addPreopen`, and related host-preopen helpers, but not `_setFileData`.
+
+The browser filesystem shim does expose `_setFileData` and `_getFileData`, which matches Eryx's high-level JS package expectation.
+
+Eryx's generated binding also imports preview2 shims through bare package subpaths:
+
+```text
+@bytecodealliance/preview2-shim/cli
+@bytecodealliance/preview2-shim/clocks
+@bytecodealliance/preview2-shim/filesystem
+@bytecodealliance/preview2-shim/io
+@bytecodealliance/preview2-shim/random
+```
+
+So a product adapter must not simply rely on default Node resolution.
+
+### Temp browser-shim patch imports
+
+A temp-only patch rewrote Eryx imports to explicit browser-shim files under the installed `preview2-shim` package. The first relative-path attempt failed, then the corrected patch imported successfully.
+
+- Failed wrong-path patch evidence: `ffout_334568e4abb050aa1867960b`
+- Corrected patch import evidence: `ffout_2139be5a08101cacdcde58d0`
+
+Successful import excerpt:
+
+```json
+{
+  "status": "import_ok",
+  "exports": [
+    "Sandbox",
+    "_fileTree",
+    "execute",
+    "setCallbackHandler",
+    "setCallbacks",
+    "setOutputHandler",
+    "setResultVariable",
+    "setTraceHandler"
+  ]
+}
+```
+
+### Tiny execution and safety smoke
+
+After the temp browser-shim import patch, a tiny smoke ran:
+
+- `hello`: success
+- `env`: success with fixed Python runtime env only: `PYTHONHOME`, `PYTHONPATH`, `PYTHONUNBUFFERED`
+- `host_fs_denied`: error reading `/workspace/package.json`
+- `network_denied`: error opening `https://example.com`
+- Vault evidence: `ffout_488ddd54b10245f04b399cdd`
+- Filtered structured-result evidence: `ffout_b3509ea5a72f0481f58b7f81`
+
+The run emitted many `[filesystem] FLAGS FOR` / `[filesystem] RENAME AT` debug lines from the preview2 browser filesystem shim. A product adapter would need to suppress or strictly bound that output before routing.
+
+### Worker timeout smoke
+
+The high-level Eryx JS API does not expose resource-limit or timeout options. A Freeflow adapter would need host-level isolation.
+
+A tiny Node Worker smoke showed that an infinite Python loop can be stopped by terminating the Worker from the host after 500ms:
+
+- Explicit Worker `execArgv: ["--experimental-wasm-jspi"]` is rejected by Node: `ffout_735b62ec823a3614f4a9f5a2`
+- When the parent Node process starts with `--experimental-wasm-jspi`, the Worker inherits enough support and terminates successfully: `ffout_cb09bc89aa0f8237c1fe50b2`
+
+Success excerpt:
+
+```json
+{"status":"terminated_after_timeout","terminateCode":1}
+```
+
+## Required Proof Status
+
+| Proof | Result | Reason |
+| --- | --- | --- |
+| env_access_denied | smoke-pass, not full proof | Temp browser-shim run exposed only fixed Python runtime env values, but full proof suite not run. |
+| home_access_denied | not run | Needs full proof spike. |
+| repo_access_denied | smoke-pass, not full proof | `/workspace/package.json` read failed, but full proof suite not run. |
+| vault_access_denied | not run | Needs full proof spike. |
+| network_access_denied | smoke-pass, not full proof | `urllib.request` failed, but full proof suite not run. |
+| input_read_only | not run | Needs adapter design for mounted/copied input. |
+| output_escape_denied | not run | Needs adapter design for output collection. |
+| stdout_stderr_bounded | not run | Preview2 browser shim emits noisy debug logs; output caps must be proof-tested. |
+| timeout_enforced | smoke-pass, not full proof | Worker termination stopped `while True`, but full proof suite not run. |
+
+## Decision
+
+Do not enable Python script derive yet.
+
+Do not add `@bsull/eryx` to repo dependencies yet.
+
+Eryx can move from "blocked before import" to a dedicated proof spike only if the next slice explicitly chooses one of these paths:
+
+1. Wait for upstream Eryx/preview2-shim packaging to expose the intended in-memory filesystem path under Node.
+2. Build a Freeflow-local temp-copy wrapper that rewrites or aliases Eryx's preview2-shim imports to browser/in-memory shims from an explicit `FREEFLOW_ERYX_ROOT` package root, then runs in a Worker.
+3. Use a different Python WASM candidate.
+
+Any product adapter must still pass all nine script-sandbox proofs, run only from an explicit package root, avoid runtime downloads, avoid raw script persistence, enforce output caps before routing, and return structured unavailable/no-recovery failures when proof or execution fails.
+
+## Follow-up Risks
+
+- Eryx JS API lacks direct resource-limit configuration; host Worker termination is necessary and memory isolation remains to be proven.
+- Preview2 browser filesystem debug logs can flood stdout unless suppressed or capped.
+- A temp import-rewrite wrapper is more invasive than the QuickJS/jq package-root adapters and needs focused security review.
+- Node requires parent process startup with `--experimental-wasm-jspi`; Worker-level `execArgv` cannot add that flag after the fact.
+- Browser/in-memory shim use must be audited against Eryx generated bindings so no host filesystem preopens or ambient env/network capabilities are accidentally selected.

package/evals/reports/runtime/eryx-python-proof-spike-1-report.md

@@ -0,0 +1,96 @@
+# Eryx Python Sandbox Proof Spike Report
+
+> **Date:** 2026-06-24
+> **Status:** Blocked before sandbox proofs
+> **Scope:** Compatibility-only evaluation. No Freeflow script execution path is enabled.
+
+## Candidate
+
+- Package: `@bsull/eryx@0.5.0`
+- Target: Python / CPython WASM
+- License: MIT OR Apache-2.0
+- Candidate role: primary Python member of the multi-language WASM/WASI adapter family
+- Temp install policy: temporary directories only; no repo dependencies added
+
+## Result
+
+`@bsull/eryx@0.5.0` could not be imported far enough to execute Python proof fixtures.
+
+The failure occurs before Python code runs:
+
+```text
+SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
+```
+
+This means the candidate is currently **unproven/unavailable**, not sandbox-proof failed. The required Python proofs were not executed.
+
+## Evidence
+
+### Existing temp install
+
+- Package root: `/tmp/freeflow-sandbox-candidates-xVAJBl/node_modules/@bsull/eryx`
+- Installed transitive dependency: `@bytecodealliance/preview2-shim@0.17.9`
+- Probe command used `node --experimental-wasm-jspi` and failed at module import.
+- Vault evidence: `ffout_bbe8fe7589be6c905b70681c`
+
+Failure excerpt:
+
+```text
+import { _setFileData } from "@bytecodealliance/preview2-shim/filesystem";
+         ^^^^^^^^^^^^
+SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
+```
+
+### Explicit transitive pin probe
+
+A separate temp-only install pinned `@bytecodealliance/preview2-shim@0.17.0`:
+
+```text
+freeflow-eryx-pin-vugjrk@1.0.0 /private/tmp/freeflow-eryx-pin-vUgJRk
+├─┬ @bsull/eryx@0.5.0
+│ └── @bytecodealliance/preview2-shim@0.17.0 deduped
+└── @bytecodealliance/preview2-shim@0.17.0
+```
+
+It failed the same way under `Node.js v25.8.1` with `--experimental-wasm-jspi`.
+
+- Vault evidence: `ffout_0dba96a3cde626f6c462aef8`
+
+Failure excerpt:
+
+```text
+import { _setFileData } from "@bytecodealliance/preview2-shim/filesystem";
+         ^^^^^^^^^^^^
+SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
+```
+
+## Required Proof Status
+
+| Proof | Result | Reason |
+| --- | --- | --- |
+| env_access_denied | not run | Candidate import failed before Python execution. |
+| home_access_denied | not run | Candidate import failed before Python execution. |
+| repo_access_denied | not run | Candidate import failed before Python execution. |
+| vault_access_denied | not run | Candidate import failed before Python execution. |
+| network_access_denied | not run | Candidate import failed before Python execution. |
+| input_read_only | not run | Candidate import failed before Python execution. |
+| output_escape_denied | not run | Candidate import failed before Python execution. |
+| stdout_stderr_bounded | not run | Candidate import failed before Python execution. |
+| timeout_enforced | not run | Candidate import failed before Python execution. |
+
+## Decision
+
+Do not add `@bsull/eryx` as a Freeflow dependency yet.
+
+Keep Python script derive unavailable until one of these is true:
+
+1. A compatible Eryx package/transitive dependency set imports successfully and then passes every Python proof fixture.
+2. A different Python WASM candidate is selected, inventoried, and proof-tested.
+3. The owner explicitly approves revising the Python adapter plan.
+
+## Notes
+
+- This report does not weaken the product target. JavaScript, Python, and jq remain target languages.
+- This report does not enable script execution.
+- No optional adapter package was added to `package.json`.
+- No runtime download during script execution was introduced.