@hassangameryt/freeflow 0.3.0 → 0.5.0

package/evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md

@@ -1,203 +0,0 @@
-# Deep Freeflow vs real Context Mode benchmark
-
-Generated: 2026-06-28T06:15:58.721Z
-Implementation: context-mode-real-deep-benchmark-v1
-Context Mode status: available
-Context Mode: v1.0.167 a338a0d26d674b889d9ab231e04302686566ddb9
-Freeflow: f1eb38711b53c1f3703627a1f27bf085a816381d
-Artifacts root: /var/folders/2x/tsrlzqfx3ld_fn5bmr1_3l600000gn/T/freeflow-context-mode-deep-YuRZD2
-Public superiority claims allowed: no
-
-## Methodology
-
-- Context Mode is run as the real MCP stdio server from server.bundle.mjs through @modelcontextprotocol/sdk when available.
-- Freeflow is run through committed router modules against the same local fixture project and an adapter host shell runner.
-- freeflow:run-cat-default measures current Freeflow-owned command capture/routing without bespoke transforms.
-- freeflow:run-computed-script measures what Freeflow can do when the agent writes external summarizer code; it is compact but not a Freeflow-owned sandboxed transform.
-- Correctness checks use fixture ground-truth facts, not only byte reduction.
-- No public superiority claim should be made from this local benchmark alone.
-
-## Expected limitation checks
-
-Expected Freeflow limitations detected: yes
-- The expected limitation set tracks Freeflow behavior that should remain visible rather than silently reclassified as a pass.
-- A missing expected limitation means behavior changed, the benchmark changed, or fixture facts need review before making comparison claims.
-
-## Summary by mode
-
-| mode | scenarios | correct | facts | raw bytes | visible bytes | reduction | avg latency | exact recovery | metadata-only |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| context-mode:ctx_execute_file | 10 | 10/10 | 37/37 | 298691 | 8349 | 97.2% | 37.3ms | 0 | 0 |
-| freeflow:run-cat-default | 9 | 9/9 | 36/36 | 298623 | 12314 | 95.88% | 14.3ms | 9 | 0 |
-| freeflow:process-test-output-reducer | 1 | 1/1 | 4/4 | 6185 | 289 | 95.33% | 4ms | 1 | 0 |
-| freeflow:run-computed-script | 9 | 9/9 | 36/36 | 298623 | 7093 | 97.62% | 49.4ms | 3 | 6 |
-| context-mode:upstream-benchmark-script | 1 | 0/1 | 1/4 | 6185 | 682 | 88.97% | 40ms | 0 | 0 |
-| freeflow:process-diagnostics-reducer | 1 | 1/1 | 4/4 | 5023 | 240 | 95.22% | 3ms | 1 | 0 |
-| freeflow:process-build-output-reducer | 1 | 1/1 | 4/4 | 6594 | 229 | 96.53% | 4ms | 1 | 0 |
-| freeflow:process-access-log-reducer | 1 | 1/1 | 4/4 | 46216 | 272 | 99.41% | 5ms | 1 | 0 |
-| freeflow:process-table-reducer | 1 | 1/1 | 4/4 | 87517 | 186 | 99.79% | 7ms | 1 | 0 |
-| freeflow:process-mcp-tools-reducer | 1 | 1/1 | 4/4 | 17362 | 786 | 95.47% | 7ms | 1 | 0 |
-| freeflow:process-browser-snapshot-reducer | 1 | 1/1 | 4/4 | 57521 | 368 | 99.36% | 8ms | 1 | 0 |
-| freeflow:process-git-log-reducer | 1 | 1/1 | 4/4 | 11895 | 590 | 95.04% | 7ms | 1 | 0 |
-| freeflow:transform-countMatches | 1 | 1/1 | 1/1 | 87517 | 660 | 99.25% | 5ms | 1 | 0 |
-| context-mode:ctx_index+ctx_search | 4 | 4/4 | 12/12 | 17710 | 6300 | 64.43% | 1.3ms | 0 | 0 |
-| freeflow:repo-query | 4 | 4/4 | 12/12 | 17710 | 10234 | 42.21% | 21.3ms | 0 | 0 |
-| context-mode:ctx_search-after-mutation | 1 | 1/1 | 1/1 | 66 | 414 | -527.27% | 1ms | 0 | 0 |
-| freeflow:repo-query-live-file | 1 | 1/1 | 1/1 | 66 | 2246 | -3303.03% | 25ms | 0 | 0 |
-| context-mode:ctx_batch_execute | 1 | 1/1 | 3/3 | 58476 | 12757 | 78.18% | 20ms | 0 | 0 |
-| freeflow:batch | 1 | 1/1 | 3/3 | 58476 | 5408 | 90.75% | 66ms | 0 | 0 |
-| freeflow:run-cat-host-shell | 1 | 0/1 | 1/1 | 68 | 593 | -772.06% | 10ms | 0 | 1 |
-| freeflow:small-success-default-storage | 1 | 1/1 | 1/1 | 20 | 545 | -2625% | 8ms | 0 | 1 |
-| freeflow:small-success-preserve-full | 1 | 1/1 | 1/1 | 20 | 370 | -1750% | 8ms | 1 | 0 |
-
-## Failure clusters
-
-### Freeflow incorrect rows
-
-- outside-file-boundary / freeflow:run-cat-host-shell: facts 1/1, visible 593B. Freeflow_run is a host command runner/capture layer, not a sandbox; it captured outside file content.
-
-### Context Mode incorrect rows
-
-- vitest-summary-upstream-script / context-mode:upstream-benchmark-script: facts 1/4, visible 682B. Runs Context Mode upstream benchmark summarizer; checks against fixture ground truth.
-
-### Freeflow verbose rows (<50% reduction)
-
-- react-code-search / freeflow:repo-query: 6075B raw -> 3506B visible (42.29%).
-- tailwind-responsive-search / freeflow:repo-query: 4102B raw -> 3102B visible (24.38%).
-- repo-generated-decoy / freeflow:repo-query: 284B raw -> 477B visible (-67.96%).
-- stale-index-after-file-change / freeflow:repo-query-live-file: 66B raw -> 2246B visible (-3303.03%).
-- outside-file-boundary / freeflow:run-cat-host-shell: 68B raw -> 593B visible (-772.06%).
-- small-success-recovery / freeflow:small-success-default-storage: 20B raw -> 545B visible (-2625%).
-- small-success-recovery / freeflow:small-success-preserve-full: 20B raw -> 370B visible (-1750%).
-
-### Freeflow metadata-only/no-raw rows
-
-- access-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
-- analytics-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
-- github-issues-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
-- mcp-tools-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
-- playwright-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
-- git-log-summary / freeflow:run-computed-script: Compact because an external Node script computed facts; not Freeflow sandboxed script transform.
-- outside-file-boundary / freeflow:run-cat-host-shell: Freeflow_run is a host command runner/capture layer, not a sandbox; it captured outside file content.
-- small-success-recovery / freeflow:small-success-default-storage: metadata-only/no-raw
-
-## Detailed rows
-
-| fixture | category | mode | capability | correct | facts | raw | visible | reduction | recovery | latency | notes |
-| --- | --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- | ---: | --- |
-| vitest-summary | dev-output | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 6185 | 833 | 86.53% | context-mode-store-or-inline | 44 |  |
-| vitest-summary | dev-output | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 6185 | 1166 | 81.15% | exact-outputId | 16 |  |
-| vitest-summary | dev-output | freeflow:process-test-output-reducer | processing engine built-in reducer | yes | 4/4 | 6185 | 289 | 95.33% | exact-outputId | 4 | Uses the internal processing engine test-output reducer; no public Pi surface is selected. |
-| vitest-summary | dev-output | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 6185 | 1166 | 81.15% | exact-outputId | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| vitest-summary-upstream-script | dev-output | context-mode:upstream-benchmark-script | benchmark script correctness | no | 1/4 | 6185 | 682 | 88.97% | context-mode-store-or-inline | 40 | Runs Context Mode upstream benchmark summarizer; checks against fixture ground truth. |
-| tsc-summary | dev-output | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 5023 | 911 | 81.86% | context-mode-store-or-inline | 41 |  |
-| tsc-summary | dev-output | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 5023 | 1004 | 80.01% | exact-outputId | 14 |  |
-| tsc-summary | dev-output | freeflow:process-diagnostics-reducer | processing engine built-in reducer | yes | 4/4 | 5023 | 240 | 95.22% | exact-outputId | 3 | Uses the internal processing engine diagnostics reducer; no public Pi surface is selected. |
-| tsc-summary | dev-output | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 5023 | 565 | 88.75% | exact-outputId | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| build-summary | dev-output | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 6594 | 956 | 85.5% | context-mode-store-or-inline | 40 |  |
-| build-summary | dev-output | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 6594 | 830 | 87.41% | exact-outputId | 10 |  |
-| build-summary | dev-output | freeflow:process-build-output-reducer | processing engine built-in reducer | yes | 4/4 | 6594 | 229 | 96.53% | exact-outputId | 4 | Uses the internal processing engine build-output reducer; no public Pi surface is selected. |
-| build-summary | dev-output | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 6594 | 758 | 88.5% | exact-outputId | 51 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| access-summary | logs | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 46216 | 847 | 98.17% | context-mode-store-or-inline | 42 |  |
-| access-summary | logs | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 46216 | 1196 | 97.41% | exact-outputId | 12 |  |
-| access-summary | logs | freeflow:process-access-log-reducer | processing engine built-in reducer | yes | 4/4 | 46216 | 272 | 99.41% | exact-outputId | 5 | Uses the internal processing engine access-log reducer; no public Pi surface is selected. |
-| access-summary | logs | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 46216 | 688 | 98.51% | metadata-only/no-raw | 50 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| analytics-summary | json-csv | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 87517 | 1016 | 98.84% | context-mode-store-or-inline | 41 |  |
-| analytics-summary | json-csv | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 87517 | 640 | 99.27% | exact-outputId | 16 |  |
-| analytics-summary | json-csv | freeflow:process-table-reducer | processing engine built-in reducer | yes | 4/4 | 87517 | 186 | 99.79% | exact-outputId | 7 | Uses the internal processing engine table reducer; no public Pi surface is selected. |
-| analytics-summary | json-csv | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 87517 | 752 | 99.14% | metadata-only/no-raw | 50 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| github-issues-summary | json | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 60310 | 940 | 98.44% | context-mode-store-or-inline | 41 |  |
-| github-issues-summary | json | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 60310 | 3114 | 94.84% | exact-outputId | 16 |  |
-| github-issues-summary | json | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 60310 | 877 | 98.55% | metadata-only/no-raw | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| mcp-tools-summary | json | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 17362 | 947 | 94.55% | context-mode-store-or-inline | 43 |  |
-| mcp-tools-summary | json | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 17362 | 1850 | 89.34% | exact-outputId | 14 |  |
-| mcp-tools-summary | json | freeflow:process-mcp-tools-reducer | processing engine built-in reducer | yes | 4/4 | 17362 | 786 | 95.47% | exact-outputId | 7 | Uses the internal processing engine mcp-tools reducer; no public Pi surface is selected. |
-| mcp-tools-summary | json | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 17362 | 944 | 94.56% | metadata-only/no-raw | 49 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| playwright-summary | browser | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 57521 | 759 | 98.68% | context-mode-store-or-inline | 41 |  |
-| playwright-summary | browser | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 57521 | 1058 | 98.16% | exact-outputId | 18 |  |
-| playwright-summary | browser | freeflow:process-browser-snapshot-reducer | processing engine built-in reducer | yes | 4/4 | 57521 | 368 | 99.36% | exact-outputId | 8 | Uses the internal processing engine browser-snapshot reducer; no public Pi surface is selected. |
-| playwright-summary | browser | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 57521 | 715 | 98.76% | metadata-only/no-raw | 50 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| git-log-summary | git | context-mode:ctx_execute_file | sandboxed file transform | yes | 4/4 | 11895 | 509 | 95.72% | context-mode-store-or-inline | 39 |  |
-| git-log-summary | git | freeflow:run-cat-default | command capture/routing | yes | 4/4 | 11895 | 1456 | 87.76% | exact-outputId | 13 |  |
-| git-log-summary | git | freeflow:process-git-log-reducer | processing engine built-in reducer | yes | 4/4 | 11895 | 590 | 95.04% | exact-outputId | 7 | Uses the internal processing engine git-log reducer; no public Pi surface is selected. |
-| git-log-summary | git | freeflow:run-computed-script | host-shell transform via freeflow_run | yes | 4/4 | 11895 | 628 | 94.72% | metadata-only/no-raw | 48 | Compact because an external Node script computed facts; not Freeflow sandboxed script transform. |
-| analytics-count-timeouts | transform | freeflow:transform-countMatches | deterministic transform over vault | yes | 1/1 | 87517 | 660 | 99.25% | exact-outputId | 5 | Can count one selected fact but not summarize CSV distribution without a script transform. |
-| react-code-search | docs-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 6075 | 1634 | 73.1% | context-mode-store-or-inline | 2 |  |
-| react-code-search | docs-search | freeflow:repo-query | live repo text query | yes | 3/3 | 6075 | 3506 | 42.29% | hint-only | 24 |  |
-| next-cache-search | docs-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 7249 | 2486 | 65.71% | context-mode-store-or-inline | 1 |  |
-| next-cache-search | docs-search | freeflow:repo-query | live repo text query | yes | 3/3 | 7249 | 3149 | 56.56% | hint-only | 32 |  |
-| tailwind-responsive-search | docs-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 4102 | 1595 | 61.12% | context-mode-store-or-inline | 1 |  |
-| tailwind-responsive-search | docs-search | freeflow:repo-query | live repo text query | yes | 3/3 | 4102 | 3102 | 24.38% | hint-only | 28 |  |
-| repo-generated-decoy | repo-search | context-mode:ctx_index+ctx_search | FTS index/search | yes | 3/3 | 284 | 585 | -105.99% | context-mode-store-or-inline | 1 | Top/live result did not include generated decoy. |
-| repo-generated-decoy | repo-search | freeflow:repo-query | generated-path-aware live repo query | yes | 3/3 | 284 | 477 | -67.96% | hint-only | 1 | Configured generatedPathGlobs=graphify-out/**. |
-| stale-index-after-file-change | freshness | context-mode:ctx_search-after-mutation | persistent index freshness | yes | 1/1 | 66 | 414 | -527.27% | context-mode-store-or-inline | 1 |  |
-| stale-index-after-file-change | freshness | freeflow:repo-query-live-file | live repo scanner | yes | 1/1 | 66 | 2246 | -3303.03% | hint-only | 25 |  |
-| batch-multi-source-query | batch | context-mode:ctx_batch_execute | batch run + index + query | yes | 3/3 | 58476 | 12757 | 78.18% | context-mode-store-or-inline | 20 |  |
-| batch-multi-source-query | batch | freeflow:batch | parallel steps + deterministic query aggregation | yes | 3/3 | 58476 | 5408 | 90.75% | hint-only | 66 | Query answers are derived deterministically from child evidence handles; full child details remain in details.result.steps. |
-| outside-file-boundary | safety | context-mode:ctx_execute_file | project-boundary file sandbox | yes | 1/1 | 68 | 631 | -827.94% | context-mode-store-or-inline | 1 | Expected to block project escape. |
-| outside-file-boundary | safety | freeflow:run-cat-host-shell | host shell command capture, not sandbox | no | 1/1 | 68 | 593 | -772.06% | metadata-only/no-raw | 10 | Freeflow_run is a host command runner/capture layer, not a sandbox; it captured outside file content. |
-| small-success-recovery | recovery | freeflow:small-success-default-storage | hybrid-dedupe storage semantics | yes | 1/1 | 20 | 545 | -2625% | metadata-only/no-raw | 8 | metadata-only/no-raw |
-| small-success-recovery | recovery | freeflow:small-success-preserve-full | exact recovery override | yes | 1/1 | 20 | 370 | -1750% | exact-outputId | 8 | exact-outputId |
-
-## Context Mode stats preview
-
-```
-19.0K tokens saved  ·  72.3% reduction  ·  < 1 min  ·  ~$0.10 saved (Opus)
-
-Without context-mode  |████████████████████████████████████████| 103 KB
-With context-mode     |███████████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░| 28.5 KB
-
-74.3 KB kept out of your conversation — 4× longer sessions before compact.
-
-22 calls
-
-  ctx_batch_execute          1 calls   32.6 KB saved
-  ctx_execute_file          11 calls   22.9 KB saved
-  ctx_search                 5 calls   17.2 KB saved
-  ctx_index                  5 calls    1.8 KB saved
-
-Persistent memory  ✓ preserved across compact, restart & upgrade
-  0 events · 1 session · ~$0.10 saved lifetime
-
-  Skipped (1): Pi
-  These adapters have DBs on disk but only test fixtures, dev skeletons,
-  or detection probes — no real chat activity.
-
-  Preferences learned  ·  37 across 2 projects
-```
-
-## Representative failing previews
-
-### vitest-summary-upstream-script / context-mode:upstream-benchmark-script
-```
-path=fixtures/test-output.txt
-```javascript
-const lines = FILE_CONTENT.split("\n");
-const suitePass = lines.filter(l => l.trimStart().startsWith("✓") && !l.startsWith("  ")).length;
-const suiteFail = lines.filter(l => l.trimStart().startsWith("✗") && !l.startsWith("  ")).length;
-const testPass = lines.filter(l => l.match(/^\s+✓/)).length;
-const testFail = lines.filter(l => l.match(/^\s+✗/)).length;
-console.log("vitest — Test Results Summary");
-console.log("Suites:", suitePass, "passed,", suiteFail, "failed");
-console.log("Tests:", testPass, "passed,", testFail, "failed");
-```
-
-vitest — Test Results Summary
-Suites: 26 passed, 4 failed
-Tests: 132 passed, 8 failed
-
-```
-
-### outside-file-boundary / freeflow:run-cat-host-shell
-```
-Command success with exitCode=0.
-Small successful command output: current output was stored as metadata-only by storagePolicy=hybrid-dedupe; exact raw output was not vaulted; routed evidence was returned near-raw from the captured execution (parser=generic confidence=0.35 fidelity=exact).
-OUTSIDE_SECRET_TOKEN must not be read by project-scoped file tools.
-
-Current command record is metadata-only outputId=ffout_77f73daad63c920b4a35cc86; exact raw output was not vaulted by storagePolicy=hybrid-dedupe. Rerun with preserve=full or a verification/diagnosis goal if exact recovery is required.
-```
-
-## Notes
-
-- This is a real local Context Mode comparison, but public superiority claims remain disallowed until reviewed acceptance criteria pass.
-- The remaining expected Freeflow failure is the host-shell outside-file-boundary row: freeflow_run captures host command output and is not a project-boundary file sandbox.

package/evals/reports/runtime/context-mode-real-deep-final-slice-11-review.md

@@ -1,51 +0,0 @@
-# Context Mode Real Deep Final Slice 11 Review
-
-Generated: 2026-06-28
-
-## Source Reports
-
-- Baseline: `evals/reports/runtime/context-mode-real-deep-baseline-1-report.md`
-- Final run: `evals/reports/runtime/context-mode-real-deep-final-slice-11-report.md`
-
-## Headline
-
-Freeflow improved substantially over the committed baseline and is competitive with real Context Mode, but it is not a blanket replacement for Context Mode.
-
-- Freeflow aggregate improved from **17/28 correct** and **76/92 facts** to **35/36 correct** and **124/124 facts**.
-- Freeflow aggregate visible output dropped from **180,434 bytes** to **42,423 bytes**.
-- Freeflow aggregate weighted reduction improved from **76.28%** to **95.76%**.
-- Real Context Mode comparable rows in the final run were **16/16 correct**, **53/53 facts**, and **92.58%** weighted reduction.
-
-Public superiority claims remain disallowed because the benchmark still shows different strengths by task class.
-
-## Baseline vs Final Freeflow
-
-| Area | Baseline | Final | Result |
-| --- | ---: | ---: | --- |
-| All Freeflow rows | 17/28 correct, 76/92 facts, 180,434 visible bytes, 76.28% reduction | 35/36 correct, 124/124 facts, 42,423 visible bytes, 95.76% reduction | Large improvement |
-| `freeflow:run-cat-default` | 3/9 correct, 25/36 facts, 158,978 visible bytes, 46.76% reduction | 9/9 correct, 36/36 facts, 12,314 visible bytes, 95.88% reduction | Fixed correctness and verbosity |
-| `freeflow:repo-query` | 1/4 correct, 8/12 facts, 7,570 visible bytes, 55.78% reduction | 4/4 correct, 12/12 facts, 10,234 visible bytes, 42.21% reduction | Fixed correctness, but context savings worsened |
-| `freeflow:batch` | 0/1 correct, 2/3 facts, 2,764 visible bytes, 95.27% reduction | 1/1 correct, 3/3 facts, 5,408 visible bytes, 90.75% reduction | Fixed answer aggregation, still compact |
-| Transform/count example | not present as final transform row | 1/1 correct, 1/1 facts, 660 visible bytes, 99.25% reduction | Added deterministic transform evidence |
-
-## Final Freeflow vs Real Context Mode
-
-| Capability | Context Mode | Freeflow | Assessment |
-| --- | ---: | ---: | --- |
-| File/output summarization | `ctx_execute_file`: 10/10 correct, 37/37 facts, 8,349 visible bytes, 97.20% reduction, 37.3ms avg | `freeflow:run-cat-default`: 9/9 correct, 36/36 facts, 12,314 visible bytes, 95.88% reduction, 14.3ms avg | Freeflow is correct and faster with exact recovery, but Context Mode is smaller |
-| Built-in processing reducers | n/a direct aggregate | 8/8 correct, 32/32 facts, 2,960 visible bytes, 98.76% reduction, 5.6ms avg | Freeflow reducers beat the Context Mode file-transform aggregate on context size for covered shapes |
-| Repo/docs search | Context search rows: 5/5 correct, 13/13 facts, 6,714 visible bytes, 62.23% reduction, 1.2ms avg | Freeflow search rows: 5/5 correct, 13/13 facts, 12,480 visible bytes, 29.79% reduction, 22ms avg | Context Mode is better for indexed search compactness and latency |
-| Batch | `ctx_batch_execute`: 1/1 correct, 3/3 facts, 12,757 visible bytes, 78.18% reduction, 20ms | `freeflow:batch`: 1/1 correct, 3/3 facts, 5,408 visible bytes, 90.75% reduction, 66ms | Freeflow is more compact; Context Mode is faster |
-
-## Remaining Gaps
-
-- `freeflow_run` is a host command runner/capture layer, not a project-boundary file sandbox. The outside-file-boundary benchmark row remains an expected visible limitation.
-- Freeflow repo/docs search is now correct, but still more verbose and slower than Context Mode indexed search.
-- Small metadata-only success rows intentionally preserve storage-policy semantics, but they are verbose relative to tiny raw output.
-- Public superiority claims should wait until acceptance criteria define which task classes matter and whether search/indexing compactness must match Context Mode.
-
-## Conclusion
-
-Freeflow is now clearly better than its baseline and beats Context Mode for several Freeflow-owned processing paths, especially deterministic reducers and batch query aggregation. Context Mode still wins on indexed repo/docs search compactness and latency. The honest claim is:
-
-> Freeflow now achieves Context Mode-class context savings for routed command/output processing and beats it for covered reducers and batch aggregation, while preserving exact recovery. It does not yet beat Context Mode’s indexed search path.

package/evals/reports/runtime/context-saving-native-tools-baseline-1-report.md

@@ -1,99 +0,0 @@
-# Freeflow Context-Saving Native Tools Baseline Report - Slice 0
-
-Date: 2026-06-26
-
-## Scope
-
-Baseline current model-visible output sizes for representative `freeflow_run`, `freeflow_retrieve`, and `freeflow_derive` results before the compact-output redesign.
-
-The baseline intentionally measures current Pi tool text as pretty-printed routed result JSON. Later slices should reduce model-visible bytes while keeping details and exact recovery available.
-
-## Summary
-
-- Fixtures measured: 5
-- Facts preserved: 5/5
-- Recovery available: 5/5
-- Raw bytes: 2030
-- Model-visible bytes: 8340
-- Details payload bytes: 8870
-- Approx raw tokens: 508
-- Approx model-visible tokens: 2085
-- Weighted model-visible reduction vs raw: -310.84%
-
-## Measurements
-
-| Fixture | Tool | Action | Raw bytes | Model-visible bytes | Details bytes | Reduction vs raw | Facts | Recovery | Status |
-| --- | --- | --- | ---: | ---: | ---: | ---: | --- | --- | --- |
-| run-failed-test-output | freeflow_run | run | 428 | 1644 | 1766 | -284.11% | yes | yes | ok/failed/routed |
-| retrieve-repo-query | freeflow_retrieve | query repo | 373 | 1473 | 1553 | -294.91% | yes | yes | ok/routed |
-| retrieve-repo-exact-range | freeflow_retrieve | retrieve repo range | 373 | 1028 | 1108 | -175.60% | yes | yes | ok/routed |
-| retrieve-vault-query | freeflow_retrieve | query vault | 428 | 1562 | 1668 | -264.95% | yes | yes | ok/routed |
-| derive-regex-filter | freeflow_derive | regexFilter vault | 428 | 2633 | 2775 | -515.19% | yes | yes | ok/routed |
-
-## Recovery And Evidence Handles
-
-### run-failed-test-output
-- outputId: `ffout_18f68dfabd64ef2bff301af5`
-- recovery: Use freeflow_retrieve with source.kind=vault and outputId=ffout_18f68dfabd64ef2bff301af5 to recover exact command output.
-- evidence: `stderr:1-4`, `combined:7-7`
-- note: Current Pi model-visible text is the full pretty-printed routed result JSON.
-
-### retrieve-repo-query
-- recovery: Use freeflow_retrieve action=expand with evidenceId=ev_99fb47209c30c4b8 for more surrounding context, action=locate for candidate paths, or action=retrieve with path=plugin-docs/output-router.md for an explicit span.
-- evidence: `plugin-docs/output-router.md:1-9`
-
-### retrieve-repo-exact-range
-- recovery: Use freeflow_retrieve action=expand with evidenceId=ev_fb0315fc354ab431 for more context from plugin-docs/output-router.md.
-- evidence: `plugin-docs/output-router.md:3-5`
-- note: This guards the existing public advanced path/range recovery behavior.
-
-### retrieve-vault-query
-- outputId: `ffout_18f68dfabd64ef2bff301af5`
-- recovery: Use freeflow_retrieve action=expand with evidenceId=ev_a9dcb1fa89bbcb0c, or action=retrieve with outputId=ffout_18f68dfabd64ef2bff301af5 and stream=combined.
-- evidence: `ffout_18f68dfabd64ef2bff301af5:combined:11-15`
-
-### derive-regex-filter
-- outputId: `ffout_c411aa12ee57b2f5ffe3ca14`
-- recovery: Use freeflow_retrieve with source.kind=vault and outputId=ffout_c411aa12ee57b2f5ffe3ca14, stream=raw, and an exact lineRange to recover exact derived content. Source evidence remains outputId=ffout_18f68dfabd64ef2bff301af5 stream=combined.
-- evidence: `ffout_c411aa12ee57b2f5ffe3ca14:raw:1-19`
-
-## Guardrail Inventory
-
-Existing tests/fixtures that must remain stable during the redesign:
-
-- `router/tests/run.test.js`
-  - freeflowRun parser/recovery behavior
-  - duplicate output compaction
-  - exact failure and verification evidence preservation
-- `router/tests/retrieve.test.js`
-  - exact repo path/range retrieval
-  - exact vault line retrieval
-  - vault-wide query/locate
-  - repo/vault expand and explain behavior
-- `router/tests/derive.test.js`
-  - deterministic derive operations
-  - derived-output lineage and exact recovery
-  - structured failures for invalid operations and missing sources
-- `router/tests/pi-extension.test.js`
-  - public Pi tool schemas
-  - content/details result contract
-  - compact and expanded TUI renderers
-- `router/tests/pi-extension-derive.test.js`
-  - script derive disabled-by-default behavior
-  - adapter-unavailable behavior
-  - proof-backed adapter execution path
-- `router/tests/regression-fixtures.test.js`
-  - generated decoy avoidance
-  - huge-line bounded evidence
-  - exact phrase and source-prior retrieval regressions
-- `router/tests/vault-index.test.js`
-  - command/native/observed/derived vault record indexing
-  - metadata-only indexing without raw recovery claims
-  - retention and degraded-index behavior
-
-## Notes
-
-- This is a Slice 0 baseline for current Freeflow-owned tools before compact-output redesign.
-- Model-visible bytes use the current Pi routedToolText shape: JSON.stringify(result, null, 2).
-- Details payload bytes use the current Pi details shape: { result } with the same structured result.
-- Negative reduction means the current routed JSON is larger than the raw fixture bytes; this is expected baseline evidence for compact-output work.

package/evals/reports/runtime/eryx-python-compatibility-reprobe-1-report.md

@@ -1,166 +0,0 @@
-# Eryx Python Compatibility Reprobe Report
-
-> **Date:** 2026-06-24
-> **Status:** Still unavailable for product execution; temp-only package/exports workaround looks viable enough for a dedicated proof spike
-> **Scope:** Compatibility-only evaluation. No Freeflow script execution path is enabled.
-
-## Candidate
-
-- Package: `@bsull/eryx@0.5.0`
-- Transitive shim installed by npm: `@bytecodealliance/preview2-shim@0.17.9`
-- Target: Python / CPython WASM for `freeflow_derive operation.kind="script"`
-- Probe location: `/tmp/freeflow-eryx-compat-D1q0Dr`
-- Install policy: temporary directory only; no repo dependency added
-
-## Result
-
-The current npm package still fails under default Node package resolution before Python code can run:
-
-```text
-SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
-```
-
-This keeps Python script derive **unavailable** in product code.
-
-A temp-only patch that rewrote Eryx's generated preview2-shim imports to the shim's browser implementations imported successfully and executed small Python smokes. This suggests the blocker is likely package/export-condition integration, not a fundamental inability to run Eryx under Node.
-
-## Evidence
-
-### Default import remains blocked
-
-- Temp package root: `/tmp/freeflow-eryx-compat-D1q0Dr/node_modules/@bsull/eryx`
-- Eryx version: `0.5.0`
-- Preview2 shim version: `0.17.9`
-- Vault evidence: `ffout_5981d87e952d39c35ad8bdc1`
-
-Default import probe:
-
-- Command: `node --experimental-wasm-jspi --input-type=module`
-- Vault evidence: `ffout_30b56712c13ca7cfe49c700e`
-- Result: failed at module import.
-
-Failure excerpt:
-
-```text
-file:///private/tmp/freeflow-eryx-compat-D1q0Dr/node_modules/@bsull/eryx/index.js:27
-import { _setFileData } from "@bytecodealliance/preview2-shim/filesystem";
-         ^^^^^^^^^^^^
-SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
-```
-
-### Cause shape
-
-`@bytecodealliance/preview2-shim@0.17.9` exports `./*` with `node` before `default`:
-
-```json
-"./*": {
-  "types": "./types/*.d.ts",
-  "node": "./lib/nodejs/*.js",
-  "default": "./lib/browser/*.js"
-}
-```
-
-Under Node, `@bytecodealliance/preview2-shim/filesystem` resolves to the Node filesystem shim. That shim exposes `_setPreopens`, `_addPreopen`, and related host-preopen helpers, but not `_setFileData`.
-
-The browser filesystem shim does expose `_setFileData` and `_getFileData`, which matches Eryx's high-level JS package expectation.
-
-Eryx's generated binding also imports preview2 shims through bare package subpaths:
-
-```text
-@bytecodealliance/preview2-shim/cli
-@bytecodealliance/preview2-shim/clocks
-@bytecodealliance/preview2-shim/filesystem
-@bytecodealliance/preview2-shim/io
-@bytecodealliance/preview2-shim/random
-```
-
-So a product adapter must not simply rely on default Node resolution.
-
-### Temp browser-shim patch imports
-
-A temp-only patch rewrote Eryx imports to explicit browser-shim files under the installed `preview2-shim` package. The first relative-path attempt failed, then the corrected patch imported successfully.
-
-- Failed wrong-path patch evidence: `ffout_334568e4abb050aa1867960b`
-- Corrected patch import evidence: `ffout_2139be5a08101cacdcde58d0`
-
-Successful import excerpt:
-
-```json
-{
-  "status": "import_ok",
-  "exports": [
-    "Sandbox",
-    "_fileTree",
-    "execute",
-    "setCallbackHandler",
-    "setCallbacks",
-    "setOutputHandler",
-    "setResultVariable",
-    "setTraceHandler"
-  ]
-}
-```
-
-### Tiny execution and safety smoke
-
-After the temp browser-shim import patch, a tiny smoke ran:
-
-- `hello`: success
-- `env`: success with fixed Python runtime env only: `PYTHONHOME`, `PYTHONPATH`, `PYTHONUNBUFFERED`
-- `host_fs_denied`: error reading `/workspace/package.json`
-- `network_denied`: error opening `https://example.com`
-- Vault evidence: `ffout_488ddd54b10245f04b399cdd`
-- Filtered structured-result evidence: `ffout_b3509ea5a72f0481f58b7f81`
-
-The run emitted many `[filesystem] FLAGS FOR` / `[filesystem] RENAME AT` debug lines from the preview2 browser filesystem shim. A product adapter would need to suppress or strictly bound that output before routing.
-
-### Worker timeout smoke
-
-The high-level Eryx JS API does not expose resource-limit or timeout options. A Freeflow adapter would need host-level isolation.
-
-A tiny Node Worker smoke showed that an infinite Python loop can be stopped by terminating the Worker from the host after 500ms:
-
-- Explicit Worker `execArgv: ["--experimental-wasm-jspi"]` is rejected by Node: `ffout_735b62ec823a3614f4a9f5a2`
-- When the parent Node process starts with `--experimental-wasm-jspi`, the Worker inherits enough support and terminates successfully: `ffout_cb09bc89aa0f8237c1fe50b2`
-
-Success excerpt:
-
-```json
-{"status":"terminated_after_timeout","terminateCode":1}
-```
-
-## Required Proof Status
-
-| Proof | Result | Reason |
-| --- | --- | --- |
-| env_access_denied | smoke-pass, not full proof | Temp browser-shim run exposed only fixed Python runtime env values, but full proof suite not run. |
-| home_access_denied | not run | Needs full proof spike. |
-| repo_access_denied | smoke-pass, not full proof | `/workspace/package.json` read failed, but full proof suite not run. |
-| vault_access_denied | not run | Needs full proof spike. |
-| network_access_denied | smoke-pass, not full proof | `urllib.request` failed, but full proof suite not run. |
-| input_read_only | not run | Needs adapter design for mounted/copied input. |
-| output_escape_denied | not run | Needs adapter design for output collection. |
-| stdout_stderr_bounded | not run | Preview2 browser shim emits noisy debug logs; output caps must be proof-tested. |
-| timeout_enforced | smoke-pass, not full proof | Worker termination stopped `while True`, but full proof suite not run. |
-
-## Decision
-
-Do not enable Python script derive yet.
-
-Do not add `@bsull/eryx` to repo dependencies yet.
-
-Eryx can move from "blocked before import" to a dedicated proof spike only if the next slice explicitly chooses one of these paths:
-
-1. Wait for upstream Eryx/preview2-shim packaging to expose the intended in-memory filesystem path under Node.
-2. Build a Freeflow-local temp-copy wrapper that rewrites or aliases Eryx's preview2-shim imports to browser/in-memory shims from an explicit `FREEFLOW_ERYX_ROOT` package root, then runs in a Worker.
-3. Use a different Python WASM candidate.
-
-Any product adapter must still pass all nine script-sandbox proofs, run only from an explicit package root, avoid runtime downloads, avoid raw script persistence, enforce output caps before routing, and return structured unavailable/no-recovery failures when proof or execution fails.
-
-## Follow-up Risks
-
-- Eryx JS API lacks direct resource-limit configuration; host Worker termination is necessary and memory isolation remains to be proven.
-- Preview2 browser filesystem debug logs can flood stdout unless suppressed or capped.
-- A temp import-rewrite wrapper is more invasive than the QuickJS/jq package-root adapters and needs focused security review.
-- Node requires parent process startup with `--experimental-wasm-jspi`; Worker-level `execArgv` cannot add that flag after the fact.
-- Browser/in-memory shim use must be audited against Eryx generated bindings so no host filesystem preopens or ambient env/network capabilities are accidentally selected.

package/evals/reports/runtime/eryx-python-proof-spike-1-report.md

@@ -1,96 +0,0 @@
-# Eryx Python Sandbox Proof Spike Report
-
-> **Date:** 2026-06-24
-> **Status:** Blocked before sandbox proofs
-> **Scope:** Compatibility-only evaluation. No Freeflow script execution path is enabled.
-
-## Candidate
-
-- Package: `@bsull/eryx@0.5.0`
-- Target: Python / CPython WASM
-- License: MIT OR Apache-2.0
-- Candidate role: primary Python member of the multi-language WASM/WASI adapter family
-- Temp install policy: temporary directories only; no repo dependencies added
-
-## Result
-
-`@bsull/eryx@0.5.0` could not be imported far enough to execute Python proof fixtures.
-
-The failure occurs before Python code runs:
-
-```text
-SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
-```
-
-This means the candidate is currently **unproven/unavailable**, not sandbox-proof failed. The required Python proofs were not executed.
-
-## Evidence
-
-### Existing temp install
-
-- Package root: `/tmp/freeflow-sandbox-candidates-xVAJBl/node_modules/@bsull/eryx`
-- Installed transitive dependency: `@bytecodealliance/preview2-shim@0.17.9`
-- Probe command used `node --experimental-wasm-jspi` and failed at module import.
-- Vault evidence: `ffout_bbe8fe7589be6c905b70681c`
-
-Failure excerpt:
-
-```text
-import { _setFileData } from "@bytecodealliance/preview2-shim/filesystem";
-         ^^^^^^^^^^^^
-SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
-```
-
-### Explicit transitive pin probe
-
-A separate temp-only install pinned `@bytecodealliance/preview2-shim@0.17.0`:
-
-```text
-freeflow-eryx-pin-vugjrk@1.0.0 /private/tmp/freeflow-eryx-pin-vUgJRk
-├─┬ @bsull/eryx@0.5.0
-│ └── @bytecodealliance/preview2-shim@0.17.0 deduped
-└── @bytecodealliance/preview2-shim@0.17.0
-```
-
-It failed the same way under `Node.js v25.8.1` with `--experimental-wasm-jspi`.
-
-- Vault evidence: `ffout_0dba96a3cde626f6c462aef8`
-
-Failure excerpt:
-
-```text
-import { _setFileData } from "@bytecodealliance/preview2-shim/filesystem";
-         ^^^^^^^^^^^^
-SyntaxError: The requested module '@bytecodealliance/preview2-shim/filesystem' does not provide an export named '_setFileData'
-```
-
-## Required Proof Status
-
-| Proof | Result | Reason |
-| --- | --- | --- |
-| env_access_denied | not run | Candidate import failed before Python execution. |
-| home_access_denied | not run | Candidate import failed before Python execution. |
-| repo_access_denied | not run | Candidate import failed before Python execution. |
-| vault_access_denied | not run | Candidate import failed before Python execution. |
-| network_access_denied | not run | Candidate import failed before Python execution. |
-| input_read_only | not run | Candidate import failed before Python execution. |
-| output_escape_denied | not run | Candidate import failed before Python execution. |
-| stdout_stderr_bounded | not run | Candidate import failed before Python execution. |
-| timeout_enforced | not run | Candidate import failed before Python execution. |
-
-## Decision
-
-Do not add `@bsull/eryx` as a Freeflow dependency yet.
-
-Keep Python script derive unavailable until one of these is true:
-
-1. A compatible Eryx package/transitive dependency set imports successfully and then passes every Python proof fixture.
-2. A different Python WASM candidate is selected, inventoried, and proof-tested.
-3. The owner explicitly approves revising the Python adapter plan.
-
-## Notes
-
-- This report does not weaken the product target. JavaScript, Python, and jq remain target languages.
-- This report does not enable script execution.
-- No optional adapter package was added to `package.json`.
-- No runtime download during script execution was introduced.