pi-jev-lens 0.1.0

package/STATUS.md

@@ -0,0 +1,391 @@
+# STATUS
+
+Written overnight 2026-09-18. Everything below was verified by running it; numbers are in the tables further down and are regenerated by `node --import tsx eval/report.ts`.
+
+## What works
+
+- **Extension loads and runs in pi 0.84.3** with `pi -e ./index.ts`, in TUI, `-p` and `--mode json`. Verified headless with `openai-codex/gpt-5.6-luna`.
+- **jev classification** via `@typesafe-ai/sdk` 0.6.0, model `jev-1.13.0`: one request per tool result, three Noul questions (needed / outcome only / durable), 400 to 800 ms, about 400 input tokens each. Runs asynchronously right after the assistant's next message, so it is off the critical path; the `context` hook only waits (bounded, 2.5 s) for stragglers.
+- **Decisions are made once and frozen**: persisted with `pi.appendEntry`, rebuilt on `session_start`, so `/resume`, `/fork` and `/reload` reproduce the same prompt. Unit tests cover the freeze, the never-remove-a-tool-result invariant, and ledger rebuild.
+- **Three seal modes** (`JEV_LENS_MODE`): `rolling`, `batch`, `budget` (default). See "What I learned" for why budget is the default.
+- **Memory file**: durable user/agent statements (and, rarely, tool pointers) go to `<project>/.pi/jev-lens.md`, deduplicated and capped; a snapshot is appended to the system prompt at session start only, so it never disturbs the cache mid-session.
+- **Eval harness**: `eval/replay.ts` (offline, no coding-model calls), `eval/generate.ts` (pi headless on a fixture repo with hidden tests), `eval/report.ts` (comparison table). Fixture: 9 tasks, one of them a five-part "compound" task that produces 18 to 20 LLM calls.
+- `/jev-lens`, `/jev-lens decisions`, `/jev-lens file` commands; footer status; JSON-lines log in `<project>/.pi/jev-lens.log`.
+
+## What I learned (this changes the design)
+
+1. **With a 10× cache discount, pruning old context is almost never a token-cost win on its own.** Keeping a 5k-token tool result cached costs about 500 token-equivalents per call. Pruning it rewrites the whole tail after it once at full price. Break-even is roughly `9 × tail / pruned` calls. In the offline replay of four baseline sessions (55 calls, 81 tool results), rolling mode saved 7.6 % of input tokens but only 1.7 % of simulated cost, because every prune broke the prefix. The live runs agree: rolling mode lowered the measured cache-hit rate on several short tasks.
+   The real payoff of pruning is **context budget**: smaller prompts, later compaction, less distraction. So the policy should spend cache rewrites deliberately, which is what `budget` mode does: hold decisions, apply them all at once when they remove at least half of the tail they would rewrite (or when the cache is cold anyway, or at compaction).
+2. **jev's decisions look right most of the time.** Across the runs it forgot repeated `npm test` outputs, `CHANGELOG.md`, an exploratory `ls`/`find`, and a `read` of an unrelated module; it kept source files the agent was editing and the design doc while writing ARCHITECTURE.md. Probabilities are well spread (needed from 0.11 to 0.70), which makes thresholds meaningful.
+3. **The failure mode to watch is "forgot the thing the agent will need in two turns."** One rolling-mode run (money-rounding) forgot `tests/parse.test.js` (P(needed)=0.16) right after the agent had read it and moved on to write a different test file; the next `npm test` failed on exactly those tests and the agent declared itself done without fixing them. Baseline fixed them. One sample, but it is the shape of the risk: the "what did the agent do next" evidence is only one turn deep. Mitigations available now: lower `JEV_LENS_FORGET_BELOW` (0.25 → 0.15), or `budget` mode, which delays application so the agent usually has finished with the item before it disappears.
+4. **Short sessions cannot show a benefit.** Eight tasks × 5 to 16 calls × 2 to 20k tokens of context: nothing to prune. Only the compound task (18 to 20 calls) gets into the regime where it matters, and even that is short compared to a real afternoon in pi. The replay harness on real long sessions is the right instrument; there are only four tiny local sessions on this machine right now.
+
+## What does not work or was not done
+
+- No pre-send compression: a tool result is always sent verbatim at least once. Compressing before first send needs a different judgment ("does the agent need this at all?") and is where the largest savings would be.
+- Only tool results are pruned. Old assistant text and thinking are untouched.
+- The durable memory notes are still thin. User and agent text rarely cross the 0.7 threshold on these synthetic tasks, so the memory file is mostly empty in the eval. It works mechanically (unit-tested, and one note landed in the validation run) but has not been exercised with real conversations.
+- Codex prompt caching is best-effort (no `prompt_cache_key`, routing by prefix hash), so per-run cache-hit numbers are noisy; compare sums, not single runs.
+- The eval scored two things unfairly at first and I rescored them: the csv-export hidden test assumed a fixed formatMoney, and the architecture-doc task forbids code changes so its score must ignore the pre-planted failing tests. `results.jsonl` marks rescored rows. One compound failure (jev, run 3) is a spec ambiguity: the agent wrote an async `loadConfig` and the hidden test expects sync.
+
+## Numbers
+
+Conditions: `baseline` = no extension, `jev` = rolling mode, `jev-budget` = budget mode (fraction 0.5, min 1000 tokens). Model `openai-codex/gpt-5.6-luna`, thinking low, one run per short task, three per compound, two per marathon. Cache numbers are the provider's own `cached_tokens`. Generated by `node --import tsx eval/report.ts`.
+
+### Live runs, per condition
+
+| condition | runs | passed | calls | uncached input | cached input | cache hit | output | wall (s) | pruned tokens | final prompt (mean) |
+|---|---|---|---|---|---|---|---|---|---|---|
+| baseline | 13 | 13 | 186 | 333.7k | 785.4k | 70% | 22.8k | 709 | 0.0k | 8.6k |
+| jev | 13 | 11 | 188 | 409.1k | 596.0k | 59% | 23.8k | 780 | 15.7k | 6.1k |
+| jev-budget | 13 | 12 | 181 | 361.8k | 661.0k | 65% | 23.3k | 763 | 3.4k | 6.3k |
+
+### Live runs, per task
+
+| task | baseline ok | jev ok | jev-budget ok | baseline calls | jev calls | jev-budget calls | baseline uncached | jev uncached | jev-budget uncached | baseline hit | jev hit | jev-budget hit |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| fix-date | 1/1 | 1/1 | 1/1 | 5 | 6 | 5 | 10.0k | 7.3k | 9.8k | 34% | 48% | 34% |
+| money-rounding | 1/1 | 0/1 | 1/1 | 10 | 11 | 10 | 16.0k | 19.4k | 17.5k | 47% | 27% | 38% |
+| csv-export | 1/1 | 1/1 | 1/1 | 10 | 9 | 15 | 18.4k | 16.5k | 28.9k | 59% | 54% | 58% |
+| rename-total | 1/1 | 1/1 | 1/1 | 11 | 16 | 11 | 21.0k | 22.3k | 17.6k | 52% | 67% | 57% |
+| add-perf | 1/1 | 1/1 | 1/1 | 9 | 11 | 10 | 14.4k | 18.4k | 13.4k | 51% | 50% | 60% |
+| validation | 1/1 | 1/1 | 1/1 | 10 | 9 | 9 | 20.5k | 10.3k | 16.3k | 45% | 67% | 48% |
+| architecture-doc | 1/1 | 1/1 | 1/1 | 6 | 6 | 4 | 30.3k | 52.3k | 25.6k | 64% | 36% | 45% |
+| config-loader | 1/1 | 1/1 | 1/1 | 9 | 11 | 10 | 16.7k | 21.7k | 20.8k | 54% | 38% | 44% |
+| compound | 3/3 | 2/3 | 3/3 | 57 | 55 | 55 | 74.4k | 97.9k | 84.9k | 72% | 61% | 67% |
+| marathon | 2/2 | 2/2 | 1/2 | 59 | 54 | 52 | 112.1k | 143.1k | 126.9k | 79% | 66% | 72% |
+
+
+Reading the tables:
+
+- **Quality.** baseline 13/13, budget 12/13, rolling 11/13. The three failures with the extension: one is the money-rounding early-forget described above (rolling), the other two (jev compound-3, jev-budget marathon-2) are the async-`loadConfig` spec ambiguity, which the prompt now rules out. No failure in budget mode traces to a prune (budget never applied a prune in the live marathons: pending prunes never reached half of the tail they would rewrite).
+- **Cache.** Rolling mode costs cache: 59 % hit versus 70 % baseline, and 23 % more uncached input tokens over the whole set. Budget mode stays near baseline (65 %).
+- **Prompt size.** Mean final prompt: 8.6k baseline, 6.1k rolling, 6.3k budget. That is the benefit pruning buys, and it grows with session length; the marathon runs ended near 27k tokens of prompt in baseline.
+
+### Offline replay of the two baseline marathon sessions (jev classifying, policy simulated)
+
+### rolling
+
+| session | calls | tool results | keep/trim/forget | input base | input pruned | saved | cache hit base | cache hit pruned | cost base | cost pruned | re-read after forget / base |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| marathon-1/| 28 | 38 | 8/0/12 | 148.9k | 124.2k | 16.6% | 86.9% | 76.1% | 32473 | 39118 | 0/0 |
+| marathon-2/| 31 | 41 | 5/1/16 | 213.7k | 167.8k | 21.5% | 90.0% | 80.8% | 40539 | 45815 | 0/0 |
+| **total** | 59 | 79 | 13/1/28 | 362.6k | 292.1k | 19.5% | 88.7% | 78.8% | 73012 | 84933 | 0/0 |
+
+Cost units = uncached tokens + 0.1 × cached tokens (simulated prefix cache). jev calls: 42, mean 391 ms.
+
+### budget
+
+| session | calls | tool results | keep/trim/forget | input base | input pruned | saved | cache hit base | cache hit pruned | cost base | cost pruned | re-read after forget / base |
+|---|---|---|---|---|---|---|---|---|---|---|---|
+| marathon-1/| 28 | 38 | 8/0/12 | 148.9k | 148.9k | 0.0% | 86.9% | 86.9% | 32473 | 32473 | 0/0 |
+| marathon-2/| 31 | 41 | 5/1/16 | 213.7k | 164.9k | 22.8% | 90.0% | 81.2% | 40539 | 44313 | 0/0 |
+| **total** | 59 | 79 | 13/1/28 | 362.6k | 313.8k | 13.5% | 88.7% | 83.9% | 73012 | 76785 | 0/0 |
+
+Cost units = uncached tokens + 0.1 × cached tokens (simulated prefix cache). jev calls: 42, mean 430 ms.
+
+
+The replay makes the trade-off explicit on long sessions: rolling removes 19 % of input tokens but, with a 10× cache discount, costs 17 % *more* (cost units 85k vs 73k) because every prune rewrites the prefix. Budget removes 14 % for +4 %. Under this pricing, pruning after first send is a context-budget tool, not a cost tool. The cost win has to come from not sending things at all (pre-send compression) or from pruning only at moments the cache is cold anyway.
+
+## Pre-send compression (added later the same night)
+
+**Approach.** Large tool results (over 1200 estimated tokens) are intercepted in pi's `tool_result` hook before they are stored or sent. Code builds candidate views that are strict subsets of the output with line numbers (`outline`, `focus`, `signals`, `sample`, `head_tail`). jev answers a Choice over the views plus a Noul "will the next step need the exact full text"; full wins when either says so. For code, a second jev step asks, per top-level block, whether the agent will need its body, and those bodies are put back (`relevant` view). The full text stays in the result's `details` (persisted, never sent) and a `recall(id, lines?, pattern?)` tool serves it back. Every recall is logged as the signal that a view was too small. Two jev calls, about 700 ms each, only for large results.
+
+**Why this is the right place.** Everything sent once is cached at 10 % for the rest of the session, so post-send pruning can only fight over that 10 % and has to rewrite the prefix to do it. Not sending is free forever and leaves the prefix alone.
+
+**Offline check** (`eval/presend-replay.ts`, all 14 baseline sessions, jev deciding, then compared with what the agent actually did next):
+
+| large results | compressed | tokens (all) | tokens sent | saved | views | edit-miss / results later edited | quote-miss |
+|---|---|---|---|---|---|---|---|
+| 28 | 18 | 79.4k | 31.0k | 60.9 % | full:10, relevant:10, sample:6, head_tail:2 | 0 / 0 | 0 |
+
+An edit-miss would be an edit of that file whose old text is not in the view; a quote-miss a 40+ character span in the next assistant message that only exists in the omitted part. There were none. Typical choices: `data/sample.csv` 5.6k → 135 tokens (sample), `src/i18n.js` 1.5k → 81 (outline), `docs/DESIGN.md` kept full (jev put 0.4 to 0.6 on full: a prose design doc read to write an architecture doc), `src/categories.js` full when the task touched categories, outline otherwise.
+
+**Live runs** (`jev-presend`: pre-send on, post-send in budget mode; same tasks and model as above):
+
+| task | runs | passed | uncached input / run | cached / run | cache hit | final prompt / run | cost units / run | recalls |
+|---|---|---|---|---|---|---|---|---|
+| marathon, baseline | 3 | 3 | 52.9k | 200.2k | 79 % | 17.7k | 72.9k | 0 |
+| marathon, jev-presend | 3 | 3 | 51.4k | 157.7k | 75 % | 14.7k | 67.1k | 0 |
+| compound, baseline | 3 | 3 | 24.8k | 63.3k | 72 % | 7.0k | 31.1k | 0 |
+| compound, jev-presend | 3 | 3 | 27.9k | 69.3k | 71 % | 7.5k | 34.8k | 0 |
+| architecture-doc, jev-presend | 1 | 1 | 21.8k | 37.4k | 63 % | 16.3k | | 1 |
+
+Cost units = uncached + 0.1 × cached, from the provider's own counts. On the marathon, where large files get read, pre-send cut cached tokens 21 %, the final prompt 17 % and cost 8 %, with the same 3/3 pass rate, cache hit within noise of baseline, and zero recalls. Compound reads no large files, so the two are equal within run-to-run noise. Across all seven pre-send runs: 12 of 19 large results compressed, one recall (the agent asked for the full `i18n.js` while writing ARCHITECTURE.md; it then got it and passed).
+
+**Verdict: viable.** It is the first variant in this project that lowers cost without touching quality or the cache. The 8 % is bounded by how much of these small synthetic sessions is large tool output; in real sessions with long files, logs and search results the share is larger.
+
+**What to extend.** Views for `grep`/`find` output, diffs and build logs; pre-send on long user pastes; per-tool thresholds learned from recall rates; and combining with budget-mode pruning, which then only has small results left to decide about.
+
+## Pre-send benchmark on real trajectories (added after the first night)
+
+Synthetic tasks read only a handful of large files, so I built a benchmark from real agent runs: 300 OpenHands
+trajectories from `nebius/SWE-rebench-openhands-trajectories` (Qwen3-Coder on SWE-rebench issues), converted to pi-style
+messages (`eval/bench/convert-openhands.ts`). They contain 19k tool results, 2070 of them large (≥ 1200 tokens, 7.1M
+tokens in total), overwhelmingly pytest runs and file views. Trajectories 0 to 199 are the training slice for
+autoresearch, 200 to 299 the holdout. Metrics (`eval/presend-score.ts`):
+
+- **saved%**: tokens not sent over all large results.
+- **edit-miss%**: large `read` results the agent edited before re-reading the file, where the edit's old text was not in the view.
+- **quote-miss%**: the next assistant message quotes a 40+ character line that only existed in the omitted part.
+- **ref-miss%**: the next two steps use an identifier that only existed in the omitted part and that the agent had not seen anywhere earlier in the session. The most honest harm proxy: it means the agent learned something from what we dropped.
+- objective = saved% − 5·edit-miss% − 2·quote-miss% − ref-miss%.
+
+New machinery built for this: tree-sitter outlines and block boundaries (`@vscode/tree-sitter-wasm`, JS/TS/Python/Go/Rust/Java/C/C++/C#/Ruby/PHP/Bash), method-level blocks inside large classes, a `testlog` view (failing tests with assertion and traceback, short summary, final counts), a `tree` view for directory listings, and line tidying that collapses decorative bars.
+
+### Holdout results (100 trajectories, 684 large results, 2.25M tokens)
+
+| variant | saved | edit-miss | quote-miss | ref-miss | objective | views |
+|---|---|---|---|---|---|---|
+| first version (regex outlines, no testlog/tree) | 55.9 % | 0/13 | 4 (0.7 %) | (not measured) | | signals 315, full 211, relevant 33, outline 30 |
+| + testlog view, method blocks | 67.0 % | 0/13 | 4 (0.6 %) | (not measured) | | signals 363, full 184, testlog 44, relevant 41 |
+| + tree view, default prompts | 69.3 % | 0/13 | 4 (0.6 %) | 27 (3.9 %) | 64.2 | signals 366, full 146, relevant 45, testlog 44, tree 31, outline 30 |
+| + autoresearch best prompt (round 1) | 68.8 % | 0/13 | 4 (0.6 %) | 26 (3.8 %) | 63.9 | signals 365, full 153, relevant 47, testlog 45, tree 31, outline 26 |
+| + test-id index in testlog, task terms in tree, stricter ref-miss | 65.1 % | 0/13 | 4 (0.6 %) | 12 (1.8 %) | 62.2 | signals 268, testlog 142, full 153, relevant 42, outline 30, tree 29 |
+| + code needs-full 0.35, focus→outline for code | 63.5 % | 0/13 | 4 (0.6 %) | 11 (1.6 %) | 60.7 | code savings fell to 18 % |
+| **current default** (code threshold back to 0.5, focus→outline kept, big assignments as blocks) | 64.7 % | 0/13 | 4 (0.6 %) | 12 (1.8 %) | 61.7 | signals 268, testlog 143, full 157, relevant 46, tree 29, outline 26 |
+| + autoresearch best prompt (round 2, on the current default) | 64.9 % | 0/13 | 4 (0.6 %) | 13 (1.9 %) | 61.8 | same view mix as the default |
+
+The last three rows trade about 4 points of savings for halving ref-miss: the `testlog` view now keeps a capped index of
+test ids (agents pick one to re-run), and `tree` keeps entries matching task terms. Under the objective's weights that
+looks like a small loss; I chose the safer default because a ref-miss is a real "the agent learned something from what we
+dropped", while the 4 points are cheap tokens. `testIds` is a view parameter, set it to 0 for the leaner variant.
+One run (v6) showed a single edit-miss caused by jev choosing `focus` for a code file with low confidence. I first
+blamed jev's run-to-run variance for that; a direct measurement (30 trajectories scored twice, 179 paired results)
+says otherwise: needs-full answers differ by 0.008 on average (max 0.06), P(full) by 0.016 (max 0.08), and 3.4 % of
+results flip between full and compressed. The flip in v6 came from a changed candidate set (new views shift the Choice
+distribution), not from noise. So "ask twice and average" is not worth a second call; the structural fix below is. The fix that costs nothing is structural: for code,
+`focus` is upgraded to `outline` (and then possibly `relevant`), which closed that case in the runs after.
+
+Per kind, current default: command 73.2 % saved (ref-miss 1.5 %), code 28.3 % (2.4 %), prose 53.3 % (0 %), listing 47.2 % (4.4 %).
+
+### Autoresearch
+
+`eval/bench/autoresearch.ts` runs a loop: a researcher model (gpt-5.6-luna via `pi -p`) reads `research/PROGRAM.md`,
+the current best variant, the prompt texts and the history, proposes one variant (prompt texts, thresholds, view
+parameters), the benchmark scores it on 50 training trajectories (about 20 s, 1.3k jev calls), and it is kept if the
+objective improves. Eight iterations: three kept. The winner rewrote the view-choice instruction to give per-kind
+guidance (signals for command output unless exact text is needed; outline/focus/relevant/full for code by purpose).
+Train objective 55.2 → 66.1. Discarded: stricter block expansion (−11), tighter focus context (−11), lower shrink gate (−10).
+Log: `research/log.jsonl`, best: `research/best.json`.
+
+Caveats: the training slice is small and the loop optimises saved% since the harm metrics stay near zero on train;
+the holdout row above is the real test, and there the round-1 winner did **not** transfer: 68.8 % vs 69.3 % for the
+default prompt, within noise. The reason is visible in the history: round 1 ran against the older view set, and its
+gain came from pushing command output towards `signals`; the `testlog` and `tree` views added meanwhile capture the same
+tokens by construction. Lesson: on this problem, new code-built views moved the number (55.9 → 69.3 % on holdout with
+zero edit-misses), prompt wording did not. Round 2 of the loop (new views, corrected ref-miss, 60 training trajectories, 8 iterations, one kept: train 63.8 → 68.5)
+is in `research/round2/`; on the holdout its winner scores 64.9 % / objective 61.8 against 64.7 % / 61.7 for the
+default, again within noise. Round 3 used **gpt-6-astra** as the researcher (100 training trajectories, 10 iterations, `research/round3/`). It
+behaved differently from luna: it left the prompts alone and swept view parameters (signals context, tail length,
+test-id count, the size gate for medium results), keeping two: `signalsCtx` 2 → 0 (train 64.5 → 67.4). On the holdout
+that variant scores 64.7 % / objective 61.6, identical to the default (64.7 % / 61.7). Three rounds, same lesson: with a 50 to 100 trajectory training slice (jev itself is nearly
+deterministic: 3.4 % of decisions flip between identical runs), gains of 3 to 10 points on train are sampling noise on holdout, whether the researcher edits prompt wording (luna) or view parameters (astra). The loop is still useful as
+a regression guard and for parameter sweeps (thresholds, `testIds`, context sizes), and two of its discards were
+informative: every variant that pushed code towards `focus` or `relevant` produced 25 % edit-miss on train. The researcher only touches text and numbers; the view builders are code
+and stay fixed within a loop.
+
+### Live check of the round-3 variant on gpt-5.6-luna
+
+`jev-presend-r3` = pre-send with `research/round3/best.json` loaded through `JEV_LENS_VARIANT`, compound and marathon × 3:
+
+| task | condition | passed | uncached / run | cached / run | hit | final prompt | cost units / run | compressed | recalls |
+|---|---|---|---|---|---|---|---|---|---|
+| marathon | baseline | 3/3 | 52.9k | 200.2k | 79 % | 17.7k | 72.9k | | 0 |
+| marathon | pre-send default | 3/3 | 51.4k | 157.7k | 75 % | 14.7k | 67.1k | 10/14 | 0 |
+| marathon | pre-send, round-3 variant | 3/3 | 71.0k | 162.1k | 70 % | 16.1k | 87.2k | 8/13 | 0 |
+| compound | baseline | 3/3 | 24.8k | 63.3k | 72 % | 7.0k | 31.1k | | 0 |
+| compound | pre-send default | 3/3 | 27.9k | 69.3k | 71 % | 7.5k | 34.8k | 0/0 | 0 |
+| compound | pre-send, round-3 variant | 2/3 | 26.2k | 38.2k | 59 % | 4.9k | 30.1k | 0/0 | 0 |
+
+Read with care: the compound failure is a Codex outage ("Our servers are currently overloaded", four calls, zero
+tokens), and the expensive marathon run had five calls with `cacheRead` 0 on an unchanged prefix during the same
+window, so the cost difference is provider noise, not the variant. Quality is unchanged (all real runs passed, zero
+recalls), and compound reads nothing large, so the variant cannot show there. Conclusion as on the benchmark: the
+round-3 variant is indistinguishable from the default; keep the default.
+
+### Round 4: Fable as the researcher (code changes allowed)
+
+I ran the loop by hand with the freedom the model-researchers did not have: new views, not just parameters. Each step
+was scored on train (0 to 99) and only adopted after holdout (200 to 299). Measurements first:
+
+- **jev variance is small** (30 trajectories scored twice): needs-full differs by 0.008 on average, P(full) by 0.016,
+  3.4 % of decisions flip. "Ask twice" was dropped as an idea.
+- **Where the remaining command tokens were**: test runs are 408 of 495 large bash outputs on train (1.7M of 2.0M
+  tokens). Of the 466k command tokens still sent on holdout, 35 % were pytest runs jev sent in full (needs-full 0.53 to
+  0.62 on truncated verbose runs), 35 % were `testlog` views averaging 1.1k tokens, bloated by pytest's column padding
+  (hundreds of spaces per test line) and by library-code frames in tracebacks (`site-packages`, `/opt/conda`).
+
+| step | holdout saved | edit-miss | quote-miss | ref-miss | objective | command | code |
+|---|---|---|---|---|---|---|---|
+| default before this round | 64.7 % | 0/13 | 0.6 % | 1.8 % | 61.7 | 73.2 % | 28.3 % |
+| testlog: collapse padding, drop library frames back to the previous boundary, `_ _ _` chain separators; `matches` view for grep; `log` view for repeated lines | 70.8 % | 0/15 | 0.4 % | 1.9 % | 68.0 | 81.3 % | 27.5 % |
+| + command needs-full threshold 0.65 (train: 0.65 and 0.80 equal) | **74.1 %** | 0/15 | 0.4 % | 1.9 % | **71.4** | 85.2 % | 27.5 % |
+| + code gate 0.6 so unsure code goes to outline + block expansion (train +0.2) | 73.5 % | 0/15 | 0.4 % | 1.9 % | 70.7 | 85.2 % | 25.1 % → discarded |
+
+The library-frame rule is the single biggest step in the whole project (+6 points on holdout, zero new misses): on the
+worst example a 30k-character pytest run went from a 12.3k-character view to 5.1k with every repo frame and assertion
+kept. The `matches` and `log` views barely fire on this dataset (2 and 4 of 689) but cost nothing.
+
+What is left: code is 128 of 689 results and 338k tokens, 82 of them sent in full although only 12 were later edited.
+jev's needs-full sits at 0.3 to 0.6 for those, which is honest: the OpenHands agent reads many files before deciding
+what to edit, and the benchmark cannot tell which read mattered. Routing the unsure ones to outline + expansion did not
+help on holdout. The next lever there is the second step itself: expand by *task relevance of the block* rather than
+"will it need the body", or send outline first and let `recall` fetch bodies on demand, which the live extension
+already supports and this benchmark cannot score.
+
+### Live check of the round-4 default on gpt-5.6-luna (marathon × 3)
+
+| condition | passed | uncached / run | cached / run | hit | final prompt | cost units / run | compressed | recalls |
+|---|---|---|---|---|---|---|---|---|
+| baseline | 3/3 | 52.9k | 200.2k | 79 % | 17.7k | 72.9k | | 0 |
+| pre-send, first version (64.7 % offline) | 3/3 | 51.4k | 157.7k | 75 % | 14.7k | 67.1k | 10/14 | 0 |
+| pre-send, round-4 default (74.1 % offline) | 3/3 | 41.3k | 148.3k | 78 % | 14.8k | **56.1k** | 9/13 | 0 |
+
+Cost per run −23 % against baseline (was −8 % with the first version), cache hit back at baseline level, same pass rate,
+zero recalls. The offline gains show up live in the same direction and roughly the same proportion.
+
+### Round 5: outline-first for code
+
+Hypothesis: code is never sent whole; the agent gets every declaration signature (now at any nesting depth, from
+tree-sitter) plus the bodies the second step expands, and `recall` for anything else. Scored on holdout with 15 later
+edits as the real test.
+
+| variant | saved | edit-miss | quote-miss | ref-miss | objective | code saved |
+|---|---|---|---|---|---|---|
+| round-4 default (gate) | 74.2 % | 0/15 | 0.6 % | 1.9 % | 71.2 | 29.4 % |
+| outline-first, expand > 0.5 | 78.5 % | 1/15 | 0.6 % | 2.3 % | 41.7 | 59.4 % |
+| outline-first, expand > 0.35 | 77.0 % | 0/15 | 0.6 % | 2.3 % | 73.5 | 48.0 % |
+| outline-first, expand > 0.5, short import header always kept | **78.6 %** | **0/15** | 0.6 % | 2.3 % | **75.2** | 58.8 % |
+| outline-first with a "task relevance" expansion question (train) | 79.5 % | 2/6 | | | −90 | discarded |
+
+The single edit-miss at 0.5 was an added `import`: the header block was not expanded. Keeping short import headers whole
+closed it at no cost. The "is this block what the task is about" phrasing for expansion failed 2 of 6 edits on train,
+so the "will it need the body" phrasing stays. Nested signatures in the outline cut code ref-miss from 6.3 % to 4.7 %.
+New default: `JEV_LENS_PRESEND_CODE_POLICY=outline`, expansion threshold 0.5.
+
+Live on gpt-5.6-luna (marathon × 3): 3/3 passed, 0 recalls, cost 60.3k units per run against 56.1k for the round-4
+default and 72.9k for baseline. Neutral within run-to-run noise: the fixture's source files are small (10 of 13 large
+results compressed either way), so the policy cannot show there; the benchmark's code bucket is where it matters.
+
+## Procedural graph prototype (`eval/action-graph.ts`)
+
+Following Lu et al., *Procedural Graphs*, I mined the 34 non-marathon runs into a graph of abstract actions (`read:src`, `edit:src`, `bash:test`, `write:test`, ...) with edge counts and success rates, then used jev as the guidance model at the 158 decision points of the 6 held-out marathon runs: given task, recent actions, the current node and its outgoing edges with statistics, choose the next procedure.
+
+| decision points | jev agrees with what the agent did | in passing runs | in failing runs | always-most-common-edge |
+|---|---|---|---|---|
+| 158 | 41 % | 41 % | 42 % | 49 % |
+
+Honest reading: as a next-step predictor with this crude state, jev is not better than edge frequency. Its disagreements are systematic and defensible (it wants `bash:test` where the agent wrote a test first, and `final_answer` where the agent kept editing), which is what a guidance model is for: flagging departures from the known-good path, not imitating the agent. The mined graph itself is already informative: after `edit:src`, passing runs go to `bash:test` 96 % of the time. The design for using it as deviation detection with a one-line nudge is in `docs/ideas.md`.
+
+## Live sessions with gpt-6-astra: bash file displays as code (2026-09-18, evening)
+
+Five real pi sessions in `jev-contract-comp` (138 LLM calls, model `gpt-6-astra`) showed jev almost idle: 41 large results, 6 compressed, 35 sent full; post-send 65 keep / 2 forget, one prune applied. The reason is the agent's habit, not the thresholds: 33 of the 41 large results were `bash`, and most of those were plain file displays (`cat src/semarb/{models,engine}.py`, `cat a.py; cat b.py; cat tests/*.py`, `sed -n '301,660p' f`). The extension typed every bash result as `command`, which only offers `signals`/`focus`/`head_tail`/`log`, so jev correctly answered "full" (0.6 to 0.93) to a question that had no code view in it.
+
+**Change.** `detectKind` now parses the shell command: when every segment is a display command (`cat`, `sed`, `head`, `tail`, `nl`, `bat`, optionally piped into `head`/`tail`/`sed`), brace groups expanded and globs kept, and the shown files are source (or all prose), the result is typed `code` (or `prose`) and gets the outline/relevant path, including tree-sitter blocks when all shown files share a language. The output must also look like code (3+ signature lines), so a `cat` that failed stays `command`. Anything mixed with `ls`, `find`, `uv run`, heredocs and so on stays `command`.
+
+**Offline replay of the five Astra sessions** (`eval/presend-replay.ts`, jev deciding):
+
+| | large | compressed | saved | edit-miss | quote-miss | ref-miss | kinds |
+|---|---|---|---|---|---|---|---|
+| before | 38 | 2 | 0.2 % | 0/1 | 0 | 0 | command 30, code 7, prose 1 |
+| after | 38 | 5 | 3.4 % | 0/1 | 0 | 2 | command 24, code 12, prose 2 |
+
+Five bash results became code; jev compressed three of them to `relevant` (3.8k → 3.3k, 9.8k → 6.4k, 5.0k → 3.5k tokens) and expanded the other two back to full. The two ref-misses are `__file__` and `SimpleNamespace`, Python names the agent did not learn from the omitted bodies, so the scorer overcounts them. The remaining 24 command results are heredoc Python, JSON dumps and mixed pipelines: the next view type to build.
+
+**Holdout benchmark** (`eval/bench/holdout-v13-bash-display.md`): 78.9 % saved, 0/15 edit-miss, 4 quote-miss, 16 ref-miss, objective 75.4, against 78.6 % / 75.2 for the previous default. No result changed kind: the OpenHands agent reads through its own file tool, so this is run-to-run noise and confirms no regression.
+
+## Exact lines for code, tidy for command output (2026-09-18, late evening)
+
+The safety commit (`2813739`) stopped shortening lines in every view except `testlog`, so that an `edit` whose oldText was copied from a view still matches the file. Measured on the holdout with the same (new) scorer, that cost 4.6 points: 78.0 % → 73.4 % saved, 92k more tokens sent, with no miss metric improving (edit-miss 0/15 both). Row by row, `signals` views of the same results tripled in size because pytest's `=====` bars and long lines were no longer collapsed; 37 results shifted from `signals` to `testlog` and 23 lost every reduced candidate.
+
+The split is now by content kind: views of code and prose keep retained lines exactly; views of command output, listings and data pass through `tidyLine`, since that output is never edited. Same scorer, same slice (`eval/bench/holdout-v14-notidy.md` and `holdout-v15-tidy-command-only.md`):
+
+| variant | saved | sent | edit-miss | quote-miss | ref-miss | objective |
+|---|---|---|---|---|---|---|
+| before the safety commit | 78.0 % | 496.6k | 0/15 | 3 | 16 | 74.8 |
+| safety commit (no tidy) | 73.4 % | 588.3k | 0/15 | 4 | 17 | 69.6 |
+| tidy for command/listing/data only | 78.2 % | 492.3k | 0/15 | 4 | 16 | 74.7 |
+
+The new scorer also checks bash file displays for edit-miss (with re-reads excluded). On the five Astra sessions it finds 1 edit-miss in 3 checked edits across all three variants: `cat semantic.py; cat cli.py; cat polymarket.py` compressed to `relevant`, and the agent then edited a block that was left out. One sample, but it is the exact risk of typing bash displays as code, and the earlier scorer could not see it. The two-turn expansion step is the place to fix it, not the parser.
+
+## Sections: jev picks the parts of command output (2026-09-18, night)
+
+**Why.** After bash file displays became code, 24 of the Astra sessions' large results were still sent full: heredoc Python output, JSON dumps, curl'ed docs, mixed pipelines. On the holdout the 25 command results still sent full were mostly `grep -A/-B` context and debug-script output. None of the existing views fit: `signals` keeps errors, `testlog` keeps failures, neither keeps "the match group the agent came for".
+
+**What was built.** `splitSections` (`src/views.ts`) carves command output into sections: grep context groups (`--` and file changes), the top-level keys of a JSON document, markdown headings, marker lines (`COMMAND:`, `URL:`, ALL-CAPS labels, `=====` bars, tracebacks) and blank-line paragraphs; small sections merge into their predecessor, unstructured output falls back to fixed chunks, at most 24 sections. A `sections` view (the first line of every section, numbered) is offered for command output. When it is chosen, the existing second jev step asks per section "will the agent need its contents" (own prompt texts, `sectionInstructions`) and puts the chosen sections back, giving `relevant`. Plain file displays that stayed `command` (mixed file types) never get section headers, since the agent may edit from them.
+
+Two policies: `gate` (jev's view choice stands, sections is one candidate) and `sections` (when jev picks full for command output but needs-full is under the command threshold, send sections and expand). Under `gate` jev chose sections for 2 of 43 Astra results, so `sections` is what makes it act.
+
+**The floor.** Under the raw `sections` policy the Astra replay saved 54 % but with 6 ref-misses, because jev's per-section probabilities on docs read for orientation are flat and low (median 0.24, `TypeSafeClient` learned from a dropped section) and nothing got expanded: 11.3k tokens became 249. That pattern is "cannot tell", not "nothing needed". `presendSectionFloor` sends full when no section reaches it. Offline simulation on runs that logged the probabilities (`eval/bench/holdout-v17-sections-floor0.json`, `expandProbs` per row):
+
+| floor | holdout saved | holdout ref-miss | Astra saved | Astra ref-miss |
+|---|---|---|---|---|
+| 0 | 79.9 % | 19 | 39.3 % | 3 |
+| 0.3 | 79.7 % | 19 | 35.0 % | 2 |
+| 0.5 | 78.4 % | 18 | 12.5 % | 2 |
+
+0.3 removes the docs case for 0.2 points on the holdout; 0.5 would give back most of the Astra gain. Default 0.3.
+
+**Autoresearch round 4** (`research/round4/`, researcher gpt-6-astra, 100 training trajectories, 8 iterations): its first proposal was the `sections` policy itself (train 76.35 → 76.98, kept); a wider command gate, finer chunks, a rewritten section question and a rewritten view description were discarded; `signalsCtx` 2 → 1 was kept (78.14). Prompt wording again moved nothing; a code-built view and one integer did.
+
+**Holdout** (same scorer, `eval/bench/holdout-v16..v19`):
+
+| variant | saved | sent | edit-miss | quote-miss | ref-miss | objective |
+|---|---|---|---|---|---|---|
+| gate (sections only as a candidate) | 78.6 % | 487.6k | 0/15 | 5 | 16 | 74.8 |
+| sections policy, floor 0 | 79.9 % | 456.7k | 0/15 | 4 | 19 | 76.2 |
+| sections policy, floor 0.3 | 79.3 % | 470.3k | 0/15 | 4 | 17 | 75.7 |
+| + signalsCtx 1 (autoresearch winner), **new default** | 81.7 % | 415.6k | 0/15 | 4 | 20 | 77.7 |
+
+The winner transfers this time (+2.0 on holdout against +1.2 on train), unlike rounds 1 to 3, at the cost of three more ref-misses (2.9 %): with less context, `signals` wins over `testlog` more often (330 vs 254 views). Astra sessions with the new defaults: 34 % saved on 53 large results (47 % on command output) against 6.5 % under gate, 0 quote-miss, 2 ref-miss (`__pycache__` and one real), and the one pre-existing edit-miss on the `cat semantic.py; cat cli.py; cat polymarket.py` read.
+
+**Left open.** The Astra ref-miss at floor 0.3 and the edit-miss both come from the expansion step keeping too little of code-like output; the expansion threshold (0.5) is the next knob, and a two-turn look-back (did the agent edit a file it read through bash) would make the scorer's edit-miss the training signal for it.
+
+## Expansion thresholds: sweep says leave them at 0.5 (2026-09-18, late night)
+
+The open item after the sections work was the second step's threshold: the Astra edit-miss came from a block jev scored 0.39 (`def proposals` in `semantic.py`, edited five messages later; the four other edited blocks scored 0.53 to 0.79), and the Astra ref-miss from a section under 0.5. Both thresholds were swept on the holdout and on the Astra replay (`eval/bench/holdout-v20-expand-*`):
+
+| threshold | holdout objective | holdout saved | holdout ref-miss | Astra saved | Astra edit-miss | Astra ref-miss |
+|---|---|---|---|---|---|---|
+| code 0.35 | 75.8 | 79.9 % | 20 | 28 % (code 6 %) | 1/9 | 2 |
+| code 0.5 (default) | 77.7 | 81.7 % | 20 | 35 % (code 13 %) | 1/9 | 2 |
+| code 0.65 | 78.5 | 83.1 % | 22 | 41 % (code 52 %) | 2/9 | 4 |
+| sections 0.35 | 77.0 | 80.9 % | 19 | 26 % | 1/9 | 1 |
+| sections 0.5 (default) | 77.7 | 81.7 % | 20 | 35 % | 1/9 | 2 |
+| sections 0.65 | 77.8 | 81.7 % | 19 | 35 % | 1/9 | 4 |
+
+Reading: the holdout objective prefers raising the code threshold, but the holdout has 15 editable code results and zero misses at every setting, so it cannot see the edit risk; the Astra replay can, and there 0.65 doubles the edit-misses. Lowering to 0.35 halves the code savings and still does not catch the 0.39 block, because jev's per-block numbers move by about ±0.2 between identical runs. A threshold is too blunt for the last miss; the next lever is structural (expand blocks referenced by an expanded block, or blocks named in the task), not numeric. Both stay at 0.5.
+
+The scorer already checks edits of files read through bash within the next 12 assistant messages, excluding re-reads, so the edit-miss is a usable training signal for that lever; what is missing is volume: one edit-miss in the Astra sessions and none on the holdout.
+
+## More data: 500 more trajectories, and the code policy flips (2026-09-19)
+
+The holdout had 15 editable code results and zero edit-misses at every setting, so it could not rank anything that touched code. `eval/bench/fetch.sh` now downloads 1300 trajectories (rows 0-199 train, 200-299 the old holdout, **300-799 the large holdout**: 3350 large results, 11.7M tokens, 46 editable; 800-1299 untouched reserve). Runs take about 20 minutes and 5k jev calls each; summaries in `eval/bench/large/`.
+
+**The large slice contradicts the small one.** The outline-first code policy, default since the first benchmark night on the strength of 0/13 edit-misses, misses 8 of 46 edits there:
+
+| variant (large slice) | saved | edit-miss | quote-miss | ref-miss | objective |
+|---|---|---|---|---|---|
+| outline-first, code expand 0.5 (old default) | 81.7 % | 8/46 (17.4 %) | 12 | 86 | −8.5 |
+| outline-first, code expand 0.65 | 82.9 % | 11/46 (23.9 %) | 12 | 95 | −40.2 |
+| outline-first, code expand 0.35 | 80.5 % | 4/46 (8.7 %) | 10 | 80 | 34.0 |
+| gate policy, code expand 0.5 | 78.3 % | 2/46 (4.3 %) | 10 | 76 | 53.7 |
+| outline-first + the two rules below | 81.4 % | 7/42 (16.7 %) | 12 | 87 | −5.3 |
+| gate + the two rules | 78.3 % | 2/42 (4.8 %) | 11 | 78 | 51.5 |
+| **gate + rules + edit results excluded from scoring (new default)** | 79.0 % | 2/26 (7.7 %) | 9 | 74 | 37.7 |
+
+What the eight misses were (`expandProbs` per row made this readable): one outline of a test file with no tree-sitter blocks, edited at the very next step; one result of the agent's own `edit` tool, whose echoed file it kept editing; six `relevant` views where the edited block scored 0.15 to 0.49 and the edit came 11 or 12 assistant messages later, with no re-read in between. No bash re-read either: the scorer's window is right, the agent really edited from what it had read. Two rules came out of the first two: the agent's own `edit`/`write` results are never reduced (`buildCandidates` returns only full, and the async builder no longer re-adds a tree-sitter outline), and outline-first requires at least two expandable blocks. They fix their two cases and nothing else; the six late edits are the outline policy itself, and no threshold reaches them: 0.35 halves code savings and still leaves four.
+
+So the code policy is now `gate` (jev's needs-full and full-mass gates decide; a chosen view still gets block expansion). It saves 3 points less on the large slice and 4 on the old holdout (77.8 % / objective 74.3 there, 0/7 edit-miss), and it is the first setting with a positive objective on the Astra sessions: 30.8 % saved on 60 large results, 0/9 edit-miss, 1 ref-miss, objective 29.1. Code from Astra's bash displays now goes full (0 % saved on 13 results); the 45 % on command output carries it.
+
+The excluded edit results also shrink the editable count from 46 to 26, which is why the last row's rate is higher than the row above it with the same two misses.
+
+**Lesson for the research loop.** Every conclusion about code views drawn from the 100-trajectory holdout was drawn from 13 to 15 editable results, and the one that mattered was wrong. Editable results are the scarce evidence; the large slice has 26 after excluding edit echoes, the reserve 800-1299 should have a similar number, and autoresearch should be scored on the large slice for any variant that touches code, even at 20 minutes per evaluation.
+
+## What to try next
+
+1. **Pre-send judgment**: built, see above. Next: let the autoresearch researcher write view builders (one per content kind, sandboxed, verified as strict line subsets) instead of only prompt text and thresholds; three rounds of the latter transferred nothing, every code-built view did. And a structural rule for the second step: expand blocks referenced by an expanded block or named in the task.
+2. **Two-turn evidence** before a forget: only forget once the agent has produced two later assistant messages without touching the item, or lower `JEV_LENS_FORGET_BELOW` to 0.15.
+3. **Run the replay harness on real, long pi sessions** from `~/.pi/agent/sessions` as they accumulate (the Astra sessions are the first); and score code-touching variants on the 300-799 slice, not the 100-trajectory holdout.
+4. Tune `JEV_LENS_BUDGET_FRACTION` (0.5 assumes about 18 more calls will follow; 0.25 assumes 36) or trigger on context percentage instead, so budget mode actually fires in hour-long sessions before compaction does.