@hicaru/pi-rlm 0.3.13 → 0.3.15

package/README.md

@@ -144,19 +144,35 @@ delegating only when their task genuinely decomposes further.
 
 ## Benchmarks
 
-Tested against `rlm-lab` prompt bake-off and full dual-mode RLM runtime benchmarks
-on `poolside/laguna-xs-2.1:free` (a free ~32B model):
-
-| Benchmark | Mode | Result |
-|-----------|------|--------|
-| Main orchestrator (7 scenarios) | prompt bake-off | **0.958** mean score (v3 fewshot arm) |
-| RLM worker (4 scenarios) | prompt bake-off | **0.94** mean score (v2 contract arm) |
-| Needle-in-haystack (3 needles) | classic RLM | **recall 1.0** |
-| CodeQA timeout | classic RLM | **correct** (~3.7k tokens) |
-| Coding (retry fix) | orchestrator | **correct** (file edited) |
-| Live smoke needle | classic RLM | **hit** (~5k tokens) |
-
-> On a *free* model. Frontier models do even better.
+E2E runs of the real engine against OpenRouter chat models — the lite suite
+(`needle` multi-needle recall, `codeqa` repo-QA, `coding` fix task; 7 tasks × 2 passes
+per model). Deterministic graders (recall / gold containment / regex), no LLM-as-judge.
+
+Latest results — small models (≤32B parameters, paid tier):
+
+| Model | Params | Score | Accuracy | Latency/task |
+|-------|--------|-------|----------|--------------|
+| `qwen/qwen3-30b-a3b-instruct-2507` | MoE 30B / 3B active | **14/14** | **100%** | ~15s |
+| `google/gemma-3-27b-it` | dense 27B | 12/14 | 86% | ~26s |
+| `mistralai/mistral-small-3.2-24b-instruct` | dense 24B | 12/14 | 86% | ~28s |
+
+Raw per-task rows (correct, recall, latency, tokens, cost) live in
+`bench/runs/bench-<ts>.jsonl` — one JSONL row per task, committed as history.
+
+### Run the benchmarks
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...        # required — env vars are the only key transport
+
+bun run bench                              # lite suite: needle + codeqa + coding
+bun run bench --suite needle --limit 1     # one suite, first task only
+bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
+bun run bench --list                       # print tasks, no engine / no key
+bun run bench --suite paper                # paper tier: s_niah, oolong, browsecomp, codeqa_lb (downloads datasets)
+```
+
+Suites: `all` (lite, default) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
+`oolong` · `browsecomp` · `codeqa_lb`.
 
 ## Security
 

package/README.ru.md

@@ -135,6 +135,38 @@ rm -rf ~/.pi/agent/extensions/rlm
 ## Логи запусков
 
 
+## Бенчмарки
+
+E2E-прогоны реального движка против моделей OpenRouter — lite-сьют (`needle` поиск игл в
+куче, `codeqa` вопросы по коду, `coding` задача-фикс; 7 задач × 2 прохода на модель).
+Детерминированные грейдеры (recall / вхождение эталона / regex), без LLM-судьи.
+
+Актуальные результаты — маленькие модели (≤32B параметров, платный тариф):
+
+| Модель | Параметры | Счёт | Точность | Latency/задача |
+|--------|-----------|------|----------|----------------|
+| `qwen/qwen3-30b-a3b-instruct-2507` | MoE 30B / 3B активных | **14/14** | **100%** | ~15s |
+| `google/gemma-3-27b-it` | dense 27B | 12/14 | 86% | ~26s |
+| `mistralai/mistral-small-3.2-24b-instruct` | dense 24B | 12/14 | 86% | ~28s |
+
+Построчные результаты (correct, recall, latency, токены, цена) — в
+`bench/runs/bench-<ts>.jsonl`, одна JSONL-строка на задачу.
+
+### Как запустить
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...        # обязателен — ключи ходят только через env
+
+bun run bench                              # lite-сьют: needle + codeqa + coding
+bun run bench --suite needle --limit 1     # один сьют, первая задача
+bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
+bun run bench --list                       # показать задачи, без движка и ключа
+bun run bench --suite paper                # paper-сьют: s_niah, oolong, browsecomp, codeqa_lb (скачивает датасеты)
+```
+
+Сьюты: `all` (lite, по умолчанию) · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
+`oolong` · `browsecomp` · `codeqa_lb`.
+
 ## Безопасность
 
 - **Изоляция ключей**: ключи провайдеров хранятся только в TypeScript (`AuthStorage`); песочница получает промпты и возвращает текст, но никогда не получает ключи.

package/README.zh-CN.md

@@ -157,6 +157,38 @@ rm -rf ~/.pi/agent/extensions/rlm
   连续错误上限。
 - **信任**：本地安装需要 Pi 项目信任。
 
+## 基准测试
+
+针对 OpenRouter 聊天模型对真实引擎进行端到端测试 —— 精简套件（`needle` 多针召回、
+`codeqa` 代码库问答、`coding` 修复任务；每个模型 7 个任务 × 2 轮）。确定性评分
+（召回率 / 标准答案包含 / 正则），无 LLM 评审。
+
+最新结果 —— 小型模型（≤32B 参数，付费额度）：
+
+| 模型 | 参数 | 得分 | 准确率 | 每任务延迟 |
+|------|------|------|--------|------------|
+| `qwen/qwen3-30b-a3b-instruct-2507` | MoE 30B / 3B 激活 | **14/14** | **100%** | ~15s |
+| `google/gemma-3-27b-it` | dense 27B | 12/14 | 86% | ~26s |
+| `mistralai/mistral-small-3.2-24b-instruct` | dense 24B | 12/14 | 86% | ~28s |
+
+逐任务原始数据（正确性、召回率、延迟、token、成本）位于 `bench/runs/bench-<ts>.jsonl`
+—— 每个任务一行 JSONL，作为历史记录提交。
+
+### 运行基准测试
+
+```bash
+export OPENROUTER_API_KEY=sk-or-...        # 必需 —— 密钥仅通过环境变量传递
+
+bun run bench                              # 精简套件：needle + codeqa + coding
+bun run bench --suite needle --limit 1     # 单个套件，仅第一个任务
+bun run bench --model openrouter/qwen/qwen3-30b-a3b-instruct-2507
+bun run bench --list                       # 仅列出任务，无需引擎和密钥
+bun run bench --suite paper                # paper 套件：s_niah, oolong, browsecomp, codeqa_lb（需下载数据集）
+```
+
+套件：`all`（精简版，默认） · `needle` · `codeqa` · `coding` · `paper` · `s_niah` ·
+`oolong` · `browsecomp` · `codeqa_lb`。
+
 ## 项目布局
 
 ```

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "@hicaru/pi-rlm",
-	"version": "0.3.13",
+	"version": "0.3.15",
 	"author": "hicaru",
 	"repository": {
 		"type": "git",
@@ -53,6 +53,6 @@
 	},
 	"dependencies": {
 		"@earendil-works/pi-agent-core": "^0.84.1",
-		"@firecrawl/anydoc": "^0.1.7"
+		"@firecrawl/anydoc": "^0.2.4"
 	}
 }

package/src/bridge/handlers/emitting.ts

@@ -56,9 +56,13 @@ export async function emitting<T>(
 
   let costUsd = 0;
   let tokens = 0;
+  let tokensIn = 0;
+  let tokensOut = 0;
   const track = (u: Usage): void => {
     costUsd += u.cost.total;
     tokens += u.totalTokens;
+    tokensIn += u.input;
+    tokensOut += u.output;
   };
 
   try {
@@ -70,6 +74,8 @@ export async function emitting<T>(
       resultPreview: summary.preview,
       costUsd,
       tokens,
+      tokensIn,
+      tokensOut,
       detail: summary.error,
       failedCount: summary.failed,
       totalCount: summary.total,
@@ -83,6 +89,8 @@ export async function emitting<T>(
       resultPreview: msg,
       costUsd,
       tokens,
+      tokensIn,
+      tokensOut,
       detail: msg,
     });
     throw err;

package/src/bridge/handlers/rlm-query.ts

@@ -206,6 +206,8 @@ async function childRun(
       id: subId,
       status: "done",
       resultPreview: res.answer.slice(0, 200),
+      tokensIn: res.inputTokens,
+      tokensOut: res.outputTokens,
     });
     return res;
   } catch (err: unknown) {

package/src/context/source-doc.ts

@@ -25,7 +25,8 @@ function skipReasonFromError(err: unknown): SkipReason {
     const code: unknown = err.code;
     // Preserve anydoc ConvertErrorCode values rather than collapsing to "convert-failed".
     if (code === "unsupported" || code === "malformed" || code === "encrypted"
-      || code === "resourceLimit" || code === "missingPart" || code === "io") {
+      || code === "resourceLimit" || code === "missingPart" || code === "io"
+      || code === "needsOcr" || code === "hosted") {
       return code;
     }
   }

package/src/context/types.ts

@@ -27,6 +27,10 @@ export type SkipReason =
   | "unsupported"
   | "malformed"
   | "encrypted"
+  /** v0.2.4: scanned/image-only PDF pages that need OCR (anydoc doesn't do OCR). */
+  | "needsOcr"
+  /** v0.2.4: `ocr: 'hosted'` could not reach Firecrawl Parse. */
+  | "hosted"
   | "resourceLimit"
   | "missingPart"
   | "io"

package/src/core/answer.ts

@@ -3,9 +3,12 @@
 import type { ReplResult } from "../sandbox/protocol.ts";
 import { formatReplStderr } from "../text/repl-output.ts";
 
-/** First non-null final answer across a turn's executed blocks, or null. */
+/** First non-blank final answer across a turn's executed blocks, or null. A blank capture counts
+ *  as absent (H2): an empty `answer.ready` flip must never terminate a run with "". */
 export function finalAnswerOf(results: readonly ReplResult[]): string | null {
-  for (const r of results) if (r.finalAnswer != null) return r.finalAnswer;
+  for (const r of results) {
+    if (r.finalAnswer != null && r.finalAnswer.trim() !== "") return r.finalAnswer;
+  }
   return null;
 }
 

package/src/core/budget.ts

@@ -107,8 +107,30 @@ export class TokenBudget {
 }
 
 /** Cap derivation (v5 `resolve_budget`): share × context window, clamped by the task cap. */
+/**
+ * Minimum context window (tokens) for the token-budget cascade to engage at all.
+ *
+ * The formula (window × budgetShare) assumes the window is large enough that a fraction of it
+ * is a meaningful working budget. Below this floor the derived cap shrinks below a task's FIXED
+ * overhead (system prompt + per-turn history re-send + sub-LLM calls) and strangles the run —
+ * a 32k window would cap a task at 8k tokens, less than the protocol scaffolding alone.
+ * So for smaller windows the rule does not apply: the budget is effectively unbounded and runs
+ * stay bounded by maxIterations / maxErrors / wall-clock instead.
+ */
+export const BUDGET_WINDOW_FLOOR = 250_000;
+
+/** An effective budget that can never trigger — the cascade "switched off" without changing
+ *  any call-site types (budget: TokenBudget | undefined). */
+function unboundedBudget(config: RlmConfig): TokenBudget {
+  return new TokenBudget(Number.MAX_SAFE_INTEGER, {
+    softFrac: config.budgetSoftFrac,
+    maxContinuations: config.budgetMaxContinuations,
+  });
+}
+
 export function resolveBudget(contextWindow: number | undefined, config: RlmConfig): TokenBudget {
   const ctx = contextWindow !== undefined && contextWindow > 0 ? contextWindow : 32_000;
+  if (ctx < BUDGET_WINDOW_FLOOR) return unboundedBudget(config);
   const shareCap = Math.floor(ctx * config.budgetShare);
   const cap = config.budgetTaskCap > 0 ? Math.min(shareCap, config.budgetTaskCap) : shareCap;
   return new TokenBudget(Math.max(cap, 1), {

package/src/core/engine.ts

@@ -20,12 +20,14 @@ import { TaskLedger, contextSig, taskKey } from "./ledger.ts";
 import { type MemoryStore, rootContextPaths } from "./memory.ts";
 import { type ChatMsg, modelComplete } from "../bridge/model.ts";
 import { buildRlmSystemPrompt } from "../prompts/system.ts";
-import { buildTurnPrompt, FINALIZE_PROMPT } from "../prompts/user.ts";
+import { buildTurnPrompt, FINALIZE_PROMPT, RETRIEVAL_NUDGE } from "../prompts/user.ts";
 import type { RlmEmitter } from "../tool/rlm-events.ts";
 import type { SubcallPhase } from "../tool/rlm-details.ts";
 import { PythonSandbox, SANDBOX_WATCHDOG_HEARTBEAT_MS } from "../sandbox/sandbox.ts";
+import type { ReplResult } from "../sandbox/protocol.ts";
 import { pinContext, type PinnedContext } from "../sandbox/context-file.ts";
 import { previewStdout, previewText } from "../text/preview.ts";
+import { findReplBlocks } from "../text/parsing.ts";
 import { contextLength, contextSizeStats, contextTypeLabel } from "../text/tokens.ts";
 import { finalAnswerOf, formatReplOutputs, latestAnswerContentOf, turnHadError } from "./answer.ts";
 import { compactHistory, elideOldToolPayloads, shouldCompact } from "./compaction.ts";
@@ -248,6 +250,9 @@ export function createEngine(deps: EngineDeps): RunRlm {
     // v5 budget cascade state: the wrap-up note fires for exactly ONE turn after crossing soft.
     let softFired = false;
     let softNoteTurn = -1;
+    // H3: retrieval-discipline coach — one-shot per run; children inherit it via the same loop.
+    let sawRetrieval = false;
+    let retrievalNudged = false;
 
     try {
       const meta = {
@@ -355,11 +360,15 @@ export function createEngine(deps: EngineDeps): RunRlm {
         // v5 [ledger] blackboard + [memory] notes — each silent ("") when it has nothing to say.
         const ledgerBlock = deps.config.enableLedger ? runLedger.injectBlock() : "";
         const memoryBlock = rootMemory !== undefined ? rootMemory.injectBlock(input.rootPrompt) : "";
+        // H3: after two retrieval-free turns, inject the coach nudge exactly once, for one turn.
+        const nudgeNow = i >= 2 && !sawRetrieval && !retrievalNudged;
+        if (nudgeNow) retrievalNudged = true;
         const notes =
           [
             i === softNoteTurn ? WRAP_UP_BUDGET : undefined,
             ledgerBlock === "" ? undefined : ledgerBlock,
             memoryBlock === "" ? undefined : memoryBlock,
+            nudgeNow ? RETRIEVAL_NUDGE : undefined,
           ]
             .filter((s): s is string => s !== undefined)
             .join("\n\n") || undefined;
@@ -379,6 +388,7 @@ export function createEngine(deps: EngineDeps): RunRlm {
           complete: deps.complete,
           onPhase: reportPhase,
         });
+        if (turn.blocks.some((b) => /\b(?:search|grep_context)\s*\(/.test(b))) sawRetrieval = true;
         const allBlocks = turn.blocks.length > 0
           ? turn.blocks.map((b) => previewText(b, 400)).join("\n")
           : previewText(turn.response, 400);
@@ -386,8 +396,17 @@ export function createEngine(deps: EngineDeps): RunRlm {
           emitter.emitSubcallUpdated({ id: selfReportId, args: `▶ ${allBlocks}`, resultPreview: previewStdout(turn.results) });
         }
         limits.addUsage(turn.usage);
-        if (selfReportId) emitter.emitSubcallUpdated({ id: selfReportId, costUsd: turn.usage.cost.total, tokens: turn.usage.totalTokens });
-        else emitter.emitRootUsage(turn.usage.cost.total, turn.usage.totalTokens);
+        if (selfReportId) {
+          emitter.emitSubcallUpdated({
+            id: selfReportId,
+            costUsd: turn.usage.cost.total,
+            tokens: turn.usage.totalTokens,
+            tokensIn: turn.usage.input,
+            tokensOut: turn.usage.output,
+          });
+        } else {
+          emitter.emitRootUsage(turn.usage.cost.total, turn.usage.totalTokens, turn.usage.input, turn.usage.output);
+        }
         deps.onUsage?.(turn.usage, "root");
         const answerContent = latestAnswerContentOf(turn.results);
         if (answerContent) best = answerContent;
@@ -458,7 +477,7 @@ export function createEngine(deps: EngineDeps): RunRlm {
         }
       }
       if (pendingReplOutputs) appendUserMessage(history, pendingReplOutputs);
-      const finalized = result(await finalize(history, model, deps, limits), deps.config.maxIterations, limits);
+      const finalized = result(await finalize(history, model, deps, limits, sandbox), deps.config.maxIterations, limits);
       persistRoot(finalized.answer);
       lastAnswer = finalized.answer;
       return finalized;
@@ -516,8 +535,16 @@ function contextWindowOrFallback(model: Model<Api>, registry: ModelContextRegist
   return registry.limitFor(`${model.provider}/${model.id}`);
 }
 
-/** Out of turns: ask the model for its best final answer (plain text). */
-async function finalize(history: ChatMsg[], model: Model<Api>, deps: EngineDeps, limits: LimitGuard): Promise<string> {
+/** Out of turns: ask the model for its best final answer. FINALIZE_PROMPT asks for a fenced
+ *  ```repl``` block, so execute it like any turn and prefer the captured answer (H2) — a raw
+ *  fence echoed verbatim must never become the run answer. Plain text stays the fallback. */
+async function finalize(
+  history: ChatMsg[],
+  model: Model<Api>,
+  deps: EngineDeps,
+  limits: LimitGuard,
+  sandbox: PythonSandbox,
+): Promise<string> {
   const finalHistory = [...history];
   appendUserMessage(finalHistory, FINALIZE_PROMPT);
   const complete = deps.complete ?? modelComplete;
@@ -528,5 +555,12 @@ async function finalize(history: ChatMsg[], model: Model<Api>, deps: EngineDeps,
     signal: deps.signal,
   });
   limits.addUsage(usage);
+  const blocks = findReplBlocks(text);
+  const results = new Array<ReplResult>(blocks.length);
+  for (let i = 0; i < blocks.length; i++) {
+    results[i] = await sandbox.exec(blocks[i]);
+  }
+  const final = finalAnswerOf(results) ?? latestAnswerContentOf(results);
+  if (final !== null && final.trim() !== "") return final.trim();
   return text.trim();
 }

package/src/prompts/user.ts

@@ -26,3 +26,11 @@ export const FINALIZE_PROMPT =
   "You are out of turns. Finalize NOW: set `answer[\"content\"]` and `answer[\"ready\"] = True` " +
     "(fenced ```repl```) with your best final answer from everything you have gathered. " +
     "Only if the REPL is unavailable, answer as plain text.";
+
+/** One-shot retrieval-discipline nudge (the engine owns the when — see core/engine.ts). The
+ *  context is external by design, so a model that never calls search()/grep_context() is
+ *  guessing from padding vocabulary; after two retrieval-free turns it gets this once. */
+export const RETRIEVAL_NUDGE =
+  "[coach] You have not inspected the external context yet — it is NOT included in this " +
+  "chat, and guessing is useless: the text is padding. On THIS turn, call search(\"...\") " +
+  "or grep_context(\"...\") inside a ```repl block before answering.";

package/src/sandbox/py/retrieval.py

@@ -36,6 +36,30 @@ def _chunk_text(text: str, chunk_chars: int) -> list[str]:
     return chunks
 
 
+def _snippet_window(text: str, terms: set[str]) -> str:
+    """Slice `text` around the earliest occurrence of any query term, capped at _SNIPPET_CHARS.
+
+    BM25 finds the right window chunk; the snippet must show the match, not the chunk head.
+    Clipped edges get "..." markers, which count toward the cap (the body is trimmed to fit).
+    Falls back to the chunk head when no term occurs (tokenize/camelCase mismatches).
+    """
+    lowered = text.lower()
+    hits = [p for p in (lowered.find(t) for t in terms) if p >= 0]
+    if not hits:
+        return text[:_SNIPPET_CHARS]
+    start = max(0, min(hits) - _SNIPPET_LEAD)
+    end = min(len(text), start + _SNIPPET_CHARS)
+    lead = "..." if start > 0 else ""
+    trail = "..." if end < len(text) else ""
+    body = text[start:end]
+    if len(body) > _SNIPPET_CHARS - len(lead) - len(trail):
+        body = body[:_SNIPPET_CHARS - len(lead) - len(trail)]
+    if start + len(body) < len(text):
+        trail = "..."  # trimming pulled the window edge back inside the chunk
+        body = body[:_SNIPPET_CHARS - len(lead) - len(trail)]
+    return lead + body + trail
+
+
 # ---- deterministic retrieval over `context` -----------------------------------------------
 #
 # The RLM paper's trajectories retrieve by having the root model hand-write regex over the
@@ -48,6 +72,7 @@ def _chunk_text(text: str, chunk_chars: int) -> list[str]:
 _INDEX_WINDOW_LINES = 40       # a window is the retrieval unit: big enough to carry meaning
 _INDEX_MAX_WINDOWS = 20_000    # ceiling so a huge add_context() cannot exhaust worker memory
 _SNIPPET_CHARS = 400
+_SNIPPET_LEAD = 100            # chars of lead-in kept before the earliest matched term
 _GREP_HARD_CAP = 200           # absolute ceiling on returned grep hits, whatever k asks for
 _BM25_K1 = 1.2
 _BM25_B = 0.75
@@ -165,9 +190,10 @@ class _Bm25Index:
             return []
         top = heapq.nlargest(k, scores.items(), key=lambda kv: kv[1])
         out: list[dict[str, Any]] = [None] * len(top)  # type: ignore[list-item]
+        term_set = set(terms)
         for i, (idx, score) in enumerate(top):
             text = self.texts[idx]
-            snip = text[:_SNIPPET_CHARS]
+            snip = _snippet_window(text, term_set)
             # Both `snippet` and `text` so agents never KeyError mixing search vs grep shapes.
             out[i] = {
                 "path": self.paths[idx],

package/src/sandbox/py/worker.py

@@ -22,6 +22,7 @@ awaited in a LATER exec than the one that started it.
 from __future__ import annotations
 
 import argparse
+import ast
 import io
 import json
 import os
@@ -61,7 +62,15 @@ class _AnswerDict(dict):
     def __setitem__(self, key, value):
         super().__setitem__(key, value)
         if key == "ready" and value:
-            self._on_ready(self.get("content", ""))
+            content = self.get("content", "")
+            # An empty ready-flip must not capture "": defer it — a later non-blank content
+            # assignment while ready stays True (branch below) is the real submission.
+            if str(content).strip():
+                self._on_ready(content)
+        elif key == "content" and self.get("ready"):
+            content = str(value)
+            if content.strip():
+                self._on_ready(content)
 
 
 
@@ -151,7 +160,7 @@ class Worker(WorkerScaffold):
             if isinstance(cur, dict):
                 for k, v in cur.items():
                     dict.__setitem__(ans, k, v)
-                if cur.get("ready") and self._final_answer is None:
+                if cur.get("ready") and self._final_answer is None and str(cur.get("content", "")).strip():
                     self._final_answer = str(cur.get("content", ""))
             ns["answer"] = ans
         # Single context variable (RLM paper: the context lives in the environment and
@@ -295,8 +304,25 @@ class Worker(WorkerScaffold):
 
     def _exec(self, code: str, ns: dict[str, Any]) -> None:
         t = self.exec_timeout_s
+        # Jupyter-style auto-echo: plain exec() discards a trailing bare expression's value
+        # (models write `search("...")` and never see the hits), so split the block: exec the
+        # head, eval the tail, and repr() the value when it is not None (None stays silent).
+        tree = ast.parse(code)
+        tail = tree.body[-1] if tree.body else None
+        tail_is_expr = isinstance(tail, ast.Expr)
+
+        def _run() -> None:
+            if tail_is_expr and tail is not None:
+                head = ast.Module(body=tree.body[:-1], type_ignores=[])
+                exec(compile(head, "<repl>", "exec"), ns, ns)  # noqa: S102
+                value = eval(compile(ast.Expression(tail.value), "<repl>", "eval"), ns, ns)  # noqa: S102
+                if value is not None:
+                    print(repr(value))
+            else:
+                exec(compile(tree, "<repl>", "exec"), ns, ns)  # noqa: S102
+
         if t <= 0 or not hasattr(signal, "SIGALRM"):
-            exec(compile(code, "<repl>", "exec"), ns, ns)  # noqa: S102
+            _run()
             return
 
         def _alarm(signum, frame):  # noqa: ARG001
@@ -305,7 +331,7 @@ class Worker(WorkerScaffold):
         old = signal.signal(signal.SIGALRM, _alarm)
         signal.setitimer(signal.ITIMER_REAL, t)
         try:
-            exec(compile(code, "<repl>", "exec"), ns, ns)  # noqa: S102
+            _run()
         finally:
             signal.setitimer(signal.ITIMER_REAL, 0)
             signal.signal(signal.SIGALRM, old)

package/src/text/parsing.ts

@@ -2,23 +2,44 @@
  * Parsing helpers: extract ```repl``` code blocks from a model response.
  *
  * The RLM root model emits Python wrapped in fenced blocks tagged `repl`. We extract those
- * blocks in order; everything else is prose the model uses to think out loud.
+ * blocks in order; everything else is prose the model uses to think out loud. Small instruct
+ * models often finalize inside ```python / ```py fences (or bare ones) instead; when a response
+ * contains no `repl` block at all, those code-ish fences are used as a fallback so the engine
+ * still executes their code. Other language tags (```text, ```json, ...) are never executed —
+ * the sandbox only runs Python.
  */
 
 const FENCE = /(`{3,})[ \t]*repl[ \t]*\r?\n([\s\S]*?)\1/g;
+// The info string is captured so a rejected tag (e.g. ```text) still consumes its whole fence —
+// otherwise that fence's own closing ``` could later match as a bare opener and swallow code.
+const FALLBACK_FENCE = /(`{3,})[ \t]*([^`\r\n]*)[ \t]*\r?\n([\s\S]*?)\1/g;
+const PYTHON_TAG = /^py(thon)?$/i;
 
-/** Return every ```repl``` block body, in document order. */
-export function findReplBlocks(text: string): string[] {
+/** Shared fence scan: run `re` over `text`, keep bodies the selector accepts (same trimming). */
+function collectFences(text: string, re: RegExp, select: (m: RegExpExecArray) => string | null): string[] {
   const blocks: string[] = [];
   let m: RegExpExecArray | null;
-  FENCE.lastIndex = 0;
-  while ((m = FENCE.exec(text)) !== null) {
-    const code = m[2] ?? "";
-    if (code.trim()) blocks.push(code.replace(/\s+$/, ""));
+  re.lastIndex = 0;
+  while ((m = re.exec(text)) !== null) {
+    const code = select(m);
+    if (code !== null && code.trim()) blocks.push(code.replace(/\s+$/, ""));
   }
   return blocks;
 }
 
+/**
+ * Return every ```repl``` block body, in document order. If the response has none, fall back to
+ * ```python / ```py / untagged fences — never other language tags, and never a mix of both kinds.
+ */
+export function findReplBlocks(text: string): string[] {
+  const repl = collectFences(text, FENCE, (m) => m[2] ?? "");
+  if (repl.length > 0) return repl;
+  return collectFences(text, FALLBACK_FENCE, (m) => {
+    const tag = m[2] ?? "";
+    return tag === "" || PYTHON_TAG.test(tag) ? (m[3] ?? "") : null;
+  });
+}
+
 /** Truncate REPL stdout for the model's context window (head + tail, with an elision note). */
 export function truncateOutput(text: string, limit = 20_000): string {
   if (text.length <= limit) return text;

package/src/tool/repl-tool.ts

@@ -267,6 +267,8 @@ export function createReplTool(deps: ReplToolDeps): ToolDefinition<typeof ReplTo
           return modelRef(m) ?? m.id;
         },
         rootTokens: () => store.getRootUsage().tokens,
+        rootTokensIn: () => store.getRootUsage().tokensIn,
+        rootTokensOut: () => store.getRootUsage().tokensOut,
       });
       let capturedStdout = "";
       let capturedStderr = "";

package/src/tool/rlm-aggregator.ts

@@ -55,7 +55,7 @@ export class RlmEventAggregator extends EmitterListener {
   }
 
   private handleRootUsage(event: RootUsageEvent): void {
-    this.store.addRootUsage(event.costUsd, event.tokens);
+    this.store.addRootUsage(event.costUsd, event.tokens, event.tokensIn, event.tokensOut);
     this.notify();
   }
 

package/src/tool/rlm-details.ts

@@ -33,6 +33,9 @@ export interface RlmSubcall {
   readonly endedAt?: number;
   readonly costUsd: number;
   readonly tokens: number;
+  /** In/out split (input / output) — mirrors tokens. */
+  readonly tokensIn: number;
+  readonly tokensOut: number;
   /** For batch subcalls: failed prompt count (partial failure). */
   readonly failedCount?: number;
   /** For batch subcalls: total prompt count. */
@@ -46,7 +49,7 @@ export interface RlmDetails {
   readonly rootPrompt: string;
   readonly turns: { readonly current: number; readonly max: number };
   readonly subcalls: readonly RlmSubcall[];
-  readonly totals: { readonly costUsd: number; readonly tokens: number };
+  readonly totals: { readonly costUsd: number; readonly tokens: number; readonly tokensIn: number; readonly tokensOut: number };
   readonly answer?: string;
 }
 

package/src/tool/rlm-events.ts

@@ -41,6 +41,9 @@ export interface SubcallUpdatedEvent {
   readonly costUsd?: number;
   /** Delta — additive on both the subcall and running totals. */
   readonly tokens?: number;
+  /** Deltas for the in/out split shown in the tree (input / output). Additive like tokens. */
+  readonly tokensIn?: number;
+  readonly tokensOut?: number;
   /** For batch subcalls: failed prompt count. */
   readonly failedCount?: number;
   /** For batch subcalls: total prompt count. */
@@ -55,6 +58,8 @@ export interface TurnEvent {
 export interface RootUsageEvent {
   readonly costUsd: number;
   readonly tokens: number;
+  readonly tokensIn?: number;
+  readonly tokensOut?: number;
 }
 
 export interface AnswerEvent {
@@ -114,8 +119,8 @@ export class RlmEmitter {
   }
 
   /** Accumulate usage directly to root-level totals. */
-  emitRootUsage(costUsd: number, tokens: number): void {
-    this.ee.emit("root-usage", { costUsd, tokens } satisfies RootUsageEvent);
+  emitRootUsage(costUsd: number, tokens: number, tokensIn?: number, tokensOut?: number): void {
+    this.ee.emit("root-usage", { costUsd, tokens, tokensIn, tokensOut } satisfies RootUsageEvent);
   }
 
   /** Set the final answer text (root-only). */

package/src/tool/rlm-tool.ts

@@ -53,7 +53,7 @@ export function createRlmTool(controller: RlmController, runRegistry?: RunRegist
         rootPrompt: "",
         turns: { current: 0, max: 0 },
         subcalls: [],
-        totals: { costUsd: 0, tokens: 0 },
+        totals: { costUsd: 0, tokens: 0, tokensIn: 0, tokensOut: 0 },
       }));
       if (!validation.ok) return validation.error;
       const params = validation.value;
@@ -79,6 +79,8 @@ export function createRlmTool(controller: RlmController, runRegistry?: RunRegist
           return m === undefined ? undefined : modelRef(m) ?? m.id;
         },
         rootTokens: () => aggregator.getRootUsage().tokens,
+        rootTokensIn: () => aggregator.getRootUsage().tokensIn,
+        rootTokensOut: () => aggregator.getRootUsage().tokensOut,
       });
 
       // Wire abort signal to controller

package/src/tool/subcall-render.ts

@@ -9,7 +9,7 @@
 import { Text } from "@earendil-works/pi-tui";
 import { keyText } from "@earendil-works/pi-coding-agent";
 import type { SubcallStatus } from "./rlm-details.ts";
-import { formatTokens, spinnerFrame } from "../ui/theme.ts";
+import { formatTokens, formatTokensSplit, spinnerFrame } from "../ui/theme.ts";
 import type { Theme } from "@earendil-works/pi-coding-agent";
 
 // ── Glyphs ──
@@ -25,15 +25,21 @@ export function headlineStatusGlyph(status: SubcallStatus | "aborted" | "done",
 
 // ── Stats formatting ──
 
-/** The `4.2k tok · 812ms` run of a card header. Omits any zero component. */
+/** The `4.2k tok · 812ms` run of a card header. In/out split when known; omits zero parts. */
 export function cardStatsLine(
-  totals: { readonly tokens: number },
+  totals: { readonly tokens: number; readonly tokensIn?: number; readonly tokensOut?: number },
   theme: Theme,
   extra?: string,
   backgroundPending?: number,
 ): string {
   const parts: string[] = [];
-  if (totals.tokens > 0) parts.push(`${formatTokens(totals.tokens)} tok`);
+  if (totals.tokens > 0) {
+    parts.push(
+      totals.tokensOut !== undefined && totals.tokensOut > 0
+        ? formatTokensSplit(totals.tokensIn ?? totals.tokens, totals.tokensOut)
+        : `${formatTokens(totals.tokens)} tok`,
+    );
+  }
   if (extra) parts.push(extra);
   const line = theme.fg("dim", parts.join(" · "));
   // The one thing no single line can show: spawned work that may outlive this block.

package/src/tool/subcall-store.ts

@@ -18,6 +18,9 @@ type MutableSubcall = {
 export interface SubcallTotals {
   readonly costUsd: number;
   readonly tokens: number;
+  /** In/out split (input / output) — mirrors tokens, shown separately in the tree. */
+  readonly tokensIn: number;
+  readonly tokensOut: number;
 }
 
 export class SubcallStore extends EmitterListener {
@@ -25,8 +28,12 @@ export class SubcallStore extends EmitterListener {
 
   private totalCostUsd = 0;
   private totalTokens = 0;
+  private totalTokensIn = 0;
+  private totalTokensOut = 0;
   private rootCostUsd = 0;
   private rootTokens = 0;
+  private rootTokensIn = 0;
+  private rootTokensOut = 0;
 
   constructor(emitter: RlmEmitter, private readonly onChange?: () => void) {
     super();
@@ -52,6 +59,8 @@ export class SubcallStore extends EmitterListener {
       startedAt: Date.now(),
       costUsd: 0,
       tokens: 0,
+      tokensIn: 0,
+      tokensOut: 0,
     });
   }
 
@@ -75,6 +84,14 @@ export class SubcallStore extends EmitterListener {
       sc.tokens += event.tokens;
       this.totalTokens += event.tokens;
     }
+    if (event.tokensIn !== undefined) {
+      sc.tokensIn += event.tokensIn;
+      this.totalTokensIn += event.tokensIn;
+    }
+    if (event.tokensOut !== undefined) {
+      sc.tokensOut += event.tokensOut;
+      this.totalTokensOut += event.tokensOut;
+    }
     if (event.failedCount !== undefined) sc.failedCount = event.failedCount;
     if (event.totalCount !== undefined) sc.totalCount = event.totalCount;
   }
@@ -87,8 +104,8 @@ export class SubcallStore extends EmitterListener {
   }
 
   /** Snapshot running totals. O(1). */
-  getTotals(): { readonly costUsd: number; readonly tokens: number } {
-    return { costUsd: this.totalCostUsd, tokens: this.totalTokens };
+  getTotals(): SubcallTotals {
+    return { costUsd: this.totalCostUsd, tokens: this.totalTokens, tokensIn: this.totalTokensIn, tokensOut: this.totalTokensOut };
   }
 
   /**
@@ -126,33 +143,43 @@ export class SubcallStore extends EmitterListener {
     const taken: RlmSubcall[] = [];
     let costUsd = 0;
     let tokens = 0;
+    let tokensIn = 0;
+    let tokensOut = 0;
     for (const root of children.get(undefined) ?? []) {
       const subtree = settledSubtree(root);
       if (subtree === undefined) continue;
       for (const node of subtree) {
         costUsd += node.costUsd;
         tokens += node.tokens;
+        tokensIn += node.tokensIn;
+        tokensOut += node.tokensOut;
         taken.push(Object.freeze({ ...node, status: node.status as SubcallStatus }));
         this.subcalls.delete(node.id);
       }
     }
     this.totalCostUsd -= costUsd;
     this.totalTokens -= tokens;
-    return { subcalls: taken, totals: { costUsd, tokens } };
+    this.totalTokensIn -= tokensIn;
+    this.totalTokensOut -= tokensOut;
+    return { subcalls: taken, totals: { costUsd, tokens, tokensIn, tokensOut } };
   }
 
   // ── Root usage (delegated from RlmEventAggregator) ──
 
   /** Accumulate root-level usage into shared totals. Called by aggregator. */
-  addRootUsage(costUsd: number, tokens: number): void {
+  addRootUsage(costUsd: number, tokens: number, tokensIn = 0, tokensOut = 0): void {
     this.totalCostUsd += costUsd;
     this.totalTokens += tokens;
+    this.totalTokensIn += tokensIn;
+    this.totalTokensOut += tokensOut;
     this.rootCostUsd += costUsd;
     this.rootTokens += tokens;
+    this.rootTokensIn += tokensIn;
+    this.rootTokensOut += tokensOut;
   }
 
   /** Root engine's OWN spend (driver-model turns only) — never blends sub-call models. */
   getRootUsage(): SubcallTotals {
-    return { costUsd: this.rootCostUsd, tokens: this.rootTokens };
+    return { costUsd: this.rootCostUsd, tokens: this.rootTokens, tokensIn: this.rootTokensIn, tokensOut: this.rootTokensOut };
   }
 }

package/src/ui/panel/run-registry.ts

@@ -29,6 +29,8 @@ export interface RunRegistration {
   readonly rootModel?: () => string | undefined;
   /** Root's OWN spend (driver-model turns) — never a subtree sum across models. */
   readonly rootTokens?: () => number;
+  readonly rootTokensIn?: () => number;
+  readonly rootTokensOut?: () => number;
   /** Persistent entries (background work) stay hidden until they hold subcalls. */
   readonly hideWhenEmpty?: boolean;
 }
@@ -44,6 +46,8 @@ export interface RunEntry {
   readonly turns: () => { readonly current: number; readonly max: number };
   readonly rootModel: () => string | undefined;
   readonly rootTokens: () => number;
+  readonly rootTokensIn: () => number;
+  readonly rootTokensOut: () => number;
   readonly hideWhenEmpty: boolean;
 }
 
@@ -66,6 +70,8 @@ export class RunRegistry {
       turns: run.turns ?? (() => DEFAULT_TURNS),
       rootModel: run.rootModel ?? (() => undefined),
       rootTokens: run.rootTokens ?? (() => 0),
+      rootTokensIn: run.rootTokensIn ?? (() => 0),
+      rootTokensOut: run.rootTokensOut ?? (() => 0),
       hideWhenEmpty: run.hideWhenEmpty ?? false,
     };
     this.entries.set(run.runId, entry);
@@ -119,6 +125,8 @@ export class RunRegistry {
         rootPhase: entry.rootPhase(),
         rootModel: entry.rootModel(),
         rootTokens: entry.rootTokens(),
+        rootTokensIn: entry.rootTokensIn(),
+        rootTokensOut: entry.rootTokensOut(),
         subcalls: entry.subcalls(),
       });
     }

package/src/ui/theme.ts

@@ -12,6 +12,12 @@ export function formatTokens(n: number): string {
   return String(n);
 }
 
+/** `190.2k↑ 18.6k↓ tok` — the in/out split. Falls back to plain when out is zero. */
+export function formatTokensSplit(tokensIn: number, tokensOut: number): string {
+  if (tokensOut > 0) return `${formatTokens(tokensIn)}↑ ${formatTokens(tokensOut)}↓ tok`;
+  return `${formatTokens(tokensIn)} tok`;
+}
+
 export function formatDuration(ms: number): string {
   const s = ms / 1000;
   return s < 60 ? `${s.toFixed(1)}s` : `${Math.floor(s / 60)}m${Math.round(s % 60)}s`;

package/src/ui/tree/tree-model.ts

@@ -26,6 +26,8 @@ export interface RunSnapshot {
   readonly rootModel?: string;
   /** Root's OWN token spend (driver-model turns) — never a subtree sum. */
   readonly rootTokens: number;
+  readonly rootTokensIn: number;
+  readonly rootTokensOut: number;
   readonly subcalls: readonly RlmSubcall[];
 }
 
@@ -46,6 +48,8 @@ export interface NodeRow {
   readonly label: string;
   /** The row's OWN token spend for its OWN model — never a subtree sum. */
   readonly tokens: number;
+  readonly tokensIn: number;
+  readonly tokensOut: number;
   readonly model?: string;
 }
 
@@ -64,6 +68,8 @@ export interface GroupRow {
   readonly model?: string;
   /** Sum over members — one model only (the group key pins it), so never a blend. */
   readonly tokens: number;
+  readonly tokensIn: number;
+  readonly tokensOut: number;
   /** SubcallStatus, or "queued" while any member parks on the rate-limit cooldown. */
   readonly icon: SubcallStatus | "queued";
   readonly expandable: boolean;
@@ -147,6 +153,8 @@ export function buildRows(
       phase: sc.phase,
       label: sc.label,
       tokens: sc.tokens,
+      tokensIn: sc.tokensIn,
+      tokensOut: sc.tokensOut,
       model: sc.model,
     });
     if (!expanded || children.length === 0) return;
@@ -181,7 +189,13 @@ export function buildRows(
     const id = `grp:${run.runId}:${entry.key}:${first.id}`;
     const expanded = expandedGroups.has(id);
     let tokens = 0;
-    for (const m of entry.members) tokens += m.tokens;
+    let tokensIn = 0;
+    let tokensOut = 0;
+    for (const m of entry.members) {
+      tokens += m.tokens;
+      tokensIn += m.tokensIn;
+      tokensOut += m.tokensOut;
+    }
     rows.push({
       type: "group",
       id,
@@ -192,6 +206,8 @@ export function buildRows(
       label: entry.label,
       model: entry.model,
       tokens,
+      tokensIn,
+      tokensOut,
       icon: iconOf(entry.status, entry.members.some((m) => m.phase === "queued") ? "queued" : undefined),
       expandable: true,
       expanded,
@@ -218,6 +234,8 @@ export function buildRows(
     phase: run.rootPhase,
     label: run.rootLabel,
     tokens: run.rootTokens,
+    tokensIn: run.rootTokensIn,
+    tokensOut: run.rootTokensOut,
     model: run.rootModel,
   });
   if (!collapsed.has(run.runId)) visitChildren(roots, 0, "");

package/src/ui/tree/tree-rows.ts

@@ -8,7 +8,7 @@
 
 import type { Theme } from "@earendil-works/pi-coding-agent";
 import { truncateToWidth, visibleWidth } from "@earendil-works/pi-tui";
-import { formatTokens, spinnerFrame } from "../theme.ts";
+import { formatTokens, formatTokensSplit, spinnerFrame } from "../theme.ts";
 import type { GroupRow, NodeRow, TreeRow } from "./tree-model.ts";
 
 const GLYPHS = Object.freeze({ done: "✓", error: "✗", queued: "◷", expanded: "▾", collapsed: "▸", leaf: " " } as const);
@@ -29,16 +29,16 @@ function iconGlyph(row: NodeRow, theme: Theme): string {
   }
 }
 
-/** Right-hand stats: "4.3k tok · gpt-5-mini" (model only when present). */
-function statsText(tokens: number, model: string | undefined): string {
-  const parts = [`${formatTokens(tokens)} tok`];
+/** Right-hand stats: "190.2k↑ 18.6k↓ tok · gpt-5-mini" (split + model only when present). */
+function statsText(tokens: number, tokensIn: number, tokensOut: number, model: string | undefined): string {
+  const parts = [tokensOut > 0 ? formatTokensSplit(tokensIn, tokensOut) : `${formatTokens(tokens)} tok`];
   if (model !== undefined) parts.push(modelShort(model));
   return parts.join(" · ");
 }
 
 /** Left/right assembly shared by node and group rows — one padding rule, no fork. */
-function assembleLine(left: string, tokens: number, model: string | undefined, selected: boolean, width: number, theme: Theme): string {
-  const right = theme.fg("dim", statsText(tokens, model));
+function assembleLine(left: string, tokens: number, tokensIn: number, tokensOut: number, model: string | undefined, selected: boolean, width: number, theme: Theme): string {
+  const right = theme.fg("dim", statsText(tokens, tokensIn, tokensOut, model));
   const gap = width - visibleWidth(left) - visibleWidth(right) - 1;
   const line = gap > 0 ? `${left}${" ".repeat(gap)}${right}` : `${truncateToWidth(left, width - 1)} `;
   return selected ? theme.fg("accent", line) : line;
@@ -48,7 +48,7 @@ function formatNode(row: NodeRow, selected: boolean, width: number, theme: Theme
   const chevron = row.expandable ? (row.expanded ? GLYPHS.expanded : GLYPHS.collapsed) : GLYPHS.leaf;
   const cursor = selected ? theme.fg("accent", "❯") : " ";
   const left = `${cursor} ${row.prefix}${chevron} ${iconGlyph(row, theme)} ${row.label}`;
-  return assembleLine(left, row.tokens, row.model, selected, width, theme);
+  return assembleLine(left, row.tokens, row.tokensIn, row.tokensOut, row.model, selected, width, theme);
 }
 
 function formatGroup(row: GroupRow, selected: boolean, width: number, theme: Theme): string {
@@ -56,7 +56,7 @@ function formatGroup(row: GroupRow, selected: boolean, width: number, theme: The
   const cursor = selected ? theme.fg("accent", "❯") : " ";
   const icon = row.icon === "done" ? theme.fg("success", GLYPHS.done) : row.icon === "error" ? theme.fg("error", GLYPHS.error) : theme.fg("warning", spinnerFrame());
   const left = `${cursor} ${row.prefix}${chevron} ${icon} ${row.label} ×${row.count}`;
-  return assembleLine(left, row.tokens, row.model, selected, width, theme);
+  return assembleLine(left, row.tokens, row.tokensIn, row.tokensOut, row.model, selected, width, theme);
 }
 
 export function formatRow(row: TreeRow, selected: boolean, width: number, theme: Theme): string {

package/src/util/retry.ts

@@ -17,10 +17,13 @@ import { ProviderCooldown, sleepMs, sharedCooldown } from "./throttle.ts";
 
 // Auth/quota failures must not be retried — they burn attempts and never recover.
 const NON_RETRYABLE_TEXT =
-  /api[ -]?key|unauthorized|forbidden|permission denied|billing|insufficient|balance|quota exceeded|not.?found|context length|too large|invalid request|malformed/i;
-// Transport/server transients — worth another attempt.
+  /api[ -]?key|unauthorized|forbidden|permission denied|billing|insufficient|balance|quota exceeded|not.?found|context length|too large|invalid request|malformed|content.?filter/i;
+// Transport/server transients — worth another attempt. "Provider finish_reason: error" is the
+// generic shape OpenRouter relays when the UPSTREAM kills a generation mid-stream (observed
+// from Cohere's free pool: native_finish_reason "error", no message, no code, partial usage) —
+// HTTP 200, so it is inherently transient-by-nature and must be retried.
 const RETRYABLE_TEXT =
-  /\b429\b|rate.?limit|overloaded|service.?unavailable|upstream|timeout|timed.?out|temporarily|try.?again|econnreset|econnrefused|etimedout|socket hang up|network|1302|速率|频率/i;
+  /\b429\b|rate.?limit|overloaded|service.?unavailable|upstream|timeout|timed.?out|temporarily|try.?again|econnreset|econnrefused|etimedout|socket hang up|network|finish.?reason: ?(error|network_error)|1302|速率|频率/i;
 const RATE_LIMIT_TEXT = /\b429\b|rate.?limit|1302|速率|频率/i;
 
 const NON_RETRYABLE_STATUS = new Set([400, 401, 402, 403, 404, 413, 422]);