pi-jev-lens 0.2.1 → 0.4.0

package/README.md

@@ -64,6 +64,8 @@ Three results shaped the design:
 
 ## Install
 
+Use pi 0.84.3 or newer. The extension uses pi's built-in tool renderers for results that it does not compress.
+
 ```sh
 pi install npm:pi-jev-lens                    # from npm
 pi install git:github.com/dizk/pi-jev-lens    # or from GitHub
@@ -73,12 +75,22 @@ jev needs a TypeSafe API key. You can get one at [console.typesafe.ai](https://c
 for the key in this order:
 
 1. `TYPESAFE_API_KEY` in the environment.
-2. The key that you stored with `/jev-lens key` inside pi. The command prompts for the key, or you can give it as
-   `/jev-lens key ts_...`. The key is stored in `~/.pi/agent/jev-lens.json`, readable only by you. jev is active from
-   the next tool result. You do not have to restart pi.
-3. A `.env` file next to the installed package. This is for development.
+2. A `.env` file next to the installed package. This is for development and supplies environment values that are not already set.
+3. The key that you stored with `/jev-lens key` inside pi. In terminal mode, the command opens a masked input field.
+   The key is stored in `~/.pi/agent/jev-lens.json`. New files are readable only by you.
+
+Use `/jev-lens key` without an argument to keep the key out of command history. The field displays only `*` characters.
+Type or paste the key, then press Enter to save. Press Esc to cancel without changing the stored key.
+In RPC or noninteractive mode, set `TYPESAFE_API_KEY`. These modes do not fall back to a visible input field.
+The key file still stores the key as plain text. Masking protects the terminal display, not the file.
+
+You can still use `/jev-lens key ts_...`, but that exposes the key in the editor and can retain it in command history.
+A new key takes effect without a restart, but the command does not validate it.
+If `TYPESAFE_API_KEY` is set, that value takes priority again after reload.
+If mock mode is forced or compression is disabled, storing a key does not change those settings.
 
-If no key is found, the extension shows a warning at startup and runs a mock classifier that compresses nothing.
+If no key is found, the extension shows a warning at startup and uses a deterministic mock classifier.
+The mock can compress results without API calls. It is not the jev model.
 
 For development, clone the repository and load it directly:
 
@@ -143,20 +155,38 @@ The footer shows the share of the session's input tokens that jev kept out of th
 jev-lens −38% of input (presend −12.3k · 5/8 · 1 recalls)
 ```
 
-The share is cut divided by sent plus cut. Sent is the provider's own count of input and cache-read tokens over all
-calls. Cut is what every compressed result saved on every call that it was part of.
+The share is cut divided by sent plus cut. Sent counts input and cache-read tokens reported by the provider since the last load.
+Cut estimates what compressed results saved on those calls, including results restored from the session.
+The percentage counts repeated savings when the same result appears in later prompts. The `presend` total counts each result once.
 
-In the transcript, a compressed result shows a header like `⌁ jev-lens outline · 179 of 1524 tokens (−88 %)`. When
-you expand it with ctrl+e, you see exactly what the model saw. These commands are available:
+After `/reload` or resume, the footer includes saved tokens from restored compressed results.
+For example, `0/3 new · 5 restored` means no new compressions among three candidates, plus five restored compressed results.
+New-result counts and recall counts start at zero after loading. `/jev-lens stats` shows new and restored savings separately.
 
-- `/jev-lens` shows the statistics and where the key comes from.
+In the transcript, a compressed result shows a header like `⌁ jev-lens outline · 179 of 1524 tokens (−88 %)`. When
+you press ctrl+o to expand, you see full output on the left and compressed output on the right.
+Retained lines align with their originals. A `−` marks each omitted line in the full output.
+Long lines wrap. On narrow terminals, the two versions appear one below the other.
+The collapsed result ends with a hint such as `… (48 lines pruned, 50 original, ctrl+o for diff)`.
+The compressed version includes omission markers but excludes the recall footer. Press ctrl+o again to collapse.
+This uses pi's tool expansion keybinding, including any custom binding.
+
+Type `/jev-lens ` and press Tab to complete subcommands.
+Use `/reload` after installing the package in a running pi session.
+
+- `/jev-lens` or `/jev-lens stats` shows the statistics, active configuration, and key source.
+- `/jev-lens help` shows command usage.
+- `/jev-lens decisions` shows post-send pruning decisions. Post-send pruning is off by default.
 - `/jev-lens list` lists the latest 200 compressed results with the tokens before and after.
-- `/jev-lens diff [n]` opens an overlay for the n-th latest result. It shows the original with the lines that the
-  model did not get marked with `−`. Press `t` to see what was sent, and `Esc` to close.
 - `/jev-lens key` stores the API key.
 
-jev-lens logs every decision to `<project>/.pi/jev-lens.log` as JSON lines. Set `JEV_LENS_UI=0` to keep pi's own
-tool rendering.
+If compression fails, jev-lens keeps the full output and shows a warning. A failed post-send classification leaves that result unchanged.
+Warnings appear at most once per stage per session. The footer shows `degraded` until a later attempt in that stage succeeds.
+Use `/jev-lens stats` to see failure counts and recovery status. Cancellation does not count as a failure.
+
+Uncompressed results, errors, and streaming updates use pi's built-in tool renderers.
+jev-lens logs every decision to `<project>/.pi/jev-lens.log` as JSON lines. Set `JEV_LENS_UI=0` to disable the
+custom savings headers and tool overrides.
 
 ### Configuration (environment)
 

package/index.ts

@@ -11,18 +11,21 @@
  */
 import { appendFileSync, mkdirSync } from "node:fs";
 import { join } from "node:path";
-import type { ExtensionAPI, ExtensionContext, Theme } from "@earendil-works/pi-coding-agent";
+import type { ExtensionAPI, ExtensionContext, ToolDefinition } from "@earendil-works/pi-coding-agent";
 import type { AgentMessage } from "./src/pi-types.ts";
-import { CONFIG_DIR_NAME } from "@earendil-works/pi-coding-agent";
+import { CONFIG_DIR_NAME, keyText } from "@earendil-works/pi-coding-agent";
 import { Type } from "typebox";
-import { createBashTool, createFindTool, createGrepTool, createLsTool, createReadTool } from "@earendil-works/pi-coding-agent";
+import { createBashToolDefinition, createFindToolDefinition, createGrepToolDefinition, createLsToolDefinition, createReadToolDefinition } from "@earendil-works/pi-coding-agent";
 import { Text } from "@earendil-works/pi-tui";
-import { DiffOverlay, listLines, savingsLine, type CompressedRecord } from "./src/ui.ts";
+import { ComparisonResult, comparisonHint, listLines, savingsLine, type CompressedRecord } from "./src/ui.ts";
 import { TypeSafeClient } from "@typesafe-ai/sdk";
 import { buildItemState, JevClassifier, MockClassifier, type Classifier } from "./src/classifier.ts";
 import { buildPresendState, decideView, DEFAULT_PROMPTS, expandRelevantBlocks, JevPresend, MockPresend, type PresendClassifier, type PromptVariant } from "./src/presend.ts";
 import { buildCandidatesAsync, extractTerms, footer } from "./src/views.ts";
 import { keyFilePath, loadConfigWithVariant, storeKey, type Config } from "./src/config.ts";
+import { Health } from "./src/health.ts";
+import { SecretInput } from "./src/secret-input.ts";
+import { commandCompletions, commandHelp } from "./src/commands.ts";
 import { ENTRY_TYPE, rebuildLedger } from "./src/ledger.ts";
 import { applyLedger, decideBucket, pendingPrunable, shouldApplyPending } from "./src/policy.ts";
 import { contentText, describeToolCall, estimateTokensOfText, toolCallsOf, truncate } from "./src/text.ts";
@@ -37,6 +40,7 @@ export default function (pi: ExtensionAPI) {
 	const { cfg, variant } = loadConfigWithVariant();
 	const prompts: PromptVariant = { ...DEFAULT_PROMPTS, ...((variant.prompts ?? {}) as Partial<PromptVariant>), viewDescriptions: { ...DEFAULT_PROMPTS.viewDescriptions, ...(((variant.prompts ?? {}) as Partial<PromptVariant>).viewDescriptions ?? {}) } };
 	const viewParams = variant.views ?? {};
+	let keySource = process.env.TYPESAFE_API_KEY === cfg.apiKey && cfg.apiKey ? "env" : cfg.apiKey ? keyFilePath() : "none";
 	let usingMock = cfg.forceMock || !cfg.apiKey;
 	let classifier: Classifier = usingMock ? new MockClassifier() : new JevClassifier(cfg);
 	let presend: PresendClassifier = usingMock ? new MockPresend() : new JevPresend(new TypeSafeClient({ apiKey: cfg.apiKey }), cfg.model, prompts);
@@ -54,7 +58,9 @@ export default function (pi: ExtensionAPI) {
 	const recordById = new Map<string, CompressedRecord>();
 	const remember = (r: CompressedRecord) => { records.push(r); recordById.set(r.id, r); if (records.length > 200) { const old = records.shift(); if (old) recordById.delete(old.id); } };
 	let lastAssistantText = "";
+	let health = new Health();
 	let presendTotals = { considered: 0, compressed: 0, tokensSaved: 0, recalls: 0 };
+	let restored = { compressed: 0, tokensSaved: 0 };
 
 	let ledger = new Map<string, Decision>();
 	/** Classifications launched but not yet resolved, keyed by toolCallId. */
@@ -96,14 +102,18 @@ export default function (pi: ExtensionAPI) {
 		return sent > 0 ? Math.round((100 * kept) / (sent + kept)) : undefined;
 	};
 	const statusText = () => {
-		const tag = usingMock ? "jev-lens(mock)" : "jev-lens";
+		const tag = !cfg.enabled ? "jev-lens(disabled)" : usingMock ? "jev-lens(mock)" : "jev-lens";
 		const pct = cutShare();
-		const lead = pct === undefined ? tag : `${tag} −${pct}% of input`;
+		const label = health.failing ? `${tag}(degraded)` : tag;
+		const lead = pct === undefined ? label : `${label} −${pct}% of input`;
 		const pruned = cfg.mode === "off" ? "" : `, pruned −${(totals.pruned / 1000).toFixed(1)}k · ${totals.applied}`;
-		return `${lead} (presend −${(presendTotals.tokensSaved / 1000).toFixed(1)}k · ${presendTotals.compressed}/${presendTotals.considered} · ${presendTotals.recalls} recalls${pruned})`;
+		const saved = presendTotals.tokensSaved + restored.tokensSaved;
+		const counts = `${presendTotals.compressed}/${presendTotals.considered}${restored.compressed ? ` new · ${restored.compressed} restored` : ""}`;
+		return `${lead} (presend −${(saved / 1000).toFixed(1)}k · ${counts} · ${presendTotals.recalls} recalls${pruned})`;
 	};
 	const status = (ctx: ExtensionContext) => {
 		if (!ctx.hasUI) return;
+		for (const warning of health.warnings()) ctx.ui.notify(warning, "warning");
 		ctx.ui.setStatus("jev-lens", statusText());
 	};
 
@@ -116,6 +126,7 @@ export default function (pi: ExtensionAPI) {
 		sessionAbort.abort();
 		sessionAbort = new AbortController();
 		lastAssistantText = "";
+		health = new Health();
 		ledger = rebuildLedger(ctx.sessionManager.getEntries());
 		buffer = [];
 		inflight.clear();
@@ -130,6 +141,7 @@ export default function (pi: ExtensionAPI) {
 		records.length = 0;
 		recordById.clear();
 		presendTotals = { considered: 0, compressed: 0, tokensSaved: 0, recalls: 0 };
+		restored = { compressed: 0, tokensSaved: 0 };
 		try {
 			mkdirSync(join(ctx.cwd, CONFIG_DIR_NAME), { recursive: true });
 			logPath = join(ctx.cwd, CONFIG_DIR_NAME, "jev-lens.log");
@@ -147,6 +159,8 @@ export default function (pi: ExtensionAPI) {
 				if (d?.full) {
 					fullOutputs.set(entry.message.toolCallId, { text: d.full, toolName: entry.message.toolName, args: d.args, view: d.view ?? "?" });
 					const sent = contentText(entry.message.content).replace(/\n\n\[jev-lens:[\s\S]*$/, "");
+					restored.compressed++;
+					restored.tokensSaved += Math.max(0, estimateTokensOfText(d.full) - estimateTokensOfText(sent));
 					remember({ id: entry.message.toolCallId, toolName: entry.message.toolName, args: d.args, kind: d.kind ?? "?", view: d.view ?? "?", tokensBefore: estimateTokensOfText(d.full), tokensAfter: estimateTokensOfText(sent), full: d.full, sent, included: d.included ?? [], needsFull: d.needsFull, pFull: d.p?.full, recalls: 0, at: entry.message.timestamp });
 				}
 			}
@@ -206,20 +220,19 @@ export default function (pi: ExtensionAPI) {
 		}
 	});
 
-	pi.on("agent_end", async () => {
-		const epoch = generation;
+	pi.on("agent_end", async (_event, ctx) => {
 		// No further assistant reaction is coming for the last results; classify with what we have.
 		const toClassify = buffer;
 		buffer = [];
-		for (const item of toClassify) launchClassification(item, "", [], undefined);
+		for (const item of toClassify) launchClassification(item, "", [], ctx);
 		await waitForWork([...inflight.values()]);
-		void epoch;
 	});
 
 	function launchClassification(item: PendingResult, afterText: string, afterCalls: { name: string; arguments: unknown }[], ctx?: ExtensionContext) {
 		const m = item.message;
 		if (cfg.mode === "off" || sessionAbort.signal.aborted || m.content.some((c) => c.type !== "text")) return;
 		const epoch = generation;
+		const signal = workSignal(ctx?.signal);
 		if (ledger.has(m.toolCallId) || inflight.has(m.toolCallId)) return;
 		const output = contentText(m.content);
 		const tokens = estimateTokensOfText(output);
@@ -238,9 +251,11 @@ export default function (pi: ExtensionAPI) {
 		const summary = describeToolCall(m.toolName, item.args, output.length, lines);
 		const started = Date.now();
 		const p = classifier
-			.classifyToolResult(state, workSignal(ctx?.signal))
+			.classifyToolResult(state, signal)
 			.then((probs) => {
-				if (epoch !== generation) return;
+				if (epoch !== generation || signal.aborted) return;
+				health.success("postsend");
+				if (ctx) status(ctx);
 				const decision: Decision = {
 					id: m.toolCallId,
 					toolName: m.toolName,
@@ -256,7 +271,10 @@ export default function (pi: ExtensionAPI) {
 				log({ event: "decision", id: decision.id, tool: m.toolName, bucket: decision.bucket, p: probs, tokens, ms: Date.now() - started, summary });
 			})
 			.catch((err) => {
-				if (epoch === generation) log({ event: "classify_error", id: m.toolCallId, error: String(err?.message ?? err) });
+				if (epoch !== generation || signal.aborted) return;
+				health.failure("postsend", err);
+				log({ event: "classify_error", id: m.toolCallId, error: health.lines()[1] });
+				if (ctx) status(ctx);
 			})
 			.finally(() => { if (epoch === generation) inflight.delete(m.toolCallId); });
 		inflight.set(m.toolCallId, p);
@@ -348,16 +366,16 @@ export default function (pi: ExtensionAPI) {
 		if (event.content.some((c) => c.type === "image")) return;
 		presendTotals.considered++;
 		const started = Date.now();
-		const terms = extractTerms(latestUser, lastAssistantText, JSON.stringify(event.input ?? {}));
-		const cands = await buildCandidatesAsync(event.toolName, event.input, text, terms, viewParams);
-		if (epoch !== generation || signal.aborted) return;
-		if (cands.views.length < 2) {
-			log({ event: "presend", id: event.toolCallId, tool: event.toolName, tokens, view: "full", reason: "no-candidates" });
-			return;
-		}
-		const totalLines = text.split("\n").length;
-		const state = buildPresendState(cfg, { firstUser, latestUser, agentText: lastAssistantText, toolName: event.toolName, args: event.input, isError: event.isError, cands, totalLines, totalChars: text.length });
 		try {
+			const terms = extractTerms(latestUser, lastAssistantText, JSON.stringify(event.input ?? {}));
+			const cands = await buildCandidatesAsync(event.toolName, event.input, text, terms, viewParams);
+			if (epoch !== generation || signal.aborted) return;
+			if (cands.views.length < 2) {
+				log({ event: "presend", id: event.toolCallId, tool: event.toolName, tokens, view: "full", reason: "no-candidates" });
+				return;
+			}
+			const totalLines = text.split("\n").length;
+			const state = buildPresendState(cfg, { firstUser, latestUser, agentText: lastAssistantText, toolName: event.toolName, args: event.input, isError: event.isError, cands, totalLines, totalChars: text.length });
 			const answer = await presend.choose(state, cands.views.map((v) => v.kind), signal);
 			if (epoch !== generation || signal.aborted) return;
 			let view = decideView(answer, cands, cfg);
@@ -368,6 +386,8 @@ export default function (pi: ExtensionAPI) {
 				if (epoch !== generation || signal.aborted) return;
 				if (ex) { view = ex.view; expanded = ex.probs.map((p, i) => (p > above ? i : -1)).filter((i) => i >= 0); }
 			}
+			health.success("presend");
+			status(ctx);
 			log({ event: "presend", id: event.toolCallId, tool: event.toolName, kind: cands.kind, tokens, view: view.kind, viewTokens: estimateTokensOfText(view.text), chosen: answer.choice, needsFull: answer.needsFull, p: answer.probabilities, confidence: answer.confidence, expanded, candidates: cands.views.map((v) => `${v.kind}:${v.chars}`), ms: Date.now() - started });
 			if (view.kind === "full") return;
 			presendTotals.compressed++;
@@ -378,7 +398,10 @@ export default function (pi: ExtensionAPI) {
 			status(ctx);
 			return { content: [{ type: "text", text: view.text + footer(view, event.toolCallId, totalLines) }], details };
 		} catch (err) {
-			if (epoch === generation) log({ event: "presend_error", id: event.toolCallId, error: String((err as Error)?.message ?? err) });
+			if (epoch !== generation || signal.aborted) return;
+			health.failure("presend", err);
+			log({ event: "presend_error", id: event.toolCallId, error: health.lines()[0] });
+			status(ctx);
 			return;
 		}
 	});
@@ -427,96 +450,92 @@ export default function (pi: ExtensionAPI) {
 
 	if (process.env.JEV_LENS_UI !== "0") {
 		const cwd = process.cwd();
-		const originals: Record<string, ReturnType<typeof createReadTool>> = {
-			read: createReadTool(cwd) as ReturnType<typeof createReadTool>,
-			bash: createBashTool(cwd) as unknown as ReturnType<typeof createReadTool>,
-			grep: createGrepTool(cwd) as unknown as ReturnType<typeof createReadTool>,
-			find: createFindTool(cwd) as unknown as ReturnType<typeof createReadTool>,
-			ls: createLsTool(cwd) as unknown as ReturnType<typeof createReadTool>,
-		};
-		for (const [name, original] of Object.entries(originals)) {
-			const o = original as unknown as { description: string; parameters: never; execute: (...a: unknown[]) => Promise<unknown>; renderCall?: (...a: unknown[]) => unknown; renderResult?: (...a: unknown[]) => unknown; promptSnippet?: string; promptGuidelines?: string[] };
+		// Tool definitions include pi's renderers; create*Tool() strips them.
+		const originals: ToolDefinition<any, any>[] = [
+			createReadToolDefinition(cwd), createBashToolDefinition(cwd),
+			createGrepToolDefinition(cwd), createFindToolDefinition(cwd), createLsToolDefinition(cwd),
+		];
+		for (const original of originals) {
 			pi.registerTool({
-				name,
-				label: name,
-				description: o.description,
-				parameters: o.parameters,
-				promptSnippet: o.promptSnippet,
-				promptGuidelines: o.promptGuidelines,
-				async execute(toolCallId: string, params: unknown, signal: AbortSignal | undefined, onUpdate: unknown, ctx: unknown) {
-					return (o.execute as (id: string, p: unknown, s: unknown, u: unknown, c: unknown) => Promise<never>)(toolCallId, params, signal, onUpdate, ctx);
-				},
-				renderCall(args: unknown, theme: Theme, context: unknown) {
-					if (o.renderCall) return (o.renderCall as (a: unknown, t: unknown, c: unknown) => never)(args, theme, context);
-					const a = args as Record<string, unknown>;
-					const what = typeof a.path === "string" ? a.path : typeof a.command === "string" ? a.command : typeof a.pattern === "string" ? a.pattern : "";
-					return new Text(theme.fg("toolTitle", theme.bold(`${name} `)) + theme.fg("accent", String(what)), 0, 0);
-				},
-				renderResult(result: { content: unknown }, options: { expanded: boolean }, theme: Theme, context: { toolCallId: string }) {
-					const rec = recordById.get(context.toolCallId);
-					if (!rec) {
-						if (o.renderResult) return (o.renderResult as (r: unknown, op: unknown, t: unknown, c: unknown) => never)(result, options, theme, context);
-						const text = contentText(result.content);
-						const lines = text.split("\n");
-						let out = theme.fg("success", `${lines.length} lines`);
-						if (options.expanded) out += "\n" + lines.slice(0, 200).join("\n");
-						else out += theme.fg("dim", "  " + lines[0]?.slice(0, 80));
-						return new Text(out, 0, 0);
+				...original,
+				renderResult(result, options, theme, context) {
+					let rec = recordById.get(context.toolCallId);
+					// Older rows can leave the recent-results index, but retain their comparison.
+					const d = result.details?.jevLens;
+					if (!rec && typeof d?.full === "string") {
+						const sent = contentText(result.content).replace(/\n\n\[jev-lens:[\s\S]*$/, "");
+						rec = { id: context.toolCallId, toolName: original.name, args: d.args,
+							kind: d.kind ?? "?", view: d.view ?? "?", full: d.full, sent,
+							tokensBefore: estimateTokensOfText(d.full), tokensAfter: estimateTokensOfText(sent),
+							included: d.included ?? [], recalls: 0, at: 0 };
 					}
+					if (!rec || options.isPartial || context.isError) {
+						return original.renderResult!(result, options, theme, context);
+					}
+					if (options.expanded) return new ComparisonResult(rec, theme, `${keyText("app.tools.expand")} to collapse`);
 					let out = savingsLine(rec, theme);
-					if (options.expanded) out += "\n" + rec.sent;
-					else out += "\n" + theme.fg("dim", rec.sent.split("\n").slice(0, 3).join("\n"));
+					out += "\n" + theme.fg("dim", rec.sent.split("\n").slice(0, 3).join("\n"));
+					out += "\n" + theme.fg("dim", comparisonHint(rec, keyText("app.tools.expand")));
 					return new Text(out, 0, 0);
 				},
-			} as never);
+			});
 		}
 	}
 
 	// ---- commands ----------------------------------------------------------------------
 
 	pi.registerCommand("jev-lens", {
-		description: "jev-lens: stats | list (compressed results) | diff [n] (original vs sent, overlay) | decisions | key [api-key] (store your TypeSafe key)",
+		description: "Inspect compression and setup: stats | list | decisions | key | help",
+		getArgumentCompletions: (prefix) => commandCompletions(prefix),
 		handler: async (args, ctx) => {
 			const sub = (args ?? "").trim();
-			if (sub === "key" || sub.startsWith("key ")) {
+			if (sub === "help" || sub === "--help" || sub === "-h") {
+				ctx.ui.notify(commandHelp, "info");
+				return;
+			}
+			if (/^key(?:\s|$)/.test(sub)) {
 				let key = sub.slice(3).trim();
-				if (!key) key = ((await ctx.ui.input("TypeSafe API key (from console.typesafe.ai):", "ts_...")) ?? "").trim();
-				if (!key) { ctx.ui.notify("no key entered", "info"); return; }
-				const where = storeKey(key);
+				if (!key && (!ctx.hasUI || ctx.mode !== "tui")) { ctx.ui.notify("Masked key input requires terminal mode. Set TYPESAFE_API_KEY or run /jev-lens key in interactive pi.", "warning"); return; }
+				if (!key) key = ((await ctx.ui.custom<string | undefined>((tui, theme, keys, done) =>
+					new SecretInput(theme, keys, done, () => tui.requestRender()),
+				)) ?? "").trim();
+				if (!key) { ctx.ui.notify("Key setup cancelled. The current key is unchanged.", "info"); return; }
+				let where: string;
+				try { where = storeKey(key); }
+				catch {
+					ctx.ui.notify(`Could not store the key in ${keyFilePath()}. Check directory permissions or set TYPESAFE_API_KEY.`, "error");
+					return;
+				}
 				useKey(key);
-				ctx.ui.notify(`jev-lens: key stored in ${where}; jev is active from the next tool result`, "info");
+				keySource = where;
+				const next = cfg.forceMock ? "Mock mode remains active. Unset JEV_LENS_CLASSIFIER and reload pi to use jev." : !cfg.enabled || !cfg.presend ? "Pre-send compression is disabled. See /jev-lens stats." : "jev will use this key from the next tool result. The key has not been validated.";
+				ctx.ui.notify(`jev-lens: key stored in ${where}. ${next}${process.env.TYPESAFE_API_KEY ? " TYPESAFE_API_KEY takes priority again after reload." : ""}`, "info");
 				status(ctx);
 				return;
 			}
-			if (sub.startsWith("diff")) {
-				const n = Number(sub.slice(4).trim() || "1");
-				const rec = records[records.length - (Number.isFinite(n) && n >= 1 ? n : 1)];
-				if (!rec) { ctx.ui.notify("no compressed tool result to show yet", "info"); return; }
-				if (!ctx.hasUI || ctx.mode !== "tui") { ctx.ui.notify(listLines([rec], { fg: (_c, t) => t, bold: (t) => t }).join("\n"), "info"); return; }
-				await ctx.ui.custom<void>((tui, theme, _kb, done) => {
-					const height = Math.max(12, Math.floor(((tui as { terminalHeight?: number }).terminalHeight ?? process.stdout.rows ?? 40) * 0.85));
-					const overlay = new DiffOverlay(rec, theme, height, () => done(), () => tui.requestRender());
-					return { render: (w) => overlay.render(w), handleInput: (d) => overlay.handleInput(d), invalidate: () => overlay.invalidate() };
-				}, { overlay: true, overlayOptions: { width: "92%", maxHeight: "90%", anchor: "center" } });
-				return;
-			}
 			if (sub === "list") {
 				ctx.ui.notify(listLines(records, { fg: (_c, t) => t, bold: (t) => t }).join("\n"), "info");
 				return;
 			}
 			if (sub === "decisions") {
 				const rows = [...ledger.values()].map((d) => `${d.status === "applied" ? "●" : "○"} ${d.bucket.padEnd(6)} n=${d.p.needed.toFixed(2)} o=${d.p.outcomeOnly.toFixed(2)} ${d.tokensBefore}t ${d.summary}`);
-				ctx.ui.notify(rows.join("\n") || "(no decisions yet)", "info");
+				ctx.ui.notify(rows.join("\n") || (cfg.mode === "off" ? "Post-send pruning is off (the default). Pre-send compression is separate: see /jev-lens stats." : "No post-send decisions yet."), "info");
+				return;
+			}
+			if (sub && sub !== "stats") {
+				ctx.ui.notify("Unknown subcommand or extra arguments. Run /jev-lens help for usage.", "warning");
 				return;
 			}
 			const hit = totals.input + totals.cacheRead > 0 ? Math.round((100 * totals.cacheRead) / (totals.input + totals.cacheRead)) : 0;
 			ctx.ui.notify(
 				[
-					`mode=${cfg.mode} enabled=${cfg.enabled} classifier=${usingMock ? "mock (no key: /jev-lens key)" : cfg.model} key=${process.env.TYPESAFE_API_KEY ? "env" : cfg.apiKey ? keyFilePath() : "none"}`,
-					`presend: ${presendTotals.compressed}/${presendTotals.considered} large results compressed, ≈${presendTotals.tokensSaved} tokens saved, ${presendTotals.recalls} recalls`,
+					`mode=${cfg.mode} enabled=${cfg.enabled} presend=${cfg.presend} classifier=${usingMock ? cfg.forceMock ? "mock (forced by JEV_LENS_CLASSIFIER)" : "mock (no key: /jev-lens key)" : cfg.model} key=${keySource}`,
+					`presend since load: ${presendTotals.compressed}/${presendTotals.considered} large results compressed, ≈${presendTotals.tokensSaved} tokens saved, ${presendTotals.recalls} recalls`,
+					`restored from session: ${restored.compressed} compressed results, ≈${restored.tokensSaved} tokens saved (included in footer savings)`,
 					`post-send: calls=${totals.calls} decisions=${ledger.size} applied=${totals.applied} pruned≈${totals.pruned} tokens`,
+					...health.lines(),
 					`cache: read=${totals.cacheRead} uncached=${totals.input} hit=${hit}%`,
-					`input cut: ${cutShare() ?? 0}% of the session's input tokens (≈${cut.presend + cut.pruned} of ${totals.input + totals.cacheRead + cut.presend + cut.pruned}: presend ${cut.presend}, pruned ${cut.pruned}, summed over ${totals.calls} calls)`,
+					`input cut: ${cutShare() ?? 0}% of input tokens counted since load (≈${cut.presend + cut.pruned} of ${totals.input + totals.cacheRead + cut.presend + cut.pruned}: presend ${cut.presend}, pruned ${cut.pruned}, summed over ${totals.calls} calls)`,
 				].join("\n"),
 				"info",
 			);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-jev-lens",
-  "version": "0.2.1",
+  "version": "0.4.0",
   "description": "pi extension that compresses large tool results before they reach the model: jev picks the view (outline, relevant blocks, sections, signals, testlog), full text stays recallable",
   "author": "Didrik Rognstad",
   "license": "MIT",
@@ -51,7 +51,7 @@
     "web-tree-sitter": "^0.27.0"
   },
   "peerDependencies": {
-    "@earendil-works/pi-coding-agent": "*",
+    "@earendil-works/pi-coding-agent": ">=0.84.3",
     "@earendil-works/pi-tui": "*",
     "typebox": "*"
   },

package/src/commands.ts

@@ -0,0 +1,21 @@
+import type { AutocompleteItem } from "@earendil-works/pi-tui";
+
+const commands = [
+	{ value: "stats", label: "stats", description: "Show statistics and active configuration" },
+	{ value: "list", label: "list", description: "List recent compressed results" },
+	{ value: "decisions", label: "decisions", description: "Show post-send pruning decisions" },
+	{ value: "key", label: "key", description: "Store a TypeSafe API key" },
+	{ value: "help", label: "help", description: "Show commands and usage" },
+];
+
+export const commandHelp = [
+	"/jev-lens [stats] — Show statistics and active configuration.",
+	...commands.slice(1).map((c) => `/jev-lens ${c.value} — ${c.description}.`),
+	"ctrl+o to expand tool output and compare full and compressed output (default keybinding).",
+	"Press Tab after /jev-lens to complete a subcommand.",
+].join("\n");
+
+export function commandCompletions(prefix: string): AutocompleteItem[] | null {
+	const items = commands.filter((c) => c.value.startsWith(prefix.trimStart()));
+	return items.length ? items : null;
+}

package/src/health.ts

@@ -0,0 +1,38 @@
+export type Stage = "presend" | "postsend";
+
+type StageHealth = { failures: number; failing: boolean; reason: string; notified: boolean };
+
+/** Session-local failure counters. Never expose provider error messages or credentials. */
+export class Health {
+	private stages: Record<Stage, StageHealth> = {
+		presend: { failures: 0, failing: false, reason: "", notified: false },
+		postsend: { failures: 0, failing: false, reason: "", notified: false },
+	};
+
+	failure(stage: Stage, error: unknown): void {
+		const status = error && typeof error === "object" && "status" in error ? error.status : undefined;
+		const reason = status === 401 || status === 403 ? "Check your TypeSafe API key."
+			: status === 429 ? "TypeSafe rejected the request because of a usage limit."
+			: "Check your connection and TypeSafe service availability.";
+		Object.assign(this.stages[stage], { failures: this.stages[stage].failures + 1, failing: true, reason });
+	}
+
+	success(stage: Stage): void { this.stages[stage].failing = false; }
+
+	get failing(): boolean { return Object.values(this.stages).some((s) => s.failing); }
+
+	/** At most one warning per stage per session, including work completed without a UI context. */
+	warnings(): string[] {
+		return (Object.entries(this.stages) as [Stage, StageHealth][]).flatMap(([stage, state]) => {
+			if (!state.failures || state.notified) return [];
+			state.notified = true;
+			const effect = stage === "presend" ? "Full output was kept." : "The affected result was not pruned.";
+			return [`jev-lens: ${stage === "presend" ? "Pre-send compression" : "Post-send classification"} failed. ${effect} ${state.reason} See /jev-lens stats. Further failures appear there without repeated warnings.`];
+		});
+	}
+
+	lines(): string[] {
+		return (Object.entries(this.stages) as [Stage, StageHealth][]).map(([stage, state]) =>
+			`${stage} failures: ${state.failures}${state.failures ? state.failing ? ` (last attempt failed). ${state.reason}` : " (a later attempt succeeded)" : ""}`);
+	}
+}

package/src/secret-input.ts

@@ -0,0 +1,124 @@
+import { CURSOR_MARKER, decodeKittyPrintable, truncateToWidth, type Component, type Focusable, type KeybindingsManager } from "@earendil-works/pi-tui";
+import type { ThemeLike } from "./ui.ts";
+
+const PASTE_START = "\x1b[200~", PASTE_END = "\x1b[201~";
+const MAX_LENGTH = 4096;
+
+/** A secret-only editor: no plaintext rendering, history, clipboard, undo, or reveal action. */
+export class SecretInput implements Component, Focusable {
+	focused = false;
+	private value: string[] = [];
+	private cursor = 0;
+	private paste: string | undefined;
+	private pasteTooLong = false;
+	private error = "";
+	private closed = false;
+
+	constructor(
+		private theme: ThemeLike,
+		private keys: Pick<KeybindingsManager, "matches" | "getKeys">,
+		private done: (value: string | undefined) => void,
+		private requestRender: () => void,
+	) {}
+
+	private insert(text: string): void {
+		const chars = Array.from(text);
+		if (/[\x00-\x1f\x7f-\x9f]/.test(text)) {
+			this.error = "Paste a single-line API key. Control characters are not allowed.";
+		} else if (this.value.length + chars.length > MAX_LENGTH) {
+			this.error = `The key is too long. Maximum: ${MAX_LENGTH} characters.`;
+		} else {
+			this.value.splice(this.cursor, 0, ...chars);
+			this.cursor += chars.length;
+			this.error = "";
+		}
+	}
+
+	private finish(value?: string): void {
+		this.dispose();
+		this.done(value);
+	}
+
+	/** Drop references on submit, cancel, or external teardown. JS cannot guarantee memory erasure. */
+	dispose(): void {
+		this.value.fill("");
+		this.value = [];
+		this.cursor = 0;
+		this.paste = undefined;
+		this.error = "";
+		this.closed = true;
+	}
+
+	handleInput(data: string): void {
+		if (this.closed) return;
+		if (this.paste === undefined && data.startsWith(PASTE_START)) {
+			this.paste = "";
+			this.pasteTooLong = false;
+			data = data.slice(PASTE_START.length);
+		}
+		if (this.paste !== undefined) {
+			this.paste += data;
+			const end = this.paste.indexOf(PASTE_END);
+			if (end >= 0) {
+				const text = this.paste.slice(0, end);
+				const remaining = this.paste.slice(end + PASTE_END.length);
+				this.paste = undefined;
+				if (this.pasteTooLong) this.error = "The pasted key is too long. Paste only the API key.";
+				else this.insert(text.trim());
+				if (remaining) this.handleInput(remaining);
+			} else if (this.paste.length > MAX_LENGTH + PASTE_END.length) {
+				this.pasteTooLong = true;
+				this.paste = this.paste.slice(-PASTE_END.length); // retain a possible split end marker
+			}
+		} else if (this.keys.matches(data, "tui.select.cancel")) {
+			this.finish();
+		} else if (this.keys.matches(data, "tui.input.submit")) {
+			if (!this.error) this.finish(this.value.join("").trim() || undefined);
+		} else if (this.keys.matches(data, "tui.editor.cursorLeft")) {
+			this.cursor = Math.max(0, this.cursor - 1);
+		} else if (this.keys.matches(data, "tui.editor.cursorRight")) {
+			this.cursor = Math.min(this.value.length, this.cursor + 1);
+		} else if (this.keys.matches(data, "tui.editor.cursorLineStart")) {
+			this.cursor = 0;
+		} else if (this.keys.matches(data, "tui.editor.cursorLineEnd")) {
+			this.cursor = this.value.length;
+		} else if (this.keys.matches(data, "tui.editor.deleteCharBackward")) {
+			if (this.cursor) this.value.splice(--this.cursor, 1);
+			this.error = "";
+		} else if (this.keys.matches(data, "tui.editor.deleteCharForward")) {
+			this.value.splice(this.cursor, 1);
+			this.error = "";
+		} else if (this.keys.matches(data, "tui.editor.deleteToLineStart")) {
+			this.value.splice(0, this.cursor);
+			this.cursor = 0;
+			this.error = "";
+		} else if (this.keys.matches(data, "tui.editor.deleteToLineEnd")) {
+			this.value.splice(this.cursor);
+			this.error = "";
+		} else {
+			const text = decodeKittyPrintable(data) ?? data;
+			if (!/[\x00-\x1f\x7f-\x9f]/.test(text)) this.insert(text);
+		}
+		this.requestRender();
+	}
+
+	invalidate(): void {}
+
+	render(width: number): string[] {
+		if (width < 1) return [""];
+		const room = Math.max(1, width - 2);
+		const start = Math.max(0, this.cursor - room + 1);
+		const before = "*".repeat(this.cursor - start);
+		const after = "*".repeat(Math.min(this.value.length - this.cursor, room - before.length - 1));
+		const cursor = this.cursor < this.value.length ? "*" : " ";
+		const marker = this.focused ? CURSOR_MARKER : "";
+		const field = `${width > 2 ? "> " : ""}${before}${marker}${this.focused ? `\x1b[7m${cursor}\x1b[27m` : cursor}${after}`;
+		return [
+			this.theme.fg("accent", "TypeSafe API key (masked)"),
+			this.theme.fg("dim", "Get a key at console.typesafe.ai. Type or paste it below."),
+			field,
+			this.theme.fg("dim", `${this.keys.getKeys("tui.input.submit").join("/")}: save · ${this.keys.getKeys("tui.select.cancel").join("/")}: cancel`),
+			...(this.error ? [this.theme.fg("warning", this.error)] : []),
+		].map((line) => truncateToWidth(line, width));
+	}
+}

package/src/ui.ts

@@ -2,11 +2,9 @@
  * TUI integration: what the model got versus what the tool really returned.
  *
  * - Built-in tools (read, bash, grep, find, ls) are re-registered with renderers that show, for
- *   a compressed result, a one-line savings header and (expanded) the exact text the model saw.
- * - `/jev-lens diff [n]` opens an overlay with the original output, omitted lines marked, and
- *   `t` toggles to the sent view.
+ *   a compressed result, a savings header and an inline comparison when expanded.
  */
-import { matchesKey, truncateToWidth } from "@earendil-works/pi-tui";
+import { truncateToWidth, visibleWidth, wrapTextWithAnsi } from "@earendil-works/pi-tui";
 
 /** Everything needed to show one compressed result. Stored per session and in the tool result details. */
 export interface CompressedRecord {
@@ -39,11 +37,16 @@ export function savingsLine(r: CompressedRecord, theme: ThemeLike): string {
 		theme.fg("accent", "⌁ jev-lens ") +
 		theme.fg("toolTitle", theme.bold(r.view)) +
 		theme.fg("muted", ` · ${r.tokensAfter} of ${r.tokensBefore} tokens (−${pct} %)`) +
-		(r.recalls ? theme.fg("warning", ` · recalled ${r.recalls}×`) : "") +
-		theme.fg("dim", " · /jev-lens diff")
+		(r.recalls ? theme.fg("warning", ` · recalled ${r.recalls}×`) : "")
 	);
 }
 
+export function comparisonHint(r: CompressedRecord, key: string): string {
+	const original = r.full.split("\n").length;
+	const pruned = original - new Set(r.included).size;
+	return `… (${pruned} ${pruned === 1 ? "line" : "lines"} pruned, ${original} original, ${key} for diff)`;
+}
+
 function describeArgs(toolName: string, args: unknown): string {
 	const a = (args ?? {}) as Record<string, unknown>;
 	if (typeof a.path === "string") return a.path;
@@ -52,83 +55,59 @@ function describeArgs(toolName: string, args: unknown): string {
 	return JSON.stringify(a).slice(0, 80);
 }
 
-/**
- * Overlay showing the original output with omitted lines marked (mode "annotated"), or exactly
- * what was sent (mode "sent"). Keys: ↑/↓ scroll, PgUp/PgDn, Home/End, t toggle, Esc/q close.
- */
-export class DiffOverlay {
-	private offset = 0;
-	private mode: "annotated" | "sent" = "annotated";
+/** Inline comparison. Pi owns expansion and transcript scrolling; no extra key handler. */
+export class ComparisonResult {
 	private cache?: { width: number; lines: string[] };
-	constructor(
-		private record: CompressedRecord,
-		private theme: ThemeLike,
-		private height: number,
-		private onClose: () => void,
-		private onChange?: () => void,
-	) {}
+	constructor(private record: CompressedRecord, private theme: ThemeLike, private hint: string) {}
 
-	private bodyLines(width: number): string[] {
+	render(width: number): string[] {
+		if (width < 1) return [];
+		if (this.cache?.width === width) return this.cache.lines;
 		const r = this.record;
-		const out: string[] = [];
-		if (this.mode === "sent") {
-			for (const l of r.sent.split("\n")) out.push(truncateToWidth(l, width));
-			return out;
-		}
-		const inc = new Set(r.included);
-		const lines = r.full.split("\n");
-		const w = String(lines.length).length;
-		for (let i = 0; i < lines.length; i++) {
-			const n = String(i + 1).padStart(w);
-			if (inc.has(i + 1)) out.push(truncateToWidth(this.theme.fg("dim", `${n} `) + this.theme.fg("text", "│ ") + lines[i], width));
-			else out.push(truncateToWidth(this.theme.fg("toolDiffRemoved", `${n} − ${lines[i]}`), width));
+		const full = r.full.split("\n");
+		const included = new Set(r.included);
+		const digits = String(full.length).length;
+		const original = (i: number) => this.theme.fg(included.has(i + 1) ? "text" : "toolDiffRemoved",
+			`${String(i + 1).padStart(digits)} ${included.has(i + 1) ? "│" : "−"} ${full[i]}`);
+		const wrap = (text: string, columns: number) => wrapTextWithAnsi(text.replace(/\t/g, "    "), columns)
+			.map((line) => truncateToWidth(line, columns));
+		const out = wrap(savingsLine(r, this.theme), width);
+		out.push(...wrap(this.theme.fg("dim", this.hint), width));
+		out.push(...wrap(this.theme.fg("dim", "− omitted from model input · compressed view excludes the recall footer"), width));
+		if (width < 100) {
+			out.push(...wrap(this.theme.bold("Full output"), width));
+			for (let i = 0; i < full.length; i++) out.push(...wrap(original(i), width));
+			out.push("", ...wrap(this.theme.bold("Compressed output"), width));
+			for (const line of r.sent.split("\n")) out.push(...wrap(line, width));
+		} else {
+			const leftWidth = Math.floor((width - 3) / 2);
+			const rightWidth = width - leftWidth - 3;
+			const row = (left: string, right: string) => {
+				const a = wrap(left, leftWidth), b = wrap(right, rightWidth);
+				for (let i = 0; i < Math.max(a.length, b.length); i++) {
+					const l = a[i] ?? "";
+					out.push(l + " ".repeat(Math.max(0, leftWidth - visibleWidth(l))) + this.theme.fg("dim", " │ ") + (b[i] ?? ""));
+				}
+			};
+			row(this.theme.bold("Full output"), this.theme.bold("Compressed output"));
+			let cursor = 0;
+			// Align numbered view lines with their originals. Keep markers and any other
+			// generated view text verbatim on separate rows, rather than reconstructing it.
+			for (const line of r.sent.split("\n")) {
+				const match = /^\s*(\d+)│ /.exec(line);
+				const n = match ? Number(match[1]) : 0;
+				if (n > cursor && n <= full.length && included.has(n)) {
+					while (cursor < n - 1) row(original(cursor++), "");
+					row(original(cursor++), line);
+				} else row("", line);
+			}
+			while (cursor < full.length) row(original(cursor++), "");
 		}
+		this.cache = { width, lines: out };
 		return out;
 	}
 
-	header(width: number): string[] {
-		const r = this.record;
-		const pct = r.tokensBefore ? Math.round((100 * (r.tokensBefore - r.tokensAfter)) / r.tokensBefore) : 0;
-		const omitted = r.full.split("\n").length - r.included.length;
-		return [
-			truncateToWidth(this.theme.fg("accent", this.theme.bold(`jev-lens · ${r.toolName} ${describeArgs(r.toolName, r.args)}`)), width),
-			truncateToWidth(this.theme.fg("muted", `${r.kind} → ${r.view} · ${r.tokensAfter} of ${r.tokensBefore} tokens (−${pct} %) · ${omitted} of ${r.full.split("\n").length} lines omitted` + (r.needsFull !== undefined ? ` · P(needs full)=${r.needsFull.toFixed(2)} P(full)=${(r.pFull ?? 0).toFixed(2)}` : "") + (r.recalls ? ` · recalled ${r.recalls}×` : "")), width),
-			truncateToWidth(this.theme.fg("dim", this.mode === "annotated" ? "original output; − marks lines the model did not get · t: show what was sent · ↑↓ PgUp PgDn · Esc" : "exactly what the model got · t: show original with omissions · ↑↓ PgUp PgDn · Esc"), width),
-			"",
-		];
-	}
-
-	render(width: number): string[] {
-		if (this.cache && this.cache.width === width) return this.cache.lines;
-		const head = this.header(width);
-		const body = this.bodyLines(width);
-		const room = Math.max(3, this.height - head.length - 1);
-		const maxOffset = Math.max(0, body.length - room);
-		if (this.offset > maxOffset) this.offset = maxOffset;
-		const slice = body.slice(this.offset, this.offset + room);
-		const footer = truncateToWidth(this.theme.fg("dim", `lines ${body.length ? this.offset + 1 : 0}-${Math.min(body.length, this.offset + room)} of ${body.length}`), width);
-		this.cache = { width, lines: [...head, ...slice, footer] };
-		return this.cache.lines;
-	}
-
-	handleInput(data: string): void {
-		const page = Math.max(1, this.height - 6);
-		if (matchesKey(data, "escape") || data === "q") { this.onClose(); return; }
-		else if (matchesKey(data, "up")) this.offset = Math.max(0, this.offset - 1);
-		else if (matchesKey(data, "down")) this.offset += 1;
-		else if (matchesKey(data, "pageUp")) this.offset = Math.max(0, this.offset - page);
-		else if (matchesKey(data, "pageDown")) this.offset += page;
-		else if (matchesKey(data, "home")) this.offset = 0;
-		else if (matchesKey(data, "end")) this.offset = Number.MAX_SAFE_INTEGER;
-		else if (data === "t") { this.mode = this.mode === "annotated" ? "sent" : "annotated"; this.offset = 0; }
-		else return;
-		this.invalidate();
-		this.onChange?.();
-	}
-
-	invalidate(): void {
-		this.cache = undefined;
-	}
+	invalidate(): void { this.cache = undefined; }
 }
 
 /** One row per compressed result, newest last. */