pi-jev-lens 0.4.1 → 0.5.0

package/README.md

@@ -1,5 +1,8 @@
 # pi-jev-lens
 
+The pi extension of [jev-lens](https://github.com/dizk/jev-lens). The core that builds and chooses views is the
+`jev-lens` package in the same repository; a Claude Code plugin uses the same core.
+
 Your coding agent reads a 600-line file to change one function. jev-lens sends the model the outline of the file and
 that one function. The agent can ask for the rest with the `recall` tool, and the footer of the result tells it so.
 
@@ -67,8 +70,7 @@ Three results shaped the design:
 Use pi 0.84.3 or newer. The extension uses pi's built-in tool renderers for results that it does not compress.
 
 ```sh
-pi install npm:pi-jev-lens                    # from npm
-pi install git:github.com/dizk/pi-jev-lens    # or from GitHub
+pi install npm:pi-jev-lens
 ```
 
 jev needs a TypeSafe API key. You can get one at [console.typesafe.ai](https://console.typesafe.ai). jev-lens looks
@@ -95,9 +97,9 @@ The mock can compress results without API calls. It is not the jev model.
 For development, clone the repository and load it directly:
 
 ```sh
-git clone https://github.com/dizk/pi-jev-lens.git && cd pi-jev-lens && npm install
+git clone https://github.com/dizk/jev-lens.git && cd jev-lens && npm install
 echo 'TYPESAFE_API_KEY=...' > .env
-pi -e ./index.ts
+pi -e ./packages/pi/index.ts
 ```
 
 ## How it works
@@ -236,7 +238,7 @@ node --import tsx eval/presend-replay.ts <session dir>  # replay your own pi ses
 node --import tsx eval/bench/autoresearch.ts --iterations 8   # a researcher model tunes the prompts and thresholds
 ```
 
-STATUS.md is the research log. It lists every variant that we tried, its numbers, and why the defaults are what they
+[STATUS.md](../../STATUS.md) at the repository root is the research log. It lists every variant that we tried, its numbers, and why the defaults are what they
 are. In short: new code-built views moved the numbers, prompt wording did not, and the small holdout was wrong about
 code until the slice was five times larger.
 
@@ -258,17 +260,21 @@ never removes a tool result. It only rewrites it.
 
 ## Using this as a reference
 
-The pieces do not depend on pi. You can lift them into another agent:
+The pieces that do not depend on pi live in the `jev-lens` core package (`packages/core`), which any agent can use
+as a library; its README shows the API. What is where:
 
 | piece | file | depends on |
 |---|---|---|
-| candidate views | `src/views.ts` | nothing. The async code views can load `src/treesitter.ts` |
-| tree-sitter blocks and signatures | `src/treesitter.ts` | `web-tree-sitter`, `@vscode/tree-sitter-wasm`, `@binclusive/tree-sitter-kotlin-wasm` |
-| the jev questions, the state shape, the decision rule, block expansion | `src/presend.ts` | `@typesafe-ai/sdk` |
-| the parser for bash display commands | `src/shell-display.ts` | nothing |
-| the hook wiring for pi (tool_result, the recall tool, the UI) | `index.ts`, `src/ui.ts` | pi |
+| the pre-send pipeline: candidates, jev's choice, expansion, footer | `packages/core/src/lens.ts` | the rows below |
+| candidate views | `packages/core/src/views.ts` | nothing. The async code views can load `treesitter.ts` |
+| tree-sitter blocks and signatures | `packages/core/src/treesitter.ts` | `web-tree-sitter`, `@vscode/tree-sitter-wasm`, `@binclusive/tree-sitter-kotlin-wasm` |
+| the jev questions, the state shape, the decision rule, block expansion | `packages/core/src/presend.ts` | `@typesafe-ai/sdk` |
+| the parser for bash display commands | `packages/core/src/shell-display.ts` | nothing |
+| the recall tool's slicing | `packages/core/src/recall.ts` | nothing |
+| the wiring for pi (tool_result, the recall tool, the UI) | `packages/pi/index.ts`, `packages/pi/src/ui.ts` | pi |
+| the wiring for Claude Code (PostToolUse hook, MCP server) | `packages/claude-code/src/` | Claude Code |
 | the benchmark and the metrics on real trajectories | `eval/presend-score.ts`, `eval/bench/` | run `eval/bench/fetch.sh` first |
-| post-send decisions and the frozen ledger | `src/classifier.ts`, `src/policy.ts`, `src/ledger.ts` | `@typesafe-ai/sdk` |
+| post-send decisions and the frozen ledger (pi only) | `packages/core/src/classifier.ts`, `packages/pi/src/policy.ts`, `packages/pi/src/ledger.ts` | `@typesafe-ai/sdk` |
 
 The sequence is this. When a large tool result arrives, code builds the views from the text. jev says which view is
 enough and whether the exact text is needed. Code applies the selection policy. If needed, a second request expands
@@ -277,11 +283,11 @@ full text stays in the details of the result.
 
 ## Contributing and license
 
-Issues and pull requests are welcome at [github.com/dizk/pi-jev-lens](https://github.com/dizk/pi-jev-lens).
+Issues and pull requests are welcome at [github.com/dizk/jev-lens](https://github.com/dizk/jev-lens).
 `npm test` runs the unit tests with the mock classifier. `npm run typecheck` runs tsc. A change to how views are
 built or chosen must come with benchmark numbers. If the change touches code views, you must use the 500-trajectory
 slice.
 
-For npm publication through GitHub Releases, see [Release to npm](docs/releasing.md).
+For npm publication through GitHub Releases, see [Release to npm](../../docs/releasing.md).
 
 MIT, see LICENSE.

package/index.ts

@@ -18,18 +18,15 @@ import { Type } from "typebox";
 import { createBashToolDefinition, createFindToolDefinition, createGrepToolDefinition, createLsToolDefinition, createReadToolDefinition } from "@earendil-works/pi-coding-agent";
 import { Text } from "@earendil-works/pi-tui";
 import { ComparisonResult, comparisonHint, listLines, savingsLine, type CompressedRecord } from "./src/ui.ts";
-import { TypeSafeClient } from "@typesafe-ai/sdk";
-import { buildItemState, JevClassifier, MockClassifier, type Classifier } from "./src/classifier.ts";
-import { buildPresendState, decideView, DEFAULT_PROMPTS, expandRelevantBlocks, JevPresend, MockPresend, type PresendClassifier, type PromptVariant } from "./src/presend.ts";
-import { buildCandidatesAsync, extractTerms, footer } from "./src/views.ts";
-import { keyFilePath, loadConfigWithVariant, storeKey, type Config } from "./src/config.ts";
-import { Health } from "./src/health.ts";
+import {
+	buildItemState, contentText, createPresend, describeToolCall, estimateTokensOfText, Health, JevClassifier, keyFilePath,
+	Lens, loadConfigWithVariant, MockClassifier, promptsWithVariant, RECALL_DESCRIPTION, RECALL_PARAM_DESCRIPTIONS, recallMissText, sliceRecall, storeKey, toolCallsOf,
+	type CallStats, type Classifier, type Decision,
+} from "jev-lens";
 import { SecretInput } from "./src/secret-input.ts";
 import { commandCompletions, commandHelp } from "./src/commands.ts";
 import { ENTRY_TYPE, rebuildLedger } from "./src/ledger.ts";
 import { applyLedger, decideBucket, pendingPrunable, shouldApplyPending } from "./src/policy.ts";
-import { contentText, describeToolCall, estimateTokensOfText, toolCallsOf, truncate } from "./src/text.ts";
-import type { CallStats, Decision } from "./src/types.ts";
 
 interface PendingResult {
 	message: AgentMessage & { role: "toolResult" };
@@ -38,18 +35,18 @@ interface PendingResult {
 
 export default function (pi: ExtensionAPI) {
 	const { cfg, variant } = loadConfigWithVariant();
-	const prompts: PromptVariant = { ...DEFAULT_PROMPTS, ...((variant.prompts ?? {}) as Partial<PromptVariant>), viewDescriptions: { ...DEFAULT_PROMPTS.viewDescriptions, ...(((variant.prompts ?? {}) as Partial<PromptVariant>).viewDescriptions ?? {}) } };
+	const prompts = promptsWithVariant(variant);
 	const viewParams = variant.views ?? {};
 	let keySource = process.env.TYPESAFE_API_KEY === cfg.apiKey && cfg.apiKey ? "env" : cfg.apiKey ? keyFilePath() : "none";
-	let usingMock = cfg.forceMock || !cfg.apiKey;
+	let { presend, mock: usingMock } = createPresend(cfg, prompts);
 	let classifier: Classifier = usingMock ? new MockClassifier() : new JevClassifier(cfg);
-	let presend: PresendClassifier = usingMock ? new MockPresend() : new JevPresend(new TypeSafeClient({ apiKey: cfg.apiKey }), cfg.model, prompts);
+	let lens = new Lens({ cfg, presend, viewParams });
 	/** Switch from the mock to jev once a key is available (from `/jev-lens key`), without a restart. */
 	const useKey = (apiKey: string) => {
 		cfg.apiKey = apiKey;
-		usingMock = cfg.forceMock;
+		({ presend, mock: usingMock } = createPresend(cfg, prompts));
 		classifier = usingMock ? new MockClassifier() : new JevClassifier(cfg);
-		presend = usingMock ? new MockPresend() : new JevPresend(new TypeSafeClient({ apiKey }), cfg.model, prompts);
+		lens = new Lens({ cfg, presend, viewParams });
 	};
 	/** Full text of compressed tool results, by toolCallId, for the recall tool (also persisted in result details). */
 	const fullOutputs = new Map<string, { text: string; toolName: string; args: unknown; view: string }>();
@@ -365,38 +362,25 @@ export default function (pi: ExtensionAPI) {
 		if (tokens < cfg.presendMinTokens) return;
 		if (event.content.some((c) => c.type === "image")) return;
 		presendTotals.considered++;
-		const started = Date.now();
 		try {
-			const terms = extractTerms(latestUser, lastAssistantText, JSON.stringify(event.input ?? {}));
-			const cands = await buildCandidatesAsync(event.toolName, event.input, text, terms, viewParams);
+			const out = await lens.compress({ toolCallId: event.toolCallId, toolName: event.toolName, args: event.input, text, isError: event.isError, context: { firstUser, latestUser, agentText: lastAssistantText } }, signal);
 			if (epoch !== generation || signal.aborted) return;
-			if (cands.views.length < 2) {
+			if (out.reason === "no-candidates") {
 				log({ event: "presend", id: event.toolCallId, tool: event.toolName, tokens, view: "full", reason: "no-candidates" });
 				return;
 			}
-			const totalLines = text.split("\n").length;
-			const state = buildPresendState(cfg, { firstUser, latestUser, agentText: lastAssistantText, toolName: event.toolName, args: event.input, isError: event.isError, cands, totalLines, totalChars: text.length });
-			const answer = await presend.choose(state, cands.views.map((v) => v.kind), signal);
-			if (epoch !== generation || signal.aborted) return;
-			let view = decideView(answer, cands, cfg);
-			let expanded: number[] | undefined;
-			if (view.kind !== "full") {
-				const above = cands.kind === "command" ? cfg.presendSectionExpandAbove : cfg.presendExpandAbove;
-				const ex = await expandRelevantBlocks(presend, state, text, cands, view, above, signal, cands.blocks, cfg.presendSectionFloor);
-				if (epoch !== generation || signal.aborted) return;
-				if (ex) { view = ex.view; expanded = ex.probs.map((p, i) => (p > above ? i : -1)).filter((i) => i >= 0); }
-			}
 			health.success("presend");
 			status(ctx);
-			log({ event: "presend", id: event.toolCallId, tool: event.toolName, kind: cands.kind, tokens, view: view.kind, viewTokens: estimateTokensOfText(view.text), chosen: answer.choice, needsFull: answer.needsFull, p: answer.probabilities, confidence: answer.confidence, expanded, candidates: cands.views.map((v) => `${v.kind}:${v.chars}`), ms: Date.now() - started });
-			if (view.kind === "full") return;
+			const view = out.view!, answer = out.answer!;
+			log({ event: "presend", id: event.toolCallId, tool: event.toolName, kind: out.kind, tokens, view: view.kind, viewTokens: out.sentTokens, chosen: answer.choice, needsFull: answer.needsFull, p: answer.probabilities, confidence: answer.confidence, expanded: out.expanded, candidates: out.candidates, ms: out.ms });
+			if (!out.compressed) return;
 			presendTotals.compressed++;
-			presendTotals.tokensSaved += tokens - estimateTokensOfText(view.text);
+			presendTotals.tokensSaved += tokens - out.sentTokens;
 			fullOutputs.set(event.toolCallId, { text, toolName: event.toolName, args: event.input, view: view.kind });
-			remember({ id: event.toolCallId, toolName: event.toolName, args: event.input, kind: cands.kind, view: view.kind, tokensBefore: tokens, tokensAfter: estimateTokensOfText(view.text), full: text, sent: view.text, included: view.included, needsFull: answer.needsFull, pFull: answer.probabilities.full, recalls: 0, at: Date.now() });
-			const details = { ...((event.details as object) ?? {}), jevLens: { full: text, view: view.kind, kind: cands.kind, args: event.input, p: answer.probabilities, needsFull: answer.needsFull, included: view.included } };
+			remember({ id: event.toolCallId, toolName: event.toolName, args: event.input, kind: out.kind!, view: view.kind, tokensBefore: tokens, tokensAfter: out.sentTokens, full: text, sent: view.text, included: view.included, needsFull: answer.needsFull, pFull: answer.probabilities.full, recalls: 0, at: Date.now() });
+			const details = { ...((event.details as object) ?? {}), jevLens: { full: text, view: view.kind, kind: out.kind, args: event.input, p: answer.probabilities, needsFull: answer.needsFull, included: view.included } };
 			status(ctx);
-			return { content: [{ type: "text", text: view.text + footer(view, event.toolCallId, totalLines) }], details };
+			return { content: [{ type: "text", text: out.text }], details };
 		} catch (err) {
 			if (epoch !== generation || signal.aborted) return;
 			health.failure("presend", err);
@@ -409,11 +393,11 @@ export default function (pi: ExtensionAPI) {
 	pi.registerTool({
 		name: "recall",
 		label: "Recall",
-		description: "Return the full output of an earlier tool call that jev-lens showed in a reduced view (or that was pruned). Pass the id from the [jev-lens: ...] note. Optionally restrict to a line range \"a-b\" or to lines matching a pattern (case-insensitive substring or /regex/).",
+		description: RECALL_DESCRIPTION,
 		parameters: Type.Object({
-			id: Type.String({ description: "toolCallId from the jev-lens note" }),
-			lines: Type.Optional(Type.String({ description: "Line range like 120-180 (1-based, inclusive)" })),
-			pattern: Type.Optional(Type.String({ description: "Only lines matching this substring or /regex/, with 2 lines of context" })),
+			id: Type.String({ description: RECALL_PARAM_DESCRIPTIONS.id }),
+			lines: Type.Optional(Type.String({ description: RECALL_PARAM_DESCRIPTIONS.lines })),
+			pattern: Type.Optional(Type.String({ description: RECALL_PARAM_DESCRIPTIONS.pattern })),
 		}),
 		async execute(_toolCallId, params) {
 			presendTotals.recalls++;
@@ -421,28 +405,9 @@ export default function (pi: ExtensionAPI) {
 			if (rec) rec.recalls++;
 			const hit = fullOutputs.get(params.id);
 			log({ event: "recall", id: params.id, found: !!hit, lines: params.lines, pattern: params.pattern });
-			if (!hit) return { content: [{ type: "text", text: `No stored output for id ${params.id}. Re-run the original tool instead.` }], details: { id: params.id, lines: 0 } };
-			const all = hit.text.split("\n");
-			let idx = all.map((_, i) => i);
-			if (params.lines) {
-				const m = params.lines.match(/^(\d+)\s*-\s*(\d+)$/);
-				if (!m) return { content: [{ type: "text", text: "lines must look like 120-180" }], details: { id: params.id, lines: 0 } };
-				const a = Math.max(1, Number(m[1])), b = Math.min(all.length, Number(m[2]));
-				idx = idx.filter((i) => i + 1 >= a && i + 1 <= b);
-			}
-			if (params.pattern) {
-				let test: (l: string) => boolean;
-				const rx = params.pattern.match(/^\/(.*)\/([a-z]*)$/);
-				if (rx) { const re = new RegExp(rx[1], rx[2].includes("i") ? rx[2] : rx[2] + "i"); test = (l) => re.test(l); }
-				else { const needle = params.pattern.toLowerCase(); test = (l) => l.toLowerCase().includes(needle); }
-				const keep = new Set<number>();
-				for (const i of idx) if (test(all[i])) for (let j = Math.max(0, i - 2); j <= Math.min(all.length - 1, i + 2); j++) keep.add(j);
-				idx = idx.filter((i) => keep.has(i));
-			}
-			const width = String(all.length).length;
-			const body = idx.length === all.length ? hit.text : idx.map((i) => `${String(i + 1).padStart(width)}│ ${all[i]}`).join("\n");
-			const header = idx.length === all.length ? "" : `[${idx.length} of ${all.length} lines from ${hit.toolName} ${truncate(JSON.stringify(hit.args ?? {}), 80)}]\n`;
-			return { content: [{ type: "text", text: header + body }], details: { id: params.id, lines: idx.length } };
+			if (!hit) return { content: [{ type: "text", text: recallMissText(params.id) }], details: { id: params.id, lines: 0 } };
+			const slice = sliceRecall(hit, params);
+			return { content: [{ type: "text", text: slice.text }], details: { id: params.id, lines: slice.count } };
 		},
 	});
 

package/package.json

@@ -1,16 +1,17 @@
 {
   "name": "pi-jev-lens",
-  "version": "0.4.1",
+  "version": "0.5.0",
   "description": "pi extension that compresses large tool results before they reach the model: jev picks the view (outline, relevant blocks, sections, signals, testlog), full text stays recallable",
   "author": "Didrik Rognstad",
   "license": "MIT",
   "repository": {
     "type": "git",
-    "url": "git+https://github.com/dizk/pi-jev-lens.git"
+    "url": "git+https://github.com/dizk/jev-lens.git",
+    "directory": "packages/pi"
   },
-  "homepage": "https://github.com/dizk/pi-jev-lens#readme",
+  "homepage": "https://github.com/dizk/jev-lens#readme",
   "bugs": {
-    "url": "https://github.com/dizk/pi-jev-lens/issues"
+    "url": "https://github.com/dizk/jev-lens/issues"
   },
   "keywords": [
     "pi-package",
@@ -31,37 +32,19 @@
     "index.ts",
     "src/",
     "README.md",
-    "LICENSE",
-    "STATUS.md"
+    "LICENSE"
   ],
-  "scripts": {
-    "test": "vitest run",
-    "replay": "node --import tsx eval/replay.ts",
-    "typecheck": "tsc --noEmit"
-  },
   "pi": {
     "extensions": [
       "./index.ts"
     ]
   },
   "dependencies": {
-    "@binclusive/tree-sitter-kotlin-wasm": "^0.1.0",
-    "@typesafe-ai/sdk": "^0.6.0",
-    "@vscode/tree-sitter-wasm": "^0.3.1",
-    "web-tree-sitter": "^0.27.0"
+    "jev-lens": "^0.5.0"
   },
   "peerDependencies": {
     "@earendil-works/pi-coding-agent": ">=0.84.3",
     "@earendil-works/pi-tui": "*",
     "typebox": "*"
-  },
-  "devDependencies": {
-    "@types/node": "^26.6.1",
-    "tsx": "^4.23.13",
-    "typescript": "^7.0.2",
-    "vitest": "^5.0.1",
-    "@earendil-works/pi-coding-agent": "^0.84.3",
-    "@earendil-works/pi-tui": "^0.84.3",
-    "typebox": "^1.3.33"
   }
 }

package/src/ledger.ts

@@ -1,4 +1,4 @@
-import type { Decision } from "./types.ts";
+import type { Decision } from "jev-lens";
 
 export const ENTRY_TYPE = "jev-lens";
 

package/src/policy.ts

@@ -1,7 +1,7 @@
 import type { AgentMessage, ToolResultMessage } from "./pi-types.ts";
-import type { Config } from "./config.ts";
-import type { Bucket, Decision } from "./types.ts";
-import { contentText, estimateTokensOfText } from "./text.ts";
+import type { Config } from "jev-lens";
+import type { Bucket, Decision } from "jev-lens";
+import { contentText, estimateTokensOfText } from "jev-lens";
 
 export const STUB_PREFIX = "[jev-lens pruned:";