pi-repl-py 0.6.6 → 0.6.8

package/README.md

@@ -64,22 +64,25 @@ On **Termux (Android)**, the `postinstall` venv build can fail because `ipykerne
 
 A **helper** is a `.py` file that gets exec'd into every kernel, so whatever it defines
 (like functions, classes, constants, imports, or a module that manages a tricky piece of
-complexity) is available in the workspace. Drop a file in the one helpers directory and
-restart the session; e.g. `helpers/double.py` defining `def double(x)` becomes callable as
-`double(...)`. It ships **empty** (shell and file IO are already plain Python), so a fresh
-install preloads nothing until you add one. Each helper's `helper_description` is shown to
-the model verbatim; the full contract lives in [docs/helpers.md](docs/helpers.md).
+complexity) is available in the workspace. Drop a file in a `.pi/helpers/` directory in
+your project (or `~/.pi/agent/pi-repl/helpers/` for every project) and restart the session;
+e.g. `helpers/double.py` defining `def double(x)` becomes callable as `double(...)`. Global
+helpers ship **empty** (shell and file IO are already plain Python), so a fresh install
+preloads nothing until you add one. Project helpers shadow same-named global ones. Each
+helper's `helper_description` is shown to the model verbatim; the full contract lives in
+[docs/helpers.md](docs/helpers.md).
 
-Everything the extension keeps lives under one folder in your home directory:
+The extension keeps its runtime under one folder in your home directory:
 
 ```
 ~/.pi/agent/pi-repl/
   venv/         the Python interpreter + ipykernel
-  helpers/      your helpers (created empty on install; every *.py loads)
+  helpers/      global helpers (created empty on install; every *.py loads)
   state/        per-session namespace snapshots
 ```
 
-The helpers directory is fixed at `~/.pi/agent/pi-repl/helpers`. No config file.
+Project helpers live in `<project>/.pi/helpers/` instead; both tiers are scanned with the
+project one first. No config file.
 
 Changing a helper (adding/removing a file, renaming one with a `_` prefix) needs a
 **session restart / `/reload`**: the prompt list is built when `execute` is registered and

package/docs/ARCHITECTURE.md

@@ -175,11 +175,13 @@ watchdog timeout.
   state/        per-session namespace snapshots
 ```
 
-The helpers directory is fixed at `~/.pi/agent/pi-repl/helpers` (matching the kernel's
-`readHelperSources` default), so both sides are guaranteed to read the same directory. The
-venv is built automatically, and the interpreter follows the order above. No setting is
-needed. The per-cell silence watchdog is off by default (`PI_REPL_TIMEOUT_MS=0`: a silent
-but working cell may run on).
+Helpers merge project and global dirs: `resolveHelperDirs` walks from the working
+directory up to the git root collecting `.pi/helpers/`, then appends
+`~/.pi/agent/pi-repl/helpers`. Both the prompt loader and the kernel's `readHelperSources`
+walk the same ordered list with first-seen-wins, so a project helper shadows the same-named
+global one and both sides are guaranteed to agree. The venv is built automatically, and the
+interpreter follows the order above. No setting is needed. The per-cell silence watchdog is
+off by default (`PI_REPL_TIMEOUT_MS=0`: a silent but working cell may run on).
 
 ## Reference documentation
 

package/docs/helpers.md

@@ -9,13 +9,26 @@ name or any other public name in the file.
 
 ## Where helpers live
 
+Helpers come from two places: a **project** directory and a **global** directory.
+
 ```text
-~/.pi/agent/pi-repl/helpers/
+<project>/.pi/helpers/                 project helpers (looked up from the working dir)
+~/.pi/agent/pi-repl/helpers/           global helpers (every project)
 ```
 
-The directory is created empty when pi-repl is installed. Every `.py` file in it is loaded when the evaluator starts. Files whose names begin with `_` are ignored.
+The global directory is created empty when pi-repl is installed. In a project, any
+`.pi/helpers/` directory is picked up by walking up from the working directory to the
+git repo root, so a helper works no matter how deep in the project you are.
+
+Every `.py` file found is loaded when the evaluator starts; files whose names begin with
+`_` are ignored.
+
+The two tiers merge: a project helper **shadows** a same-named global helper, and global
+helpers fill in whatever the project does not define. One file name appears once in the
+tool prompt and once in the kernel.
 
-After adding, changing, renaming, or disabling a helper, run `/reload` or start a new `pi --repl` session. The running evaluator does not watch the directory for changes.
+After adding, changing, renaming, or disabling a helper, run `/reload` or start a new
+`pi --repl` session. The running evaluator does not watch the directories for changes.
 
 ## A small function helper
 
@@ -109,7 +122,7 @@ Use docstrings for argument details, defaults, return values, errors, environmen
 
 ## How loading works
 
-At startup, two parts of pi-repl read the same helper directory:
+At startup, two parts of pi-repl read the same merged helper list (project dirs first, global last):
 
 1. The kernel executes each eligible `.py` file. Its definitions become names in the Python workspace.
 2. The host reads `helper_description` to build the helper guidance shown to the model.
@@ -196,7 +209,7 @@ The loader skips it. Rename it back and reload when you want it again.
 
 ## Checklist
 
-- [ ] The file is in `~/.pi/agent/pi-repl/helpers/`.
+- [ ] The file is in `~/.pi/agent/pi-repl/helpers/` (global) or in `<project>/.pi/helpers/` (project-scoped).
 - [ ] Its public names and call shapes are clear.
 - [ ] `helper_description` is short enough for every-turn context.
 - [ ] Detailed behavior is in docstrings.

package/index.ts

@@ -125,7 +125,7 @@ export default function (pi: ExtensionAPI) {
 		label: "execute",
 		description: EXECUTE_DESCRIPTION,
 		promptSnippet: EXECUTE_PROMPT_SNIPPET,
-		promptGuidelines: buildExecutePromptGuidelines(),
+		promptGuidelines: buildExecutePromptGuidelines(process.cwd()),
 		parameters: executeSchema,
 		renderShell: "self",
 		renderCall(args, theme, context) {

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "pi-repl-py",
-	"version": "0.6.6",
+	"version": "0.6.8",
 	"type": "module",
 	"description": "A pi extension with a single tool: execute, running a TypeScript host with a persistent Python (ipykernel) evaluator and a user-configurable toolbox of functions.",
 	"keywords": [

package/src/engine/helpers-locate.ts

@@ -0,0 +1,24 @@
+// --- shared helper-dir resolution: prompt and kernel must read the same ordered list ---
+import { existsSync } from "node:fs";
+import { homedir } from "node:os";
+import { dirname, join, resolve } from "node:path";
+
+const GLOBAL_HELPERS_DIR = join(homedir(), ".pi", "agent", "pi-repl", "helpers");
+
+/** Ordered candidate dirs: nearest .pi/helpers up to the git root, then the global dir last. */
+export function resolveHelperDirs(cwd?: string, globalDir?: string): string[] {
+	const dirs: string[] = [];
+	if (cwd) {
+		let cur = resolve(cwd);
+		for (;;) {
+			const d = join(cur, ".pi", "helpers");
+			if (existsSync(d)) dirs.push(d);
+			if (existsSync(join(cur, ".git"))) break;
+			const parent = dirname(cur);
+			if (parent === cur) break;
+			cur = parent;
+		}
+	}
+	dirs.push(globalDir ?? GLOBAL_HELPERS_DIR);
+	return dirs;
+}

package/src/engine/index.ts

@@ -20,12 +20,12 @@ function resolvePythonPath(_cwd: string | undefined): string {
 	return process.env.PYTHON ?? "python3";
 }
 
-const DEFAULT_MAX_OUTPUT_CHARS = 65536;
+const DEFAULT_MAX_OUTPUT_CHARS = 46080;
 /** Per-line cap: one genuinely oversized line must not own the channel budget, while legitimately long
  * REPL output (JSON, reprs, errors) still fits under the cap in one piece. Generous enough that only
  * pathological giant lines are trimmed, unlike pi's grep where the line cap keeps matches terse. */
 export const MAX_OUTPUT_LINE_CHARS = 4096;
-const ABORT_GRACE_MS = 500;
+const ABORT_GRACE_MS = 20_000;
 const DEFAULT_SNAPSHOT_DEBOUNCE_MS = 1500;
 
 interface EngineExecuteError {
@@ -50,7 +50,7 @@ export interface ExecuteOptions {
 	/** Aborting cancels the cell via kernel interrupt; the namespace is preserved. */
 	signal?: AbortSignal;
 	onStream?: (chunk: string, name: "stdout" | "stderr") => void;
-	/** Cap stdout / stderr / result at this many characters. Default 65536. */
+	/** Cap stdout / stderr / result at this many characters. Default 45K. */
 	maxOutputChars?: number;
 }
 

package/src/engine/kernel.ts

@@ -3,8 +3,9 @@
 import { type ChildProcess, spawn } from "node:child_process";
 import { randomUUID } from "node:crypto";
 import { existsSync, readdirSync, readFileSync, rmSync } from "node:fs";
-import { homedir, tmpdir } from "node:os";
+import { tmpdir } from "node:os";
 import { join } from "node:path";
+import { resolveHelperDirs } from "./helpers-locate.js";
 import {
 	type ConnectionFile,
 	executeRequest,
@@ -62,18 +63,22 @@ function resolveCwd(requested?: string): string {
 	if (requested && existsSync(requested)) return requested;
 	return process.cwd();
 }
-function readHelperSources(dir?: string): { name: string; source: string }[] {
-	// --- one fixed dir, resolved like the prompt side (helpers.ts) so both always agree ---
-	const d = dir ?? join(homedir(), ".pi", "agent", "pi-repl", "helpers");
-	if (!existsSync(d)) return [];
+function readHelperSources(dirs: string[]): { name: string; source: string }[] {
+	// --- merged dirs come pre-ordered (project first, global last); first-seen name wins ---
+	const seen = new Set<string>();
 	const out: { name: string; source: string }[] = [];
-	for (const file of readdirSync(d).sort()) {
-		if (!file.endsWith(".py")) continue;
-		const name = file.slice(0, -3);
-		if (!/^[A-Za-z_]\w*$/.test(name) || name.startsWith("_")) continue;
-		try {
-			out.push({ name, source: readFileSync(join(d, file), "utf8") });
-		} catch {}
+	for (const d of dirs) {
+		if (!existsSync(d)) continue;
+		for (const file of readdirSync(d).sort()) {
+			if (!file.endsWith(".py")) continue;
+			const name = file.slice(0, -3);
+			if (!/^[A-Za-z_]\w*$/.test(name) || name.startsWith("_")) continue;
+			if (seen.has(name)) continue;
+			seen.add(name);
+			try {
+				out.push({ name, source: readFileSync(join(d, file), "utf8") });
+			} catch {}
+		}
 	}
 	return out;
 }
@@ -184,7 +189,9 @@ export class KernelClient {
 
 	private constructor(conn: ConnectionFile, opts: KernelOptions) {
 		this.session = new JupyterSession({ key: conn.key });
-		this.helperSources = readHelperSources(opts.env?.PI_HELPERS_DIR);
+		this.helperSources = opts.env?.PI_HELPERS_DIR
+			? readHelperSources([opts.env.PI_HELPERS_DIR])
+			: readHelperSources(resolveHelperDirs(opts.cwd, opts.env?.PI_HELPERS_GLOBAL_DIR));
 		this.timeoutMs = opts.timeoutMs ?? 0;
 	}
 

package/src/extension/helpers.ts

@@ -1,8 +1,9 @@
-/** Loads helpers from the ONE fixed dir; `helper_description` surfaces verbatim (no signature parsing). */
+/** Loads helpers from project then global dirs; `helper_description` surfaces verbatim (no signature parsing). */
 
 import { existsSync, readdirSync, readFileSync } from "node:fs";
 import { homedir } from "node:os";
 import { join } from "node:path";
+import { resolveHelperDirs } from "../engine/helpers-locate.js";
 
 const DEFAULT_HELPERS_DIR = join(homedir(), ".pi", "agent", "pi-repl", "helpers");
 
@@ -11,36 +12,49 @@ interface HelperEntry {
 	description: string; // full helper_description body, "" if absent
 }
 
-/** Extract `helper_description = """..."""` (or `'''`) verbatim; no signature parsing. */
+/** Extract `helper_description` verbatim; no signature parsing. */
 function parseDescription(source: string): string {
 	const m = source.match(/helper_description\s*=\s*("""|''')([\s\S]*?)\1/);
 	return m ? m[2].trim() : "";
 }
 
-/** Load {name → entry} for each non-underscore *.py in the helpers dir. */
-function loadHelperEntries(dir?: string): HelperEntry[] {
-	const d = dir ?? DEFAULT_HELPERS_DIR;
-	if (!existsSync(d)) return [];
+/** Merge entries from ordered dirs; first-seen name wins, so a project helper shadows the global one. */
+function loadHelperEntries(dirs: string[]): HelperEntry[] {
+	const seen = new Set<string>();
 	const entries: HelperEntry[] = [];
-	for (const file of readdirSync(d).sort()) {
-		if (!file.endsWith(".py")) continue;
-		const name = file.slice(0, -3);
-		if (!/^[A-Za-z_]\w*$/.test(name)) continue;
-		// --- underscore-prefixed files are neither loaded nor advertised ---
-		if (name.startsWith("_")) continue;
-		try {
-			const source = readFileSync(join(d, file), "utf8");
-			entries.push({ name, description: parseDescription(source) });
-		} catch {}
+	for (const d of dirs) {
+		if (!existsSync(d)) continue;
+		for (const file of readdirSync(d).sort()) {
+			if (!file.endsWith(".py")) continue;
+			const name = file.slice(0, -3);
+			if (!/^[A-Za-z_]\w*$/.test(name)) continue;
+			// --- underscore-prefixed files are neither loaded nor advertised ---
+			if (name.startsWith("_")) continue;
+			if (seen.has(name)) continue;
+			seen.add(name);
+			try {
+				const source = readFileSync(join(d, file), "utf8");
+				entries.push({ name, description: parseDescription(source) });
+			} catch {}
+		}
 	}
 	return entries;
 }
 
-/** The prompt-facing list, one bullet per loaded file (verbatim description, or an introspection pointer). */
+/** The prompt-facing list for ONE dir: verbatim description, or an introspection pointer. */
 export function buildHelpersMap(dir?: string): string[] {
-	return loadHelperEntries(dir).map((t) =>
+	return loadHelperEntries([dir ?? DEFAULT_HELPERS_DIR]).map((t) =>
 		t.description
-			? `- ${t.description.replace(/\n/g, "\n  ")}`
-			: `- ${t.name} (no description, inspect it with print(${t.name}.__doc__))`,
+			? t.description.replace(/\n/g, "\n  ")
+			: `${t.name} (no description, inspect it with print(${t.name}.__doc__))`,
+	);
+}
+
+/** The prompt-facing list at a cwd: project .pi/helpers first (up to the git root), global fallback, project shadows. */
+export function buildHelpersMapForCwd(cwd: string, globalDir?: string): string[] {
+	return loadHelperEntries(resolveHelperDirs(cwd, globalDir)).map((t) =>
+		t.description
+			? t.description.replace(/\n/g, "\n  ")
+			: `${t.name} (no description, inspect it with print(${t.name}.__doc__))`,
 	);
 }

package/src/extension/prompt.ts

@@ -1,52 +1,23 @@
-// --- execute tool: the model-facing contract + workspace doctrine (pure, no pi/helper dep) ---
+// --- execute tool: the model-facing contract, shaped exactly like pi's built-in tools ---
+// description = rich short behavior; promptSnippet = one-liner; guidelines = flat bullets.
 
 export const executeToolDescription =
-	"Execute Python cells in a persistent ipython kernel; state survives across cells and turns, replacing " +
-	"the default read, bash, edit, write, and search tools. Let a cell's returned value prove the work, " +
-	"not prose restating it.";
+	"Execute Python cells in a persistent Python shell that is your entire workspace: it is where you read, " +
+	"write, run, and move, all in one instrument. The state you build, files, and subprocesses survive " +
+	"from one call to the next. Returns stdout, stderr, and the value of the last expression. Output is " +
+	"truncated to 45K with a marker.";
 
-export const executePromptSnippet =
-	"Execute Python cells in a persistent ipython kernel (replaces read, bash, edit, write, and search; state survives across cells and turns)";
+export const executePromptSnippet = "Execute Python in a persistent shell (read, write, run, search, and more)";
 
-// --- the workspace doctrine riding the execute tool ---
+// --- the model-facing guidelines, flat bullets like pi's own tool contributions ---
 export function buildPromptGuidelines(preloaded: string[]): string[] {
 	return [
-		"## Your only workspace",
-		"You are an engineer in a persistent Python REPL. `execute` is the only callable surface, it replaces read, bash, edit, write, and search. What you define (variables, functions, imports) survives across cells and turns, so define any function once and call it in later cells. The work is proven by the result each cell returns, and by nothing else.",
-		"",
-		"## Reason, then say, then stop",
-		"Reason inside the cell, not the transcript: do the thinking in variables and filters, return only the outcome. End on an assignment, a bare expression auto-prints. Keep the reasoning you need, drop the rest, the returned result is the evidence of the work, not the words around it.",
-		"",
-		"## The environment answers you",
-		"The cell's output is the ground truth, what actually ran, what errored, what came back. Trust it over any narrative: if a cell already proved it, point at that. When you're unsure what a fetch contains, read a slice, don't guess and don't dump it whole to 'check'.",
-		"",
-		"## Gather, slice, decide",
-		"Fetch into a variable, never into the transcript. Search results, reads, command output, file contents, assign. A bare expression prints, so end those cells on the assignment. Then advance on a bounded slice: print only the fragment that decides the next step, hold the rest in the variable, peel into the pieces you need without re-fetching, and when the reasoning lands, print the conclusion.",
-		"",
-		"Reading whole is fine when the task needs all of it, hold it and reason on it; the point isn't to never read fully, it's to not re-fetch the same big thing twice.",
-		"",
-		"## Output format",
-		"In reply text: the conclusion and the handful of results that prove it, the slice you acted on, the returned value, a one-line takeaway. Do not transcribe the run, restate every variable, or narrate what the cell already showed.",
-		"",
-		"## Edits and repo discipline",
-		"Surgical old-text/new-text: read the region, fix an exact unique anchor that appears once, replace, verify. Many small edits over one big rewrite, a parse error can strand an anchor; after an error, read the file back from disk first. Make the smallest valid change, preserve conventions, never invent files, APIs, conventions, or test results. Prune generated dirs when walking trees. Pass a `timeout` to any `subprocess.run(...)`, a silent cell must die, not hang.",
-		"",
-		"## Print is expensive",
-		"Every token you print is spent from the context you need for the turns to come, and it never comes back. Treat printing as debt, not reward. Print only the exact fragment the next decision consumes and hold the whole in a variable. Every other print is waste, it buys nothing and closes the room you have left to think. Ask before you print: does this decide the next step, or is it just noise? When it is noise, cut it. When in doubt, cut it. A tight transcript is the sign you actually worked; a bloated one is the sign you did not.",
-		"",
-		...(preloaded.length
-			? [
-					"## Helpers",
-					"These helpers are already defined in the workspace namespace. Use them by name as you would any other loaded function, class, or variable. Their code already executed at kernel boot. Descriptions appear below.",
-					"",
-					...preloaded,
-					"",
-				]
-			: []),
-		"## Environment & rescue",
-		"The evaluator runs in a project-local venv, not the system Python. Do not install a project's dependencies into the evaluator; run external projects through their own interface. If output begins with `<repl_engine_reset>`, the kernel rebuilt, re-verify a revived variable before reusing it.",
-		"",
-		"## These rules are the surface",
-		"The rules above are the surface of how this workspace works, not the whole of it. Internalize their intent, apply it to cases they don't mention, and follow them diligently.",
+		"State persists across cells, so keep building on it.",
+		"Find, filter, fetch, read: narrow the output in Python, then print the exact slice you need.",
+		"Keep the result in a variable and reuse it, instead of re-fetching the same thing.",
+		"Make surgical, precise changes over rewrites or whole-file dumps: a small unique anchor, replace, verify, read the file back before trusting it.",
+		"The evaluator runs in a project-local venv. Do not install a project's dependencies into it; run external projects through their own interface. If output begins with <repl_engine_reset>, the kernel rebuilt; re-verify a revived variable.",
+		...(preloaded.length ? ["Preloaded helpers, use them as any loaded function or variable:", ...preloaded] : []),
+		"Be concise.",
 	];
 }

package/src/extension/tool-meta.ts

@@ -1,15 +1,15 @@
 // --- tool-meta: thin surface assembling the execute tool's prompt from pure modules ---
 // --- the model contract lives in prompt.ts; only the helpers wiring stays here ---
 
-import { buildHelpersMap } from "./helpers.js";
+import { buildHelpersMap, buildHelpersMapForCwd } from "./helpers.js";
 import { buildPromptGuidelines, executePromptSnippet, executeToolDescription } from "./prompt.js";
 
 export const EXECUTE_DESCRIPTION = executeToolDescription;
 export const EXECUTE_PROMPT_SNIPPET = executePromptSnippet;
 
-// --- build the guidelines from the one helpers dir (default ~/.pi/agent/pi-repl/helpers) ---
-export function buildExecutePromptGuidelines(): string[] {
-	const map = buildHelpersMap();
+// --- build the guidelines from project + global helper dirs ---
+export function buildExecutePromptGuidelines(cwd?: string): string[] {
+	const map = cwd ? buildHelpersMapForCwd(cwd) : buildHelpersMap();
 	const preloaded = map.length > 0 ? map : [];
 	return buildPromptGuidelines(preloaded);
 }