pi-repl-py 0.1.1 → 0.2.1

package/README.md

@@ -21,7 +21,7 @@ the kernel stayed alive.
 
 ```bash
 # from a clone, one-time setup
-just setup            # npm install + a project-local .venv with the guest deps
+just setup            # npm install + a project-local .venv with ipykernel
 
 # run a session
 pi --repl
@@ -34,49 +34,49 @@ A plain `pi` session is untouched; the extension is dormant until `--repl` is pa
 
 `npm install` runs a `postinstall` that creates the Python venv the evaluator needs, at a stable
 per-user path (`~/.pi/agent/pi-repl/venv`). If `python3` or the network is missing, it prints a
-clear notice. How the interpreter is resolved is in [docs/philosophy.md](docs/philosophy.md).
+clear notice. How the interpreter is resolved is in [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 
 ## What you get
 
 - **A persistent namespace.** Variables, functions, imports, and data survive across cells and
   turns; snapshots preserve them across a best-effort restart.
 - **A real `ipython` kernel**, not a hand-rolled `exec` loop.
-- **Shell as values.** `bash("git log --oneline")` returns a `CompletedProcess` you read
-  `.stdout`/`.stderr`/`.returncode` on.
+- **Shell and file IO as plain Python.** `!cmd` and `%%bash` run shell fire-and-forget,
+  `subprocess.run(...)` brings the result back into a variable, and `open()` / `pathlib`
+  read and write files — no wrapper API to learn, and nothing extra to describe to the model.
 - **Error survival.** A cell that throws reports the traceback and the kernel keeps going.
 - **An honest evaluator.** If it restarts, it names what state it could revive and what it lost, so you don't trust memory that's gone.
 
-## The toolbox
+## Helpers
 
-A small set of Python functions is preloaded into every kernel and surfaced to the
-model through the `execute` tool's prompt guidance (their signatures + one-line
-summaries are listed there, and `ls()`/`help()` discover them at runtime), so the
-model can call `read`, `write`, `edit`, and `bash` without reimplementing them.
-Set `toolboxDir` to point at your own folder; its functions are loaded **in addition to** the built-ins, and a file there with the **same name** as a built-in overrides it.
+A **helper** is a `.py` file that gets exec'd into every kernel, so whatever it defines —
+functions, classes, constants, imports, or a module that manages a tricky piece of
+complexity — is available in the workspace. Drop a file in the one helpers directory and
+restart the session; e.g. `helpers/double.py` defining `def double(x)` becomes callable as
+`double(...)`. It ships **empty** (shell and file IO are already plain Python), so a fresh
+install preloads nothing until you add one. Each helper's `helper_description` is shown to
+the model verbatim; the full contract lives in [docs/helpers.md](docs/helpers.md).
 
 Everything the extension keeps lives under one folder in your home directory:
 
 ```
 ~/.pi/agent/pi-repl/
-  config.json   settings (toolboxDir, pythonPath, timeoutMs)
   venv/         the Python interpreter + ipykernel
-  functions/    your custom toolbox functions, if any
+  helpers/      your helpers (created empty on install; every *.py loads)
   state/        per-session namespace snapshots
 ```
 
-These functions are the evaluator's standard file-and-shell surface; the model reaches for them as its builtins and composes its own reusable tools on top.
+The helpers directory is fixed at `~/.pi/agent/pi-repl/helpers` — no config file.
 
-The function list shown to the model is built when the `execute` tool is
-registered, so changing the toolbox (adding/removing a file, renaming one with a
-`_` prefix) needs a **session restart / `/reload`** for the prompt to reflect it —
-the kernel also only loads the toolbox at boot.
-
-- Adding a function (the file contract, docstrings, disabling): [docs/how-to-functions.md](docs/how-to-functions.md)
+Changing a helper (adding/removing a file, renaming one with a `_` prefix) needs a
+**session restart / `/reload`**: the prompt list is built when `execute` is registered and
+the kernel execs helpers only at boot.
 
 ## Configuration
 
-`~/.pi/agent/pi-repl/config.json` (or `$PI_REPL_CONFIG`) sets `toolboxDir`, `pythonPath`, and timeouts.
-Full keys and path rules: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
+There is deliberately no configuration file. Everything is arranged under
+`~/.pi/agent/pi-repl/`: the venv, the fixed helpers dir, and the session state.
+The Python interpreter is auto-resolved (the venv, else `$PYTHON`/`python3`).
 
 ## More
 
@@ -86,7 +86,7 @@ Full keys and path rules: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
 ## It is not
 
 - A sandbox. The kernel runs with your permissions; the toolbox trusts you.
-- A subagent framework. There are no `rlm.run` subagents; spawn a process with `bash()`.
+- A subagent framework. There is no `repl.run` API; spawn a process with `subprocess.run`.
 - A pi tool-rack. It is one `execute` tool with functions inside.
 
 ## License

package/docs/ARCHITECTURE.md

@@ -1,142 +1,182 @@
 # Architecture
 
-Two processes: the host lives inside pi, the guest owns the Python workspace.
+pi-repl runs in **two processes**: the host lives inside pi and talks the standard Jupyter
+protocol directly to a real `ipykernel` subprocess. There is no middleman and no invented
+framing between them.
 
 ```
 pi
- └─ extension (index.ts)        registers `execute`, dormant until --repl
-     └─ EngineManager (src/engine/index.ts)  spawn host: snapshots, teardown
-         │  stdin  ──▶  protocol commands (run / snapshot / restore / ping)
-         │  fd 3   ◀──  stream, done, snapshot_result, ...
-         └─ guest.py ▶ jupyter_client ▶ a real ipython kernel (subprocess)
+ └─ extension (index.ts)              registers `execute`; dormant until --repl
+     └─ EngineManager (src/engine/index.ts)   venv resolution, lazy spawn,
+         │                                   the call queue, snapshots,
+         │                                   abort grace, teardown
+         └─ KernelClient (src/engine/kernel.ts)   one ipykernel subprocess
+             ├─ ZMTP 3.0 (src/engine/zmtp.ts)     the wire protocol, by hand
+             ├─ Jupyter session (src/engine/session.ts)   framing + HMAC + JSON
+             └─ python -m ipykernel -f <connection-file>   the evaluator
 ```
 
-The host is TypeScript; the evaluator is Python in its own process. Splitting
-them is what makes a bad cell survivable: a cell can raise, leak memory, or
-wedge the guest without taking pi down, and the host, being not the thing
-that failed, always gets to report what happened.
+The host is TypeScript; the evaluator is Python in its own process. That split is what makes
+a bad cell survivable: a cell can raise, leak memory, or wedge the kernel without taking pi
+down — and the host, being not the thing that failed, always gets to report what happened.
+
+## Why the host speaks ZMTP itself
+
+The obvious way for a TypeScript host to drive an `ipykernel` is to load a ZMQ client
+library. That does not work here: libzmq's native Node bindings crash `bun`. So an earlier
+design put a Python middleman (`guest.py`) between the host and the kernel, translating a
+private JSON protocol over a file descriptor into the real Jupyter protocol.
+
+The current design removes the middleman. Instead of working around the missing library, the
+host implements the small slice of ZMTP 3.0 a Jupyter client actually needs — a DEALER
+socket for the shell and control channels, a SUB socket for iopub (`src/engine/zmtp.ts`).
+The payoff:
+
+- **one process boundary** instead of two;
+- **one standard protocol** (Jupyter) instead of a private one on top of it;
+- **no invented framing** to maintain;
+- **HMAC signed and verified by the host itself.** The old design carried a nonce scheme to
+  stop a cell from forging its own completion signal; with the host signing every message
+  with the kernel's HMAC key, that concern is moot.
 
 ## The Python environment (the venv)
 
-The evaluator is a real `ipython` kernel, so it needs a Python environment with
-`ipykernel` + `jupyter_client`. You cannot fake that with a script; it is a
-hard runtime dependency.
+The evaluator is a real `ipykernel` kernel, so it needs a Python environment with
+`ipykernel` installed. That is a hard runtime dependency — you cannot fake it with a script.
+(`jupyter_client` is *not* needed: the host is the client.)
 
 When installed as a pi package, `npm install` runs `postinstall`
-(`scripts/setup-venv.mjs`), which creates a stable per-user venv:
+(`scripts/setup-venv.mjs`), which builds a stable per-user venv:
 
 ```
 ~/.pi/agent/pi-repl/venv/bin/python3
 ```
 
-That path is stable across updates because it sits outside the ephemeral
-package dir under `~/.pi/agent/npm`. If `python3` or the network is missing at
-install time, postinstall prints a clear notice and the host falls back at
-runtime.
+That path is stable across updates because it sits outside the package's own directory,
+which npm replaces on each update. If `python3` or the network is missing at install time,
+`postinstall` prints a clear notice and the host falls back at runtime.
 
-At spawn, `resolvePythonPath` chooses the interpreter in order:
+At spawn, `resolvePythonPath` picks the interpreter in this order:
 
 1. the repo's own `.venv` (development)
-2. a cwd-local `.venv` (project)
+2. a venv in the current directory (per-project)
 3. `~/.pi/agent/pi-repl/venv` (package install)
-4. `$PYTHON` or `python3`
+4. `$PYTHON`, then `python3` (the fallback)
+
+The first one that exists wins. The tool's prompt tells the model it runs in a project-local
+venv, not the system interpreter, so it does not leak the wrong assumption into commands.
+
+## The kernel client
 
-The first existing one wins. The model is told (via `help()`) that it runs in a
-project-local venv, not the system interpreter, so it does not leak the wrong
-assumption into commands.
+`KernelClient.start` spawns `python -m ipykernel -f <connection-file>` with a per-run
+connection file in the temp directory, connects the three channels as ZMTP sockets, and
+waits for a `kernel_info_reply` before declaring the kernel ready. Cells run as standard
+Jupyter `execute_request`s, routed by `msg_id`:
 
-## The guest
+- **iopub** carries the output — `stream`, `execute_result`, `display_data`, and `error`
+  messages, plus private-MIME payloads that carry snapshot, restore, and namespace data.
+- **shell** carries the authoritative `execute_reply` (status, ename, evalue).
+- **control** carries interrupts (`interrupt_request`) and shutdown.
 
-`src/engine/guest.py` uses `jupyter_client.KernelManager` to start a real
-`ipykernel` subprocess (`python -m ipykernel`), keeps a blocking client
-attached, and stays alive for the whole session. Cells run in that kernel via
-`kc.execute(code)`, so state persists because the kernel process does.
+Two wire subtleties are load-bearing, and both are pinned by the contract tests.
 
-The wire protocol rides two channels, both load-bearing:
+**A cell is not done until two things arrive.** The shell reply and the iopub output stream
+travel on different connections, so a tiny reply can arrive before a large output has
+finished draining on iopub. A cell settles only when **both** the `execute_reply` and the
+matching iopub `status idle` (published after every byte of output) have arrived. Settling
+on the reply alone would drop output that was still in flight.
 
-*Separation.* Protocol traffic uses a dedicated pipe (fd 3). The guest's real
-stdout/stderr carry only user output, so a cell printing JSON cannot be parsed
-as a protocol message.
+**Output is capped per channel.** Each channel accumulates output against a character budget
+(`maxOutputChars`). Overflow is flagged per message — a single 10 MB print trips the cap
+within that one message, not only once a later message exhausts the budget — and the host
+appends an explicit truncation marker so the model knows output was cut.
 
-*Authentication.* Every frame carries a nonce the host mints at spawn and the
-guest erases from its environment before any cell runs. Code inside a cell
-cannot recover it. Without this, a cell could announce its own completion and
-claim success while failing — an agent that cannot trust its own results has
-nothing.
+**Cancellation is real.** An abort sends an `interrupt_request` on the control channel,
+which raises a genuine `KeyboardInterrupt` in the running cell; the namespace survives. As a
+backstop for cells wedged in C code (which ignore interrupts), the engine's abort grace then
+kills the kernel after 500 ms, and the next call rebuilds it from the last snapshot.
 
-## Toolbox loading
+## Helpers loading
 
-At boot the guest and the host both read the toolbox directory (default
-`src/engine/toolbox`, overridden by config `toolboxDir` → env `PI_TOOLBOX_DIR`).
+At boot, the kernel and the host both read **one** helpers directory — the fixed
+`~/.pi/agent/pi-repl/helpers`, created empty on install with nothing seeded into it. There is
+no shipped toolbox that merges in.
 
-- **guest** execs each `*.py` into the kernel namespace, making functions
+- **The kernel** execs each `*.py` into its namespace, so the file's functions become
   callable.
-- **host** reads the same files to build the functions list on the `execute`
-  tool's `promptGuidelines` (and the tool `description`), so the model sees the
-  real signatures and one-line summaries.
+- **The host** reads the same files to build the helper list shown in the `execute` tool's
+  prompt, so the model sees each `helper_description` verbatim.
 
-The loader reads each file's `def (...)`: signature (authoritative) and its
-`function_description = """..."""` (one-line summary, optional). Since both
-sides read the same directory, a function in the prompt also exists in the
-kernel. A file renamed with a `_` prefix is skipped by both, so a disabled
-function is never advertised where it does not load. See
-`docs/how-to-functions.md`.
+Because both sides read the same directory, anything the model is told about is also
+callable, and a file renamed with a `_` prefix is skipped by both. The
+`promptGuidelines` are built once, when the `execute` tool is registered, so a helpers
+change needs a **session restart or `/reload`** to reach the prompt — and the kernel loads
+helpers only at boot anyway.
 
-The `promptGuidelines` are built once, when the `execute` tool is registered
-(module load). A toolbox change therefore needs a **session restart / `/reload`**
-to be reflected in the prompt — the kernel also loads the toolbox only at boot.
+There are no custom discovery intrinsics (`ls()` / `help()`) injected into a bare kernel.
+The model discovers what is loaded by listing the namespace with ordinary Python:
 
-`ls()` and `help(name)` are built into the kernel (not toolbox files), so a
-bare kernel still lets the model discover what is loaded.
+```python
+[k for k in globals() if not k.startswith('_')]
+```
+
+For the full helper contract — the description, the docstring, disabling — see
+[helpers.md](helpers.md).
 
 ## Snapshots & honest resets
 
-After each successful cell the host schedules a debounced snapshot: it asks
-the guest to pickle the kernel's globals (entry-by-entry so one bad value
-costs only itself), and stores that as `namespace.snapshot` keyed to the
-session file under `~/.pi/agent/pi-repl/state/<session>/`. On a fresh engine it
-restores, and whatever cannot be pickled
-(live handles, some objects) is reported by name.
+After each successful cell, the host schedules a debounced snapshot. A private kernel cell
+pickles the kernel's `globals` (entry by entry, so one value that cannot be pickled costs
+only itself) and publishes the result back over a private MIME payload. The host stores it as
+`namespace.snapshot`, keyed to the session file under
+`~/.pi/agent/pi-repl/state/<session>/`.
 
-If the evaluator restarts, the result is prefixed with a `<rlm_engine_reset>`
-block naming what was revived and what was lost, so the model re-verifies
-before reuse rather than trusting state that is gone.
+When a fresh engine is built, it restores that snapshot. Whatever could not be pickled —
+live handles, some objects — is reported by name. If the evaluator was rebuilt mid-session,
+the result is prefixed with a `<repl_engine_reset>` block that names what was revived and
+what was lost, so the model re-verifies before reusing state that may be gone.
 
 ## Failure modes
 
 | Failure | Behaviour |
 | --- | --- |
-| Cell throws | `done { status: "error" }` with traceback; kernel namespace intact |
-| Kernel wedged | timeout → kill kernel subprocess → spawn fresh → restore snapshot |
-| Guest process dies | pending calls settle; engine reports itself down; later calls reject |
-| Host exits | guest is killed; on abrupt death it self-exits on stdin EOF |
+| Cell throws | `error` status with traceback; kernel namespace intact |
+| Cell silent or wedged | the watchdog sends an `interrupt_request`; a caller abort then kills the kernel after a 500 ms grace |
+| Kernel dies | the running cell settles with an error; the next call builds a fresh kernel and restores the last snapshot |
+| Host exits | `process.on("exit")` SIGKILLs live kernels (a child does not die with its parent) |
 | Output flood | capped per channel, truncation announced |
 
 ## Testing
 
-- **Host (bun):** `test/units.test.ts` (protocol, render, config) +
-  `test/preview-core.test.ts`.
-- **Evaluator (pytest):** `test/guest_contract.py` drives a real guest and
-  asserts persistence, error-survival, output attribution, snapshots, ls/help.
-- **Integration (slow):** `test/engine.integration.test.ts` boots a real
-  engine + guest and proves a variable survives an engine restart.
+- **Host (fast):** `test/units.test.ts` covers engine orchestration, rendering, and config;
+  `test/preview-core.test.ts` covers the preview logic.
+- **Contract (slow):** `test/engine.integration.test.ts` boots a real kernel per engine and
+  proves the guarantees the old Python contract pinned: persistence across cells,
+  error-survival, output attribution, helpers loading, snapshot/restore round-trips, output
+  caps, silence timeout, abort, and rebuild-from-snapshot after a dead kernel.
+
+The gate is `just check` — biome (format + lint) plus the host tests. `just integration`
+adds the real-kernel suite.
 
-Gate: `just check` = biome + bun test (host) + pytest (guest).
-`just integration` adds the real-host seam.
+## The fixed layout
 
-## Configuration reference
+There is no configuration file and no knobs. Everything lives under one directory in the
+user's home, and the default is the only option:
 
-Loaded from `~/.pi/agent/pi-repl/config.json` (or `$PI_REPL_CONFIG`), first-found-wins, never
-throws on a missing/malformed file.
+```
+~/.pi/agent/pi-repl/
+  venv/         the Python interpreter + ipykernel
+  helpers/      the helpers directory — every *.py loads into every kernel
+  state/        per-session namespace snapshots
+```
 
-| Key | Type / default | Meaning |
-| --- | --- | --- |
-| `toolboxDir` | string, optional | Directory of one-function-per-`.py` files that ADDS to the shipped `src/engine/toolbox` and, when a name collides, overrides that built-in. `~` is expanded; a bare relative path resolves from the process cwd (not reliable) — prefer an absolute path. |
-| `pythonPath` | string, optional | The interpreter used to spawn the guest. Omit to use `resolvePythonPath` (see venv). |
-| `timeoutMs` | number, 60000 | Per-cell execution timeout in ms. |
-| `snapshotDebounceMs` | number, 1500 | Debounce after an ok cell before snapshot, in ms. |
+The helpers directory is fixed at `~/.pi/agent/pi-repl/helpers` (matching the kernel's
+`readHelperSources` default), so both sides are guaranteed to read the same directory. The
+venv is built automatically and the interpreter resolved by the order above — no setting
+needed. The per-cell silence watchdog is off by default (`PI_REPL_TIMEOUT_MS=0`: a silent
+but working cell may run on).
 
 ## Reference documentation
 
-- Philosophy and design rationale: [docs/philosophy.md](docs/philosophy.md)
-- Adding a toolbox function: [docs/how-to-functions.md](docs/how-to-functions.md)
+- Philosophy and design rationale: [philosophy.md](philosophy.md)
+- Adding a helper: [helpers.md](helpers.md)

package/docs/helpers.md

@@ -0,0 +1,110 @@
+# How to write a helper
+
+A **helper** is a `.py` file that gets exec'd into every kernel, so anything it defines —
+functions, classes, constants, imports, or a module that owns a fragile or opaque piece of
+work — becomes part of the agent's workspace; you drop the file into one folder, restart
+the session, and it's available. Like a bookmark for code the agent keeps reaching for.
+
+This guide shows the smallest callable helper that works, then explains the parts of a
+helper file and the habits that make a helper useful.
+
+## Prerequisites
+
+- A working `pi --repl` session (the extension is installed — see the README).
+- The helpers folder, `~/.pi/agent/pi-repl/helpers/`. It is created empty on install.
+
+## The smallest helper
+
+Create this file:
+
+```python
+# ~/.pi/agent/pi-repl/helpers/double.py
+helper_description = """double(x) — multiply a value by two."""
+
+def double(x):
+    """Return x * 2. Works on ints, floats, and lists."""
+    return x * 2
+```
+
+Restart the session (`/reload`, or relaunch `pi --repl`), then check it loaded:
+
+```python
+print([k for k in globals() if not k.startswith('_')])
+# ['double', ...]
+
+print(double(21))
+# 42
+```
+
+If `double` appears in the namespace and runs, it's loaded. That is the whole loop: write
+the file, restart, use it.
+
+## How it actually works
+
+Two things happen with the file you wrote:
+
+1. **The kernel execs it at boot.** The file's source runs in the kernel before the first
+   cell, so `def double` defines a callable `double` in the workspace. Separately, the
+   filename sets the label the prompt uses to advertise it — `double.py` is listed as
+   `double`. Keep the two identical (see *Common mistakes*).
+
+2. **The model sees the description.** The `execute` tool's prompt lists each helper's
+   `helper_description`, rendered **verbatim**. Nothing is parsed: the text you put between
+   the triple-quotes is exactly what the model reads.
+
+Shell and file IO are not helpers — they are already ordinary Python
+(`subprocess.run`, `!cmd`, `%%bash`, `open`, `pathlib`). A helper is only worth writing for
+the fragile or opaque part the model can't reliably reconstruct on its own: a `web_search`
+with provider failover, a client wrapper, a conversion it keeps getting wrong.
+
+## The three parts of a helper file
+
+| Part | Required? | What it does | Who sees it |
+|---|---|---|---|
+| `def name(...)` | yes | the implementation | runs in the kernel |
+| `helper_description` | no | one line the model reads first | the prompt, verbatim |
+| a docstring | no | the full detail | `print(name.__doc__)` on demand |
+
+**The function** is the implementation. Its public name must match the filename: `double.py`
+exposes `double`. Keep the `def` name identical so callers and the namespace agree.
+
+**The description** is the model's first impression. It is the only part that reaches the
+prompt, and it is billed into context every turn, so it wants to be short. Two habits help
+(not requirements — the loader parses nothing):
+
+- Start with the call shape: `double(x) — multiply a value by two.`
+- Add an "Instead of:" line naming the hand-rolled code it replaces, so the model reaches
+  for the helper rather than rewriting the raw call. *Good:* `Instead of: subprocess.run
+  with a hand-rolled kill-on-timeout.` *Weak:* `Instead of: doing it manually.`
+
+**The docstring** holds the depth — argument types and defaults, return value, error
+behaviour, environment facts ("this runs in a project-local venv, not the system Python").
+It never enters the prompt. The model reads it on demand with `print(name.__doc__)` or
+`help(name)` when the description is not enough.
+
+## Common mistakes
+
+- **Description buried in detail.** Move anything beyond the call shape and one consequence
+  into the docstring. A long description bills into every turn.
+- **The helper decides too much.** A helper should own the murky part (the call, the
+  parsing), not the decision. If it picks the command, the routing, or the judgement, the
+  model stops reasoning.
+- **No docstring.** Without one, the model sees only a signature and the gotchas vanish.
+- **Top-level side effects.** The file runs at boot in every kernel. Keep module-level code
+  to definitions — no prints, no network, no slow imports at module scope.
+- **Filename and `def` name differ.** `helpers/foo.py` exposing `def bar` confuses callers
+  and the prompt. Keep them identical.
+
+## Disable a file without deleting it
+
+Rename it with a leading underscore (`_scratch.py`). The loader skips any file whose name
+starts with `_`, so it never reaches the kernel or the prompt. Handy for scratch work.
+
+## Checklist
+
+- [ ] Filename matches the public `def` name
+- [ ] `helper_description` is short and starts with the call shape
+- [ ] Depth lives in the docstring, not the description
+- [ ] No top-level side effects
+- [ ] Session restarted or `/reload`-ed
+- [ ] Name appears in `globals()` and the function runs