pi-repl-py 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +31 -20
- package/docs/ARCHITECTURE.md +182 -0
- package/docs/how-to-functions.md +82 -79
- package/docs/philosophy.md +67 -59
- package/index.ts +8 -31
- package/package.json +12 -6
- package/scripts/setup-venv.mjs +41 -19
- package/src/engine/index.ts +92 -433
- package/src/engine/kernel.ts +598 -0
- package/src/engine/session.ts +149 -0
- package/src/engine/zmtp.ts +251 -0
- package/src/extension/helpers.ts +46 -0
- package/src/extension/preview/candidates.ts +159 -0
- package/src/extension/preview/descriptor.ts +28 -0
- package/src/extension/preview/index.ts +31 -0
- package/src/extension/preview/scan.ts +59 -0
- package/src/extension/preview/shell.ts +156 -0
- package/src/extension/preview/types.ts +23 -0
- package/src/extension/prompt.ts +76 -0
- package/src/extension/render-core.ts +11 -31
- package/src/extension/render.ts +2 -11
- package/src/extension/session-engine.ts +9 -31
- package/src/extension/tool-meta.ts +11 -54
- package/ARCHITECTURE.md +0 -141
- package/src/engine/guest.py +0 -317
- package/src/engine/protocol.ts +0 -66
- package/src/engine/toolbox/bash.py +0 -72
- package/src/engine/toolbox/edit.py +0 -37
- package/src/engine/toolbox/read.py +0 -26
- package/src/engine/toolbox/write.py +0 -23
- package/src/extension/config.ts +0 -65
- package/src/extension/preview-core.ts +0 -518
- package/src/extension/toolbox.ts +0 -74
package/ARCHITECTURE.md
DELETED
|
@@ -1,141 +0,0 @@
|
|
|
1
|
-
# Architecture
|
|
2
|
-
|
|
3
|
-
Two processes: the host lives inside pi, the guest owns the Python workspace.
|
|
4
|
-
|
|
5
|
-
```
|
|
6
|
-
pi
|
|
7
|
-
└─ extension (index.ts) registers `execute`, dormant until --repl
|
|
8
|
-
└─ EngineManager (src/engine/index.ts) spawn host: snapshots, teardown
|
|
9
|
-
│ stdin ──▶ protocol commands (run / snapshot / restore / ping)
|
|
10
|
-
│ fd 3 ◀── stream, done, snapshot_result, ...
|
|
11
|
-
└─ guest.py ▶ jupyter_client ▶ a real ipython kernel (subprocess)
|
|
12
|
-
```
|
|
13
|
-
|
|
14
|
-
The host is TypeScript; the evaluator is Python in its own process. Splitting
|
|
15
|
-
them is what makes a bad cell survivable: a cell can raise, leak memory, or
|
|
16
|
-
wedge the guest without taking pi down, and the host, being not the thing
|
|
17
|
-
that failed, always gets to report what happened.
|
|
18
|
-
|
|
19
|
-
## The Python environment (the venv)
|
|
20
|
-
|
|
21
|
-
The evaluator is a real `ipython` kernel, so it needs a Python environment with
|
|
22
|
-
`ipykernel` + `jupyter_client`. You cannot fake that with a script; it is a
|
|
23
|
-
hard runtime dependency.
|
|
24
|
-
|
|
25
|
-
When installed as a pi package, `npm install` runs `postinstall`
|
|
26
|
-
(`scripts/setup-venv.mjs`), which creates a stable per-user venv:
|
|
27
|
-
|
|
28
|
-
```
|
|
29
|
-
~/.pi/agent/pi-repl-venv/bin/python3
|
|
30
|
-
```
|
|
31
|
-
|
|
32
|
-
That path is stable across updates because it sits outside the ephemeral
|
|
33
|
-
package dir under `~/.pi/agent/npm`. If `python3` or the network is missing at
|
|
34
|
-
install time, postinstall prints a clear notice and the host falls back at
|
|
35
|
-
runtime.
|
|
36
|
-
|
|
37
|
-
At spawn, `resolvePythonPath` chooses the interpreter in order:
|
|
38
|
-
|
|
39
|
-
1. the repo's own `.venv` (development)
|
|
40
|
-
2. a cwd-local `.venv` (project)
|
|
41
|
-
3. `~/.pi/agent/pi-repl-venv` (package install)
|
|
42
|
-
4. `$PYTHON` or `python3`
|
|
43
|
-
|
|
44
|
-
The first existing one wins. The model is told (via `help()`) that it runs in a
|
|
45
|
-
project-local venv, not the system interpreter, so it does not leak the wrong
|
|
46
|
-
assumption into commands.
|
|
47
|
-
|
|
48
|
-
## The guest
|
|
49
|
-
|
|
50
|
-
`src/engine/guest.py` uses `jupyter_client.KernelManager` to start a real
|
|
51
|
-
`ipykernel` subprocess (`python -m ipykernel`), keeps a blocking client
|
|
52
|
-
attached, and stays alive for the whole session. Cells run in that kernel via
|
|
53
|
-
`kc.execute(code)`, so state persists because the kernel process does.
|
|
54
|
-
|
|
55
|
-
The wire protocol rides two channels, both load-bearing:
|
|
56
|
-
|
|
57
|
-
*Separation.* Protocol traffic uses a dedicated pipe (fd 3). The guest's real
|
|
58
|
-
stdout/stderr carry only user output, so a cell printing JSON cannot be parsed
|
|
59
|
-
as a protocol message.
|
|
60
|
-
|
|
61
|
-
*Authentication.* Every frame carries a nonce the host mints at spawn and the
|
|
62
|
-
guest erases from its environment before any cell runs. Code inside a cell
|
|
63
|
-
cannot recover it. Without this, a cell could announce its own completion and
|
|
64
|
-
claim success while failing — an agent that cannot trust its own results has
|
|
65
|
-
nothing.
|
|
66
|
-
|
|
67
|
-
## Toolbox loading
|
|
68
|
-
|
|
69
|
-
At boot the guest and the host both read the toolbox directory (default
|
|
70
|
-
`src/engine/toolbox`, overridden by config `toolboxDir` → env `PI_TOOLBOX_DIR`).
|
|
71
|
-
|
|
72
|
-
- **guest** execs each `*.py` into the kernel namespace, making functions
|
|
73
|
-
callable.
|
|
74
|
-
- **host** reads the same files to build the functions list on the `execute`
|
|
75
|
-
tool's `promptGuidelines` (and the tool `description`), so the model sees the
|
|
76
|
-
real signatures and one-line summaries.
|
|
77
|
-
|
|
78
|
-
The loader reads each file's `def (...)`: signature (authoritative) and its
|
|
79
|
-
`function_description = """..."""` (one-line summary, optional). Since both
|
|
80
|
-
sides read the same directory, a function in the prompt also exists in the
|
|
81
|
-
kernel. A file renamed with a `_` prefix is skipped by both, so a disabled
|
|
82
|
-
function is never advertised where it does not load. See
|
|
83
|
-
`docs/how-to-functions.md`.
|
|
84
|
-
|
|
85
|
-
The `promptGuidelines` are built once, when the `execute` tool is registered
|
|
86
|
-
(module load). A toolbox change therefore needs a **session restart / `/reload`**
|
|
87
|
-
to be reflected in the prompt — the kernel also loads the toolbox only at boot.
|
|
88
|
-
|
|
89
|
-
`ls()` and `help(name)` are built into the kernel (not toolbox files), so a
|
|
90
|
-
bare kernel still lets the model discover what is loaded.
|
|
91
|
-
|
|
92
|
-
## Snapshots & honest resets
|
|
93
|
-
|
|
94
|
-
After each successful cell the host schedules a debounced snapshot: it asks
|
|
95
|
-
the guest to pickle the kernel's globals (entry-by-entry so one bad value
|
|
96
|
-
costs only itself), and stores that as `namespace.snapshot` keyed to the
|
|
97
|
-
session file. On a fresh engine it restores, and whatever cannot be pickled
|
|
98
|
-
(live handles, some objects) is reported by name.
|
|
99
|
-
|
|
100
|
-
If the evaluator restarts, the result is prefixed with a `<rlm_engine_reset>`
|
|
101
|
-
block naming what was revived and what was lost, so the model re-verifies
|
|
102
|
-
before reuse rather than trusting state that is gone.
|
|
103
|
-
|
|
104
|
-
## Failure modes
|
|
105
|
-
|
|
106
|
-
| Failure | Behaviour |
|
|
107
|
-
| --- | --- |
|
|
108
|
-
| Cell throws | `done { status: "error" }` with traceback; kernel namespace intact |
|
|
109
|
-
| Kernel wedged | timeout → kill kernel subprocess → spawn fresh → restore snapshot |
|
|
110
|
-
| Guest process dies | pending calls settle; engine reports itself down; later calls reject |
|
|
111
|
-
| Host exits | guest is killed; on abrupt death it self-exits on stdin EOF |
|
|
112
|
-
| Output flood | capped per channel, truncation announced |
|
|
113
|
-
|
|
114
|
-
## Testing
|
|
115
|
-
|
|
116
|
-
- **Host (bun):** `test/units.test.ts` (protocol, render, config) +
|
|
117
|
-
`test/preview-core.test.ts`.
|
|
118
|
-
- **Evaluator (pytest):** `test/guest_contract.py` drives a real guest and
|
|
119
|
-
asserts persistence, error-survival, output attribution, snapshots, ls/help.
|
|
120
|
-
- **Integration (slow):** `test/engine.integration.test.ts` boots a real
|
|
121
|
-
engine + guest and proves a variable survives an engine restart.
|
|
122
|
-
|
|
123
|
-
Gate: `just check` = biome + bun test (host) + pytest (guest).
|
|
124
|
-
`just integration` adds the real-host seam.
|
|
125
|
-
|
|
126
|
-
## Configuration reference
|
|
127
|
-
|
|
128
|
-
Loaded from `~/.pi/agent/pi-repl.json` (or `$PI_REPL_CONFIG`), first-found-wins, never
|
|
129
|
-
throws on a missing/malformed file.
|
|
130
|
-
|
|
131
|
-
| Key | Type / default | Meaning |
|
|
132
|
-
| --- | --- | --- |
|
|
133
|
-
| `toolboxDir` | string, optional | Directory of one-function-per-`.py` files that replaces the shipped `src/engine/toolbox`. `~` is expanded; a bare relative path resolves from the process cwd (not reliable) — prefer an absolute path. |
|
|
134
|
-
| `pythonPath` | string, optional | The interpreter used to spawn the guest. Omit to use `resolvePythonPath` (see venv). |
|
|
135
|
-
| `timeoutMs` | number, 60000 | Per-cell execution timeout in ms. |
|
|
136
|
-
| `snapshotDebounceMs` | number, 1500 | Debounce after an ok cell before snapshot, in ms. |
|
|
137
|
-
|
|
138
|
-
## Reference documentation
|
|
139
|
-
|
|
140
|
-
- Philosophy and design rationale: [docs/philosophy.md](docs/philosophy.md)
|
|
141
|
-
- Adding a toolbox function: [docs/how-to-functions.md](docs/how-to-functions.md)
|
package/src/engine/guest.py
DELETED
|
@@ -1,317 +0,0 @@
|
|
|
1
|
-
"""
|
|
2
|
-
guest.py — the real IPython kernel guest evaluator for pi-repl.
|
|
3
|
-
|
|
4
|
-
The host spawns this once. It starts a local ipykernel subprocess via
|
|
5
|
-
jupyter_client, keeps it for the session, and bridges the wire protocol to it
|
|
6
|
-
(stdin = commands, fd 3 = results). State survives because the kernel process
|
|
7
|
-
does. Frames carry a nonce the host mints and the guest erases, so agent code
|
|
8
|
-
cannot forge protocol traffic.
|
|
9
|
-
"""
|
|
10
|
-
|
|
11
|
-
from __future__ import annotations
|
|
12
|
-
|
|
13
|
-
import json
|
|
14
|
-
import os
|
|
15
|
-
import sys
|
|
16
|
-
import time
|
|
17
|
-
|
|
18
|
-
# --- protocol envelope ---
|
|
19
|
-
ENVELOPE_KEY = "__rlm"
|
|
20
|
-
NONCE_ENV = "PI_RLM_NONCE"
|
|
21
|
-
PROTOCOL_FD = 3
|
|
22
|
-
|
|
23
|
-
NONCE = os.environ.get(NONCE_ENV, "")
|
|
24
|
-
os.environ.pop(NONCE_ENV, None)
|
|
25
|
-
|
|
26
|
-
# --- per-cell timeout: 0 = no cap; else a silence watchdog (no output for N seconds) ---
|
|
27
|
-
CELL_TIMEOUT_S = float(os.environ.get("PI_REPL_TIMEOUT_MS", "0") or "0") / 1000.0
|
|
28
|
-
|
|
29
|
-
# --- snapshot/restore use a fixed window, not the cell silence timer ---
|
|
30
|
-
SNAPSHOT_TIMEOUT_S = 90.0
|
|
31
|
-
|
|
32
|
-
# --- cap buffered output so a runaway print can't grow guest memory or send one giant frame ---
|
|
33
|
-
MAX_CELL_OUTPUT_CHARS = 1_000_000
|
|
34
|
-
|
|
35
|
-
# --- fd 3 protocol writer; dup'd so we don't close the caller's fd 3 on exit ---
|
|
36
|
-
_proto = os.fdopen(os.dup(PROTOCOL_FD), "w", buffering=1)
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
def _send(msg):
|
|
40
|
-
envelope = {ENVELOPE_KEY: 1, **msg}
|
|
41
|
-
if NONCE:
|
|
42
|
-
envelope["n"] = NONCE
|
|
43
|
-
_proto.write(json.dumps(envelope) + "\n")
|
|
44
|
-
_proto.flush()
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
def _decode(line):
|
|
48
|
-
if ENVELOPE_KEY not in line:
|
|
49
|
-
return None
|
|
50
|
-
try:
|
|
51
|
-
obj = json.loads(line)
|
|
52
|
-
except Exception:
|
|
53
|
-
return None
|
|
54
|
-
if obj.get(ENVELOPE_KEY) != 1 or not isinstance(obj.get("type"), str):
|
|
55
|
-
return None
|
|
56
|
-
if NONCE and obj.get("n") != NONCE:
|
|
57
|
-
return None
|
|
58
|
-
return obj
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
# --- toolbox: one function per *.py, exec'd into every kernel (PI_TOOLBOX_DIR) ---
|
|
62
|
-
|
|
63
|
-
def _toolbox_files(directory):
|
|
64
|
-
"""Return {function_name: source} for each *.py in `directory`."""
|
|
65
|
-
if not directory:
|
|
66
|
-
return {}
|
|
67
|
-
d = os.path.expanduser(directory)
|
|
68
|
-
if not os.path.isdir(d):
|
|
69
|
-
return {}
|
|
70
|
-
names = {}
|
|
71
|
-
for entry in sorted(os.listdir(d)):
|
|
72
|
-
if not entry.endswith(".py"):
|
|
73
|
-
continue
|
|
74
|
-
name = entry[:-3]
|
|
75
|
-
if not name.isidentifier() or name.startswith("_"):
|
|
76
|
-
continue
|
|
77
|
-
try:
|
|
78
|
-
with open(os.path.join(d, entry), encoding="utf-8") as f:
|
|
79
|
-
names[name] = f.read()
|
|
80
|
-
except OSError:
|
|
81
|
-
continue
|
|
82
|
-
return names
|
|
83
|
-
|
|
84
|
-
DEFAULT_TOOLBOX_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "toolbox")
|
|
85
|
-
TOOLBOX_DIR = os.environ.get("PI_TOOLBOX_DIR", "").strip()
|
|
86
|
-
_TOOLBOX_SRC = _toolbox_files(TOOLBOX_DIR or DEFAULT_TOOLBOX_DIR)
|
|
87
|
-
|
|
88
|
-
# --- help/ls are part of the evaluator, not the toolbox ---
|
|
89
|
-
INTRINSIC = """
|
|
90
|
-
# --- ls() filters IPython-injected names out of the tool list ---
|
|
91
|
-
_RPL_LS_NOISE = {'exit', 'quit', 'get_ipython', 'open', 'display'}
|
|
92
|
-
|
|
93
|
-
def ls():
|
|
94
|
-
return sorted(n for n in globals() if n not in _RPL_LS_NOISE and not n.startswith('_') and callable(globals()[n]))
|
|
95
|
-
|
|
96
|
-
def help(name=None):
|
|
97
|
-
if name is None:
|
|
98
|
-
return ls()
|
|
99
|
-
fn = globals().get(name)
|
|
100
|
-
if fn is None or not callable(fn):
|
|
101
|
-
return f"no such function: {name!r}"
|
|
102
|
-
return fn.__doc__ or f"{name} (no docstring)"
|
|
103
|
-
"""
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
from jupyter_client import KernelManager
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
class Kernel:
|
|
110
|
-
"""A persistent subprocess ipykernel + blocking client."""
|
|
111
|
-
|
|
112
|
-
def __init__(self):
|
|
113
|
-
self.km = KernelManager(kernel_name="python3")
|
|
114
|
-
self.km.start_kernel()
|
|
115
|
-
self.kc = self.km.client()
|
|
116
|
-
self.kc.start_channels()
|
|
117
|
-
self.kc.wait_for_ready(timeout=30)
|
|
118
|
-
self._preload()
|
|
119
|
-
|
|
120
|
-
def _preload(self):
|
|
121
|
-
"""Exec every toolbox function + the intrinsic help/ls into the kernel ns."""
|
|
122
|
-
code = INTRINSIC + "\n"
|
|
123
|
-
for src in _TOOLBOX_SRC.values():
|
|
124
|
-
code += src + "\n"
|
|
125
|
-
if code.strip():
|
|
126
|
-
self.kc.execute(code)
|
|
127
|
-
self._drain()
|
|
128
|
-
|
|
129
|
-
def _drain(self):
|
|
130
|
-
try:
|
|
131
|
-
while True:
|
|
132
|
-
m = self.kc.get_iopub_msg(timeout=1)
|
|
133
|
-
if m.get("msg_type") == "status" and m.get("content", {}).get("execution_state") == "idle":
|
|
134
|
-
break
|
|
135
|
-
except Exception:
|
|
136
|
-
pass
|
|
137
|
-
|
|
138
|
-
def _drain_execution(self, code, timeout):
|
|
139
|
-
"""Run `code`; return (stdout, stderr, error_text, result, timed_out).
|
|
140
|
-
|
|
141
|
-
`timeout <= 0` means "no cap": a cell runs until it reports idle.
|
|
142
|
-
`timeout > 0` is a SILENCE watchdog — it trips only once the cell has
|
|
143
|
-
produced no message for `timeout` seconds. A silent-but-running command
|
|
144
|
-
(e.g. `find ... | sort`) is allowed to complete; a stalled one (dead
|
|
145
|
-
kernel, or nothing for the silence window) reports `timed_out=True` so
|
|
146
|
-
the caller can surface a real hang instead of faking success.
|
|
147
|
-
"""
|
|
148
|
-
msg_id = self.kc.execute(code)
|
|
149
|
-
out, err, error, result = [], [], None, None
|
|
150
|
-
out_len, err_len = 0, 0
|
|
151
|
-
timed_out = False
|
|
152
|
-
# --- silence clock only starts once the cell begins (grace for a fresh kernel) ---
|
|
153
|
-
last_activity: float | None = None
|
|
154
|
-
while True:
|
|
155
|
-
if not self.km.is_alive():
|
|
156
|
-
timed_out = True
|
|
157
|
-
break
|
|
158
|
-
if last_activity is not None and timeout and (time.monotonic() - last_activity) >= timeout:
|
|
159
|
-
timed_out = True
|
|
160
|
-
break
|
|
161
|
-
wait = (timeout - (time.monotonic() - last_activity)) if (last_activity is not None and timeout) else 0.25
|
|
162
|
-
try:
|
|
163
|
-
m = self.kc.get_iopub_msg(timeout=max(0.01, min(0.25, wait)))
|
|
164
|
-
except Exception:
|
|
165
|
-
continue
|
|
166
|
-
if m.get("parent_header", {}).get("msg_id") != msg_id:
|
|
167
|
-
continue
|
|
168
|
-
if last_activity is None:
|
|
169
|
-
last_activity = time.monotonic()
|
|
170
|
-
else:
|
|
171
|
-
last_activity = time.monotonic()
|
|
172
|
-
mt = m.get("msg_type")
|
|
173
|
-
c = m.get("content", {})
|
|
174
|
-
if mt == "stream":
|
|
175
|
-
is_out = c.get("name") == "stdout"
|
|
176
|
-
text = c.get("text", "") or ""
|
|
177
|
-
if is_out:
|
|
178
|
-
if out_len < MAX_CELL_OUTPUT_CHARS:
|
|
179
|
-
take = text[: MAX_CELL_OUTPUT_CHARS - out_len]
|
|
180
|
-
out.append(take)
|
|
181
|
-
out_len += len(take)
|
|
182
|
-
else:
|
|
183
|
-
if err_len < MAX_CELL_OUTPUT_CHARS:
|
|
184
|
-
take = text[: MAX_CELL_OUTPUT_CHARS - err_len]
|
|
185
|
-
err.append(take)
|
|
186
|
-
err_len += len(take)
|
|
187
|
-
elif mt == "execute_result":
|
|
188
|
-
result = c.get("data", {}).get("text/plain")
|
|
189
|
-
elif mt == "error":
|
|
190
|
-
error = "\n".join(c.get("traceback", ["(no traceback)"]))
|
|
191
|
-
elif mt == "status" and c.get("execution_state") == "idle":
|
|
192
|
-
break
|
|
193
|
-
if timed_out:
|
|
194
|
-
# --- best-effort cancel so the NEXT cell doesn't queue behind this one ---
|
|
195
|
-
try:
|
|
196
|
-
self.kc.interrupt_kernel()
|
|
197
|
-
except Exception:
|
|
198
|
-
pass
|
|
199
|
-
return "".join(out), "".join(err), error, result, timed_out
|
|
200
|
-
|
|
201
|
-
def execute(self, code):
|
|
202
|
-
"""Idle-sync path used by snapshot/restore; not a user cell."""
|
|
203
|
-
return self._drain_execution(code, SNAPSHOT_TIMEOUT_S)[:4]
|
|
204
|
-
|
|
205
|
-
def run_cell(self, code):
|
|
206
|
-
"""Run a user cell under the per-cell timeout, so the model learns
|
|
207
|
-
when work did not finish."""
|
|
208
|
-
return self._drain_execution(code, CELL_TIMEOUT_S)
|
|
209
|
-
|
|
210
|
-
def snapshot_globals(self):
|
|
211
|
-
# --- snapshot only user state (skip toolbox/intrinsic functions and _ names) ---
|
|
212
|
-
tool_names = sorted(set(_TOOLBOX_SRC) | {"ls", "help", "function_description"})
|
|
213
|
-
skip_names = json.dumps(tool_names)
|
|
214
|
-
out, _, _, _ = self.execute(
|
|
215
|
-
"import pickle as _pk, base64 as _b64, json as _js\n"
|
|
216
|
-
"__rlm_skip = set(" + skip_names + ") | {'In','Out','get_ipython','exit','quit','open'}\n"
|
|
217
|
-
"__rlm_v = {}\n__rlm_f = []\n"
|
|
218
|
-
"for _k, _v in list(globals().items()):\n"
|
|
219
|
-
" # skip IPython bookkeeping and names with a leading underscore\n"
|
|
220
|
-
" if _k.startswith('__') or _k.startswith('_') or _k in __rlm_skip:\n"
|
|
221
|
-
" continue\n"
|
|
222
|
-
" try:\n"
|
|
223
|
-
" __rlm_v[_k] = _b64.b64encode(_pk.dumps(_v)).decode()\n"
|
|
224
|
-
" except Exception as _e:\n"
|
|
225
|
-
" __rlm_f.append({'name': _k, 'reason': str(_e)})\n"
|
|
226
|
-
"print('__RLC_SNAPSHOT__' + _js.dumps({'vars': __rlm_v, 'failed': __rlm_f}))\n"
|
|
227
|
-
)
|
|
228
|
-
marker = "__RLC_SNAPSHOT__"
|
|
229
|
-
if marker not in out:
|
|
230
|
-
# --- marker never printed: serialization stalled; report incomplete so the host keeps the last good file ---
|
|
231
|
-
return {}, [], False
|
|
232
|
-
try:
|
|
233
|
-
o = json.loads(out.split(marker)[-1])
|
|
234
|
-
return o.get("vars", {}), o.get("failed", []), True
|
|
235
|
-
except Exception:
|
|
236
|
-
return {}, [], False
|
|
237
|
-
|
|
238
|
-
def restore_globals(self, vars_):
|
|
239
|
-
if not vars_:
|
|
240
|
-
return [], []
|
|
241
|
-
# --- restore each variable in one atomic kernel call so one failure reports itself ---
|
|
242
|
-
code2 = (
|
|
243
|
-
"import pickle as _pk, base64 as _b64, json as _js\n"
|
|
244
|
-
"__rl_r = {'restored': [], 'failed': []}\n"
|
|
245
|
-
+ "\n".join(
|
|
246
|
-
"try:\n"
|
|
247
|
-
f" globals()[{name!r}] = _pk.loads(_b64.b64decode({b64!r}))\n"
|
|
248
|
-
f" __rl_r['restored'].append({name!r})\n"
|
|
249
|
-
"except Exception as _e:\n"
|
|
250
|
-
f" __rl_r['failed'].append({{'name': {name!r}, 'reason': str(_e)}})\n"
|
|
251
|
-
for name, b64 in vars_.items()
|
|
252
|
-
)
|
|
253
|
-
+ "\nprint('__RLC_RESTORE__' + _js.dumps(__rl_r))"
|
|
254
|
-
)
|
|
255
|
-
out, _, _, _ = self.execute(code2)
|
|
256
|
-
marker = "__RLC_RESTORE__"
|
|
257
|
-
if marker not in out:
|
|
258
|
-
return [], []
|
|
259
|
-
try:
|
|
260
|
-
obj = json.loads(out.split(marker)[-1])
|
|
261
|
-
return obj.get("restored", []), obj.get("failed", [])
|
|
262
|
-
except Exception:
|
|
263
|
-
return [], []
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
def _line_error(text):
|
|
267
|
-
lines = text.split("\n")
|
|
268
|
-
return {"name": "", "message": text, "stack": lines[:12]}
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
def main():
|
|
272
|
-
kernel = Kernel()
|
|
273
|
-
_send({"type": "ready"})
|
|
274
|
-
|
|
275
|
-
for line in sys.stdin:
|
|
276
|
-
msg = _decode(line)
|
|
277
|
-
if not msg:
|
|
278
|
-
continue
|
|
279
|
-
t = msg["type"]
|
|
280
|
-
if t == "ping":
|
|
281
|
-
_send({"type": "pong", "id": msg["id"]})
|
|
282
|
-
elif t == "snapshot":
|
|
283
|
-
vars_, failed, complete = kernel.snapshot_globals()
|
|
284
|
-
_send({"type": "snapshot_result", "id": msg["id"], "vars": vars_, "failed": failed, "complete": complete})
|
|
285
|
-
elif t == "restore":
|
|
286
|
-
restored, failed = kernel.restore_globals(msg.get("vars", {}))
|
|
287
|
-
_send({"type": "restore_result", "id": msg["id"], "restored": restored, "failed": failed})
|
|
288
|
-
elif t == "list_names":
|
|
289
|
-
names = list(kernel.snapshot_globals()[0].keys())
|
|
290
|
-
_send({"type": "names_result", "id": msg["id"], "names": names})
|
|
291
|
-
elif t == "run":
|
|
292
|
-
cell_id = msg.get("cellId")
|
|
293
|
-
stdout, stderr, error, result, timed_out = kernel.run_cell(msg.get("code", ""))
|
|
294
|
-
if stdout:
|
|
295
|
-
_send({"type": "stream", "cellId": cell_id, "name": "stdout", "chunk": stdout})
|
|
296
|
-
if stderr:
|
|
297
|
-
_send({"type": "stream", "cellId": cell_id, "name": "stderr", "chunk": stderr})
|
|
298
|
-
if timed_out:
|
|
299
|
-
tmsg = {
|
|
300
|
-
"name": "Timeout",
|
|
301
|
-
"message": f"cell did not finish within {CELL_TIMEOUT_S:g}s and may still be running",
|
|
302
|
-
"stack": ["[cell timed out]"],
|
|
303
|
-
}
|
|
304
|
-
_send({"type": "done", "cellId": cell_id, "status": "error", "error": tmsg})
|
|
305
|
-
elif error:
|
|
306
|
-
_send({"type": "done", "cellId": cell_id, "status": "error", "error": _line_error(error)})
|
|
307
|
-
else:
|
|
308
|
-
_send({"type": "done", "cellId": cell_id, "status": "ok", "result": result})
|
|
309
|
-
# --- a single-threaded guest can't read 'abort' mid-cell; the host discards+rebuilds ---
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
if __name__ == "__main__":
|
|
313
|
-
try:
|
|
314
|
-
main()
|
|
315
|
-
except Exception as e:
|
|
316
|
-
_send({"type": "done", "cellId": "", "status": "error", "error": _line_error(str(e))})
|
|
317
|
-
sys.exit(1)
|
package/src/engine/protocol.ts
DELETED
|
@@ -1,66 +0,0 @@
|
|
|
1
|
-
// --- trust: fd3 is protocol-only (user output stays on stdout), and a cell can't forge frames (minted nonce) ---
|
|
2
|
-
|
|
3
|
-
interface HostToGuest {
|
|
4
|
-
run: { type: "run"; cellId: string; code: string };
|
|
5
|
-
abort: { type: "abort"; cellId: string };
|
|
6
|
-
ping: { type: "ping"; id: string };
|
|
7
|
-
snapshot: { type: "snapshot"; id: string };
|
|
8
|
-
restore: { type: "restore"; id: string; vars: Record<string, string> };
|
|
9
|
-
list_names: { type: "list_names"; id: string };
|
|
10
|
-
}
|
|
11
|
-
|
|
12
|
-
export type HostToGuestMessage = HostToGuest[keyof HostToGuest];
|
|
13
|
-
|
|
14
|
-
interface GuestToHost {
|
|
15
|
-
ready: { type: "ready" };
|
|
16
|
-
stream: { type: "stream"; cellId: string; name: "stdout" | "stderr"; chunk: string };
|
|
17
|
-
done: {
|
|
18
|
-
type: "done";
|
|
19
|
-
cellId: string;
|
|
20
|
-
status: "ok" | "error" | "aborted";
|
|
21
|
-
result?: string;
|
|
22
|
-
error?: { name: string; message: string; stack: string[] };
|
|
23
|
-
};
|
|
24
|
-
pong: { type: "pong"; id: string };
|
|
25
|
-
snapshot_result: {
|
|
26
|
-
type: "snapshot_result";
|
|
27
|
-
id: string;
|
|
28
|
-
vars: Record<string, string>;
|
|
29
|
-
failed: { name: string; reason: string }[];
|
|
30
|
-
/** False means the kernel didn't finish serializing; keep the last good file. */
|
|
31
|
-
complete?: boolean;
|
|
32
|
-
};
|
|
33
|
-
restore_result: {
|
|
34
|
-
type: "restore_result";
|
|
35
|
-
id: string;
|
|
36
|
-
restored: string[];
|
|
37
|
-
failed: { name: string; reason: string }[];
|
|
38
|
-
};
|
|
39
|
-
names_result: { type: "names_result"; id: string; names: string[] };
|
|
40
|
-
}
|
|
41
|
-
|
|
42
|
-
export type GuestToHostMessage = GuestToHost[keyof GuestToHost];
|
|
43
|
-
|
|
44
|
-
const ENVELOPE_KEY = "__rlm";
|
|
45
|
-
/** Env var carrying the per-process nonce to the guest. */
|
|
46
|
-
export const NONCE_ENV = "PI_RLM_NONCE";
|
|
47
|
-
/** Protocol pipe: guest → host. */
|
|
48
|
-
export const PROTOCOL_FD = 3;
|
|
49
|
-
|
|
50
|
-
export function encodeMessage(message: HostToGuestMessage | GuestToHostMessage, nonce?: string): string {
|
|
51
|
-
const envelope: Record<string, unknown> = { [ENVELOPE_KEY]: 1, ...message };
|
|
52
|
-
if (nonce) envelope.n = nonce;
|
|
53
|
-
return `${JSON.stringify(envelope)}\n`;
|
|
54
|
-
}
|
|
55
|
-
|
|
56
|
-
export function decodeMessage<T>(line: string, nonce?: string): T | null {
|
|
57
|
-
if (!line.trim()) return null;
|
|
58
|
-
try {
|
|
59
|
-
const parsed = JSON.parse(line);
|
|
60
|
-
if (parsed?.[ENVELOPE_KEY] !== 1 || typeof parsed.type !== "string") return null;
|
|
61
|
-
if (nonce && parsed.n !== nonce) return null;
|
|
62
|
-
return parsed as T;
|
|
63
|
-
} catch {
|
|
64
|
-
return null;
|
|
65
|
-
}
|
|
66
|
-
}
|
|
@@ -1,72 +0,0 @@
|
|
|
1
|
-
function_description = """Run a shell command in a fresh subshell and return its result."""
|
|
2
|
-
|
|
3
|
-
import os as _os
|
|
4
|
-
import signal as _sig
|
|
5
|
-
import subprocess as _sp
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
def _kill_group(proc):
|
|
9
|
-
"""Kill the whole process group of a child (its shell AND any grandchildren).
|
|
10
|
-
|
|
11
|
-
The shell's children inherit the fresh session's id, so they are the only
|
|
12
|
-
ones a timeout must reap; without this a `find | sort` that outlives the
|
|
13
|
-
call would keep chewing CPU long after bash() returned.
|
|
14
|
-
"""
|
|
15
|
-
try:
|
|
16
|
-
_os.killpg(_os.getpgid(proc.pid), _sig.SIGKILL)
|
|
17
|
-
except Exception:
|
|
18
|
-
pass
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
def bash(command, cwd=None, env=None, input=None, timeout=None):
|
|
22
|
-
"""Run a shell command and return a CompletedProcess.
|
|
23
|
-
|
|
24
|
-
Argument notes:
|
|
25
|
-
command - the shell command string to run.
|
|
26
|
-
cwd - optional directory to run it in; uses the evaluator's cwd if omitted.
|
|
27
|
-
env - optional dict of environment variables merged into the current env.
|
|
28
|
-
input - optional string to feed as stdin.
|
|
29
|
-
timeout - optional timeout in seconds; raises TimeoutExpired (after killing
|
|
30
|
-
the command's whole process group) if exceeded.
|
|
31
|
-
|
|
32
|
-
Result:
|
|
33
|
-
Returns subprocess.CompletedProcess. Read .stdout, .stderr, .returncode.
|
|
34
|
-
Example: out = bash("git log --oneline"); print(out.returncode, out.stdout)
|
|
35
|
-
|
|
36
|
-
Behaviour:
|
|
37
|
-
- Runs via the shell, so pipes/&&/etc. work. Each call runs a FRESH
|
|
38
|
-
subshell: cd, export, and shell variables do NOT carry across calls.
|
|
39
|
-
Hold state in Python variables instead.
|
|
40
|
-
- The shell runs in its own process group, so a timeout kills the group —
|
|
41
|
-
no orphaned children keep running afterwards.
|
|
42
|
-
- `env` is merged into the current environment, not a replacement.
|
|
43
|
-
|
|
44
|
-
Environment:
|
|
45
|
-
This evaluator runs in a project-local Python venv, not the system
|
|
46
|
-
interpreter. A command that starts python/pip should target the same venv.
|
|
47
|
-
"""
|
|
48
|
-
merged_env = dict(_os.environ)
|
|
49
|
-
if env:
|
|
50
|
-
merged_env.update(env)
|
|
51
|
-
with _sp.Popen(
|
|
52
|
-
command,
|
|
53
|
-
shell=True,
|
|
54
|
-
stdin=_sp.PIPE,
|
|
55
|
-
stdout=_sp.PIPE,
|
|
56
|
-
stderr=_sp.PIPE,
|
|
57
|
-
text=True,
|
|
58
|
-
cwd=cwd,
|
|
59
|
-
env=merged_env,
|
|
60
|
-
# --- new session => shell+children form one process group a timeout kills ---
|
|
61
|
-
start_new_session=_os.name == "posix",
|
|
62
|
-
) as proc:
|
|
63
|
-
try:
|
|
64
|
-
stdout, stderr = proc.communicate(input=input, timeout=timeout)
|
|
65
|
-
except _sp.TimeoutExpired:
|
|
66
|
-
_kill_group(proc)
|
|
67
|
-
try:
|
|
68
|
-
proc.communicate() # --- drain so no zombie is left ---
|
|
69
|
-
except Exception:
|
|
70
|
-
pass
|
|
71
|
-
raise
|
|
72
|
-
return _sp.CompletedProcess(args=command, returncode=proc.returncode, stdout=stdout, stderr=stderr)
|
|
@@ -1,37 +0,0 @@
|
|
|
1
|
-
function_description = """Replace old_text with new_text in a file; fails if old_text is not found exactly once."""
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
def edit(path, old_text, new_text):
|
|
5
|
-
"""Perform a targeted single replacement in a file.
|
|
6
|
-
|
|
7
|
-
Argument notes:
|
|
8
|
-
old_text - exact, unique text already in the file to replace.
|
|
9
|
-
new_text - text to substitute for it.
|
|
10
|
-
|
|
11
|
-
Behaviour:
|
|
12
|
-
- Requires old_text to appear EXACTLY once in the file. Zero or multiple
|
|
13
|
-
matches raise an error instead of guessing, so it can never silently
|
|
14
|
-
mangle a file it wasn't sure about.
|
|
15
|
-
- If it fails, the file is left untouched.
|
|
16
|
-
|
|
17
|
-
Environment:
|
|
18
|
-
This evaluator runs in a project-local Python venv, not the system
|
|
19
|
-
interpreter. For a package install that is ~/.pi/agent/pi-repl-venv; for a
|
|
20
|
-
repo checkout it is .venv/. The file edited is a real file on disk.
|
|
21
|
-
"""
|
|
22
|
-
with open(path, encoding="utf-8") as f:
|
|
23
|
-
content = f.read()
|
|
24
|
-
count = content.count(old_text)
|
|
25
|
-
if count == 0:
|
|
26
|
-
raise ValueError(
|
|
27
|
-
f"edit: could not find the given old_text in {path} — it may have already been "
|
|
28
|
-
"applied or the file changed. Re-read the file and retry."
|
|
29
|
-
)
|
|
30
|
-
if count > 1:
|
|
31
|
-
raise ValueError(
|
|
32
|
-
f"edit: old_text occurs {count} times in {path} — make it more specific so it matches exactly once."
|
|
33
|
-
)
|
|
34
|
-
content = content.replace(old_text, new_text, 1)
|
|
35
|
-
with open(path, "w", encoding="utf-8") as f:
|
|
36
|
-
f.write(content)
|
|
37
|
-
return f"edited {path}"
|
|
@@ -1,26 +0,0 @@
|
|
|
1
|
-
function_description = """Return the text of a file, optionally a slice of its lines."""
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
def read(path, offset=1, limit=None):
|
|
5
|
-
"""Read a file's UTF-8 text and return it, optionally a slice of its lines.
|
|
6
|
-
|
|
7
|
-
Argument notes:
|
|
8
|
-
offset - 1-based first line to return (default 1).
|
|
9
|
-
limit - maximum number of lines to return (default: all of them).
|
|
10
|
-
|
|
11
|
-
Behaviour:
|
|
12
|
-
- Decoding errors are replaced instead of raising, so a binary-adjacent
|
|
13
|
-
file still returns most of its text.
|
|
14
|
-
- Keeps only what you ask for so a huge file can't flood context.
|
|
15
|
-
|
|
16
|
-
Environment:
|
|
17
|
-
This evaluator runs in a project-local Python venv, not the system
|
|
18
|
-
interpreter. For a package install that is ~/.pi/agent/pi-repl-venv; for a
|
|
19
|
-
repo checkout it is .venv/. python / pip on PATH may point elsewhere, so
|
|
20
|
-
do not assume the system python is what's running.
|
|
21
|
-
"""
|
|
22
|
-
with open(path, encoding="utf-8", errors="replace") as f:
|
|
23
|
-
lines = f.readlines()
|
|
24
|
-
start = max(0, (offset or 1) - 1)
|
|
25
|
-
end = None if limit is None else start + limit
|
|
26
|
-
return "".join(lines[start:end])
|
|
@@ -1,23 +0,0 @@
|
|
|
1
|
-
function_description = """Write content to a file, creating it or overwriting its entire contents."""
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
def write(path, content):
|
|
5
|
-
"""Write a file wholesale, creating it if missing or replacing its contents.
|
|
6
|
-
|
|
7
|
-
Argument notes:
|
|
8
|
-
content - string (or anything stringifiable) to write in full.
|
|
9
|
-
|
|
10
|
-
Behaviour:
|
|
11
|
-
- Unconditional: overwrites whatever is there. There is no size cap.
|
|
12
|
-
- Use edit() for a targeted change inside an existing file; write() is for
|
|
13
|
-
a new file or a full rewrite.
|
|
14
|
-
|
|
15
|
-
Environment:
|
|
16
|
-
This evaluator runs in a project-local Python venv, not the system
|
|
17
|
-
interpreter. For a package install that is ~/.pi/agent/pi-repl-venv; for a
|
|
18
|
-
repo checkout it is .venv/. Files you write are real files on disk in the
|
|
19
|
-
working directory, visible to the host and other processes.
|
|
20
|
-
"""
|
|
21
|
-
with open(path, "w", encoding="utf-8") as f:
|
|
22
|
-
f.write(content if isinstance(content, str) else str(content))
|
|
23
|
-
return f"wrote {path} ({len(str(content))} chars)"
|