@damurka/jovian 0.1.0

package/docs/guides/environments.md

@@ -0,0 +1,80 @@
+# R and Python environments
+
+Jovian does not bundle R or Python. Each session points at an installation you choose, so different sessions (or a restarted one) can use different versions.
+
+## R
+
+```typescript
+await manager.createSession({
+    kernelType: 'r',
+    rHome: 'C:/Program Files/R/R-4.6.0',          // R_HOME — required
+    rPath: 'C:/Program Files/R/R-4.6.0/bin/x64',  // Windows: where R.dll is (default: <rHome>/bin/x64)
+    rLibs: 'D:/r-libs',                            // extra library path, where hera goes
+    pandocPath: 'C:/tools/pandoc',                 // optional
+});
+```
+
+| Option | Effect |
+|---|---|
+| `rHome` | Becomes `R_HOME` for the kernel. Find yours with `R RHOME` (or `Sys.getenv("R_HOME")` inside R). On Linux/macOS Elara loads `<rHome>/lib/libR.so` / `libR.dylib`; on Windows `R.dll` is found through `PATH`. |
+| `rPath` | Windows: directory containing `R.dll`/`Rblas.dll`, prepended to the kernel's `PATH` *before* it starts (an executable's DLL imports are resolved at process load). Defaults to `<rHome>/bin/x64`. |
+| `rLibs` | Sets `R_LIBS` and `R_LIBS_USER` (and `R_LIBS_SITE` on Windows). Packages — including `hera` — are looked up and auto-installed here. |
+| `pandocPath` | Sets `RSTUDIO_PANDOC` and prepends the directory to `PATH`, so R Markdown-style rendering works without a system pandoc. |
+
+### The `hera` package (required)
+
+Elara delegates code execution, completion, inspection and comms to the R package `hera` (`packages/hera`). **Without it R code cannot run**: the kernel still starts (it logs `WARNING: 'hera' package could not be loaded`), but every `execute` fails with `R evaluation of hera:::hera_call("execute", ...) failed (is the 'hera' package installed?)`.
+
+Install it once into the library the session uses:
+
+```r
+install.packages(c("cli", "evaluate", "glue", "IRdisplay", "jsonlite", "R6", "repr", "rlang"))
+```
+```sh
+npm run hera:install                   # = R CMD INSTALL packages/hera, from the repo root
+```
+
+or let Elara install it: set `heraSrcPath` to the `packages/hera` directory (needs the `remotes` package). At start-up Elara then installs `hera` if it is missing **or older than the source** (it compares source-file modification times with the installed `DESCRIPTION`, so editing `packages/hera/R/*.R` takes effect on the next session start). The kernel log (Themisto re-prints it with an `[elara]` prefix) says what happened: `Successfully loaded 'hera' package`, `auto-installed from ELARA_HERA_SRC`, `older than ELARA_HERA_SRC -- reinstalled`, or `WARNING: 'hera' package could not be loaded (status: …)` with one of `no_source_configured`, `source_not_found`, `remotes_unavailable`, `install_failed`.
+
+`heraSrcPath` has **no default** — without it an R session uses whichever `hera` is already installed. Keep that copy current (`npm run hera:install`, or `heraSrcPath`): `hera` >= 0.6.0.9001 streams the output of a single long-running expression live (see [Kernels](../kernels.md#executing-code-hera)); an older one still works but shows such output only when the expression ends.
+
+### Notes
+
+- **Shared library.** On Linux, R must be built with `--enable-R-shlib` (packaged R is). Elara fails with an actionable message if `libR.so` is not under `<rHome>/lib`.
+- **R ≥ 4.2 on Windows** for `readline()` over the stdin channel; older R still runs code.
+- **Encoding.** On Windows Elara switches R's locale to UTF-8 (`Sys.setlocale('LC_ALL', '.UTF-8')`) so UTF-8 content does not raise native-encoding warnings.
+- The working directory is the process's — set `workingDirectory` rather than calling `setwd()` in every session if all your relative paths hang off one project folder.
+
+## Python
+
+```typescript
+await manager.createSession({
+    kernelType: 'python',
+    pythonHome: '/usr',                       // the prefix that contains lib/libpython3.x.so (or python3NN.dll on Windows)
+    venvPath: '/projects/analysis/.venv',     // optional: make this venv's packages importable
+    pythonPath: '/projects/shared',           // optional: extra PYTHONPATH
+});
+```
+
+| Option | Effect |
+|---|---|
+| `pythonHome` | Becomes `PYTHONHOME`. Carpo scans it for the newest `python3NN.dll` (Windows, directly under the prefix), `lib/libpython3.*.so*` (Linux) or `lib/libpython3.*.dylib` (macOS) and loads it. Find it with `python -c "import sys; print(sys.base_prefix)"`. |
+| `pythonPath` | Sets `PYTHONPATH`. |
+| `venvPath` | Sets `CARPO_VENV_PATH`; on start-up the bootstrap prepends the venv's `site-packages` (`Lib/site-packages` on Windows, `lib/pythonX.Y/site-packages` on POSIX) to `sys.path`. |
+
+**venvs:** an embedded interpreter needs the *base* installation's libpython and standard library, so `pythonHome` must point at the **base** install (`sys.base_prefix`), never at the venv (inside a venv `sys.prefix` has neither) — and `venvPath` is what makes the venv's packages importable.
+
+**No Python installed?** Sessions of `kernelType: 'python'` fail to create: `createSession()` rejects with `Kernel process exited before it could register …` and the kernel's stderr (`[carpo] …`) says `No python3NN.dll was found directly under python_home …` / `No libpython3.*.so*/.dylib was found under …`. A Python without its shared library (some Linux/pyenv builds without `--enable-shared`) will not work either.
+
+## Where kernels look for things — summary
+
+| Setting | R | Python |
+|---|---|---|
+| Installation | `rHome` | `pythonHome` |
+| Extra packages | `rLibs` | `venvPath` (+ `pythonPath`) |
+| Shared-library path (POSIX) | `<rHome>/lib` added to `LD_LIBRARY_PATH` / `DYLD_LIBRARY_PATH` by the supervisor before the kernel starts | `<pythonHome>/lib`, likewise |
+| Working directory | `workingDirectory` | `workingDirectory` |
+
+## Choosing at runtime
+
+Your application decides the paths; Jovian only consumes them. The playground has a discovery module (`tools/playground/lib/env.mjs`) that finds R and Python on Windows, macOS and Linux (env vars, `R RHOME`, `python3`, the Windows registry, common install locations) — a good reference if you need the same.

package/docs/guides/history.md

@@ -0,0 +1,44 @@
+# History
+
+There are two different records of what a session has run. They answer different questions.
+
+| | `session.getHistory()` | `session.queryKernelHistory()` |
+|---|---|---|
+| Kept by | the `Session` object (client side) | the kernel (`InMemoryHistoryManager`) |
+| Contains | code **and every message it produced** (output, errors, plots) | input code only (the kernel does not record output; `output: true` returns empty strings) |
+| Covers | executions made through *this* `Session` object | everything the kernel ran, from any client, since the kernel started |
+| Survives `restart()` | yes | no — a new kernel starts empty |
+| Survives the client process | no | as long as the kernel process |
+| Limit | latest 200 executions; per execution, the newest ~500 000 characters of stdout/stderr (`entry.truncated`) | the kernel's in-memory list |
+| Cost | free (a local array) | a round trip to the kernel; queued behind a running execution |
+
+## `getHistory()`
+
+```typescript
+await session.execute('1 + 1');
+for (const entry of session.getHistory()) {
+    console.log(entry.executionCount, entry.code, entry.messages.map((m) => m.msgType));
+}
+```
+
+Entries are built from the kernel's own `execute_input` message, so they reflect exactly what ran; a silent execution (`silent: true`) publishes no `execute_input` and is not recorded. The messages are the same objects as in `ExecutionResult.output`, plus the trailing `status` and `execute_reply` messages. It is the live array — copy it if you need a snapshot. The playground uses it to rebuild a transcript after a browser refresh.
+
+**Output is bounded.** A cell that prints millions of lines (`for (i in 1:5e6) print(i)`) would otherwise make `getHistory()` — and anything that serializes it — hold and ship hundreds of megabytes. Per entry, only the newest ~500 000 characters of stream text are kept: older `stream` messages are dropped and `entry.truncated` is set to `true`. Results, display data and errors are never dropped. (`ExecutionResult.output` from `execute()` itself is not bounded.)
+
+## `queryKernelHistory()`
+
+```typescript
+const last = await session.queryKernelHistory({ n: 20 });      // [[0, 1, '1 + 1'], [0, 2, 'x <- 3'], …]
+const found = await session.queryKernelHistory({ histAccessType: 'search', pattern: '*mean*', unique: true });
+const some = await session.queryKernelHistory({ histAccessType: 'range', session: 0, start: 1, stop: 5 });
+```
+
+Each entry is `[session, lineNumber, input]` (`session` is always `0`; `lineNumber` is the execution count). Access types:
+
+| `histAccessType` | Fields | Kernel-side defaults (if you send nothing) |
+|---|---|---|
+| `'tail'` (Jovian's default) | `n`, `raw`, `output` | `n`: the library sends **100**; the kernel's own default is 10 |
+| `'search'` | `pattern` (glob: `*`, `?`), `n`, `unique`, `raw`, `output` | pattern `*`, `n` 10, `unique` false |
+| `'range'` | `session`, `start`, `stop`, `raw`, `output` | session 0, start 1, stop 10 |
+
+Only `store_history: true` (the default), non-silent executions are stored. An `execute()` that failed is stored too.

package/docs/guides/interactive-input.md

@@ -0,0 +1,48 @@
+# Interactive input
+
+`input()` (Python) and `readline()` / `readLines(stdin())` (R) block the kernel until somebody answers. Jovian carries the question over the Jupyter **stdin channel** so your application can ask its user and reply.
+
+## Opt in per call
+
+```typescript
+session.on('error', () => {});
+session.on('input_request', ({ prompt, password }) => {
+    session.sendInputReply(askTheUser(prompt, password));   // sync or async — reply whenever you have it
+});
+
+await session.execute('name = input("name? ")\nprint("hello", name)', { allowStdin: true });
+```
+
+- `allowStdin: true` is required **per `execute()` call** (it maps to the request's `allow_stdin`). The kernel then sends an `input_request` on the stdin channel; the session emits it as the `'input_request'` event with `{ prompt, password }` (`password` is `false` for `input()` / `readline()` today).
+- `sendInputReply(value)` sends the `input_reply` and unblocks the kernel. It is fire-and-forget; the result of the answer is the still-running `execute()` finishing.
+- The `input_request` is also a `'message'` event (`msgType: 'input_request'`, `channel: 'stdin'`).
+
+## Without `allowStdin`
+
+A blocking read fails fast instead of hanging the kernel forever waiting for an answer nobody will give. The kernel raises this error, whose text is:
+
+> This execution didn't allow interactive input (allow_stdin was false) -- the caller needs to opt in (e.g. execute(code, { allowStdin: true })) and be ready to answer an input_request for a blocking read like this to work.
+
+In Python it surfaces as a `RuntimeError` (so `execute()` resolves with `success: false`); in R it is reported on stderr (prefixed `input:`) and the read gets no input; the execution does not hang.
+
+## Timeouts
+
+The execution's `timeout` (default 30 s) is **cleared** the moment an `input_request` arrives — a slow human cannot trip it. Consequences:
+
+- If you never answer, the execution waits forever. Give your own UI a cancel path; if you want to abandon the kernel, `restart()` it.
+- While the kernel waits for input, `interrupt()` cannot break the read (the kernel thread is blocked in a ZMQ receive). Answer it, then interrupt if needed.
+
+## R on Windows
+
+R's `readline()` on Windows reaches the stdin channel only because Elara starts R with its own `ReadConsole` callback (R ≥ 4.2). On older R the call cannot be answered — see [Kernels](../kernels.md#elara-r).
+
+## Multiple prompts
+
+Each `input()` / `readline()` produces its own `input_request`; answer them in order:
+
+```typescript
+const answers = ['5', '7'];
+let i = 0;
+session.on('input_request', () => session.sendInputReply(answers[i++]));
+await session.execute('a = float(input("first: "))\nb = float(input("second: "))\nprint(a + b)', { allowStdin: true });
+```

package/docs/guides/interrupting.md

@@ -0,0 +1,45 @@
+# Interrupting running code
+
+`session.interrupt()` stops code that is currently running — a `Sys.sleep(60)`, a `while (TRUE) {}` loop, a Python `time.sleep(60)`, a `while True: pass` — the same way pressing Ctrl-C in a terminal would. The kernel keeps running and accepts the next `execute()`.
+
+```typescript
+const running = session.execute('Sys.sleep(60)', { timeout: 0 });
+setTimeout(async () => {
+    const acknowledged = await session.interrupt();     // true — answered while the sleep is still running
+    console.log('acknowledged:', acknowledged);
+}, 1000);
+
+const result = await running;                           // resolves within about a second
+console.log(result.success);                            // false — the execution was interrupted
+```
+
+## What you get back
+
+| | |
+|---|---|
+| `interrupt()` | `Promise<boolean>`: `true` if the kernel answered `interrupt_reply` with `status: 'ok'`, `false` if it did not within the timeout (default 5 000 ms, `interrupt({ timeout })`) or the session is gone. **Never rejects.** |
+| The interrupted `execute()` | Resolves (not rejects) with `success: false` and an `error` message in `output`. R reports an interrupt/user-break condition; Python reports `KeyboardInterrupt` (`ename: 'KeyboardInterrupt'`). |
+| Interrupting an idle kernel | Acknowledged (`true`) and otherwise a no-op. It does **not** affect the next execution. |
+
+If you gave the execution a `timeout` (default 30 s) and it fires first, `execute()` rejects with `Execution timed out after <n>ms (the kernel was interrupted)` — **the library interrupts the kernel for you**, so the code stops instead of running on unseen. Pass `interruptOnTimeout: false` to reject without interrupting; then the kernel keeps running your code (and everything after it, including `complete()` / `inspect()`, waits behind it) until you call `interrupt()` yourself or `restart()`.
+
+## How it works
+
+The kernel executes code on its main thread, which is also the thread that reads sockets, so normally a control message would wait for the running code to finish. Instead, while code runs, a **control watcher thread** reads the control socket and handles `interrupt_request` immediately (details in [Architecture](../architecture/overview.md#inside-a-kernel-process-elara--carpo)):
+
+- **R** — sets R's pending-interrupt flag (`R_interrupts_pending` on Linux/macOS, `UserBreak` on Windows). R notices at its next `R_CheckUserInterrupt()`, which its evaluator and its `Sys.sleep()` call regularly, and unwinds to top level.
+- **Python** — delivers a real `SIGINT` to the interpreter thread (`raise(SIGINT)` on Windows, `pthread_kill` on POSIX), which Python's own handler turns into `KeyboardInterrupt` and which also wakes a blocked `time.sleep()`. (The bootstrap installs `signal.default_int_handler` explicitly, because a process launched by a supervisor may have started with SIGINT ignored.) `PyErr_SetInterrupt()` alone would not wake `time.sleep()`.
+
+Other control messages that arrive during an execution (for example a `shutdown_request` from `stop()`) are queued and handled after it finishes, in order.
+
+## Limits
+
+- **Native code that does not return to the interpreter.** A long C-extension call, a blocking socket read, or R code stuck in compiled code that never calls `R_CheckUserInterrupt()` is only interrupted once it returns control.
+- **Blocked on input.** A kernel waiting for an `input_request` answer is blocked in a ZMQ receive; answer it (or `restart()`) first.
+- **A stuck kernel** that ignores the interrupt: `restart()` it — or `stop()`, which force-kills after about 2 s.
+- **Race at the very end.** An interrupt sent in the instant an execution is finishing can land just after it and either be ignored or surface as a `KeyboardInterrupt` in the next moment of Python code.
+- The POSIX implementation (Linux/macOS) is the same design but was written after the Windows one was verified; CI is what exercises it there.
+
+## Jupyter frontends
+
+The generated kernelspecs declare `"interrupt_mode": "message"`, so a frontend such as JupyterLab interrupts by sending `interrupt_request` on the control channel — which now genuinely interrupts.

package/docs/guides/playground.md

@@ -0,0 +1,33 @@
+# The playground
+
+`tools/playground` is a browser UI (Next.js + React) for driving **real** R and Python sessions through jovian's public `Session` API — every session is an actual `elara` / `carpo` process spawned by `themisto`. Use it to try an installation, reproduce a kernel bug, or see how the API behaves. Its own [README](../../tools/playground/README.md) is the reference; this page is the short version.
+
+```sh
+npm run build                # once: native binaries + dist/lib (the playground uses both)
+npm run playground:install   # once: the playground has its own node_modules; CI never installs it
+npm run playground           # http://127.0.0.1:4173
+```
+
+## What to try
+
+| Feature | How |
+|---|---|
+| New R / Python session | *New Kernel Session*: pick the kernel type, R home / Python home (pre-filled from auto-discovery), library paths, and a **working directory** (empty = the server's). |
+| Run code | `Shift+Enter`. `input()` / `readline()` show an input box. R plots render as images. |
+| Completion | **As you type**: about 180 ms after you pause with the caret at the end of an identifier of two or more characters (or right after `$`, `@`, `::`, `.`), a list of candidates appears without touching your text. `Tab` accepts, `↑/↓` move, `Enter` accepts only after you have arrowed into the list, `Esc` closes. `Tab` also asks explicitly (completing the common prefix first, then a picker). Uses `session.complete()`. |
+| Inspect | Automatic: rest the mouse (~450 ms) on a word in the input or in a cell's echoed code, or rest the caret on a word after moving it (arrows, `Home`/`End`, click), and its documentation appears; it closes when the pointer leaves. Explicit: `Shift+Tab` (or `Ctrl+I`) shows help for the symbol at the caret (`mean(` works); double-click a word in the output window, or drag-select an expression and use the *Inspect* chip. Uses `session.inspect()`. |
+| Interrupt | Stops a running cell (`session.interrupt()`); try `Sys.sleep(60)` or `import time; time.sleep(60)`. |
+| Restart / Stop / Remove | The lifecycle controls call the real protocol paths (`shutdown_request` with `restart: true/false`). Stop shuts the kernel down gracefully and force-kills it if it does not exit (there is no separate Kill — it would do the same thing); it also releases a crashed session. Remove forgets the session. |
+| Heartbeat | The **KERNEL LIFECYCLE** header shows `HEARTBEAT 0.7ms`: the round trip of the kernel's heartbeat ping (`session.status()`), polled every second. Green under 100 ms, amber under 1 s, red for "no reply" / slow / a crashed kernel, grey when there is nothing to measure. It stays green while a cell runs, because the kernel answers from its own thread. |
+| Keyboard | `Shift+Enter` run, `Tab` complete, `Shift+Tab` inspect, `Ctrl+L` clear output, `Esc` interrupt (while a cell is running; `Esc` first closes an open completion list). An interrupted cell ends with "Execution interrupted". |
+| Refresh the page | Sessions, transcripts (`session.getHistory()`), versions and working directories are rebuilt from the server. |
+
+Cells run **without a timeout** by default — press Interrupt to stop one. The automatic completion and inspect paths never wait on a busy kernel: they send `noWait` and stay silent if it is running code. For the explicit ones, a kernel that is running code cannot answer until it finishes, so the UI says so ("Kernel is running code — completing when it finishes…") and shows the result as soon as the kernel is free, instead of failing. Output that would overwhelm the page (a cell printing millions of lines) is capped: each output block keeps its last 200 000 characters and says that earlier output is not shown.
+
+## Configuration
+
+Environment variables (`PLAYGROUND_PORT`, `R_HOME`, `R_PATH`, `R_LIBS`, `PYTHONHOME`, `PYTHONPATH`, `VIRTUAL_ENV`, `JOVIAN_NATIVE_DIR`, `JOVIAN_DIST_DIR`) are listed in the playground README. The server binds to `127.0.0.1` only and rejects cross-origin state-changing requests — it executes arbitrary code, so never expose it beyond localhost.
+
+## As a reference for your own app
+
+The playground is a compact example of everything `lib/` offers: SSE streaming of `'message'` events, per-session PID, memory and heartbeat from `session.status()`, transcript restoration, completion popups and an inspect tooltip built on `complete()`/`inspect()`, and stdin handling with `'input_request'` / `sendInputReply()`. Its server-side session registry is a `globalThis` singleton so hot reloads in `next dev` do not orphan running kernels.

package/docs/guides/sessions-lifecycle.md

@@ -0,0 +1,70 @@
+# Session lifecycle
+
+```
+              createSession()                     restart()
+  (nothing) ─────────────────► ready ◄──────────────────────────────┐
+                                │  │                                │
+                    stop()      │  │  kernel dies / heartbeat lost  │
+              ┌─────────────────┘  └──────────────► crashed ────────┘  (restart() recovers it,
+              ▼                                        ('exit' event)     same session id)
+           stopped   (terminal — create a new session)
+```
+
+Supervisor-side session `status` values: `starting`, `ready`, `stopped`, `crashed`.
+
+## Creating
+
+```typescript
+const session = await manager.createSession({
+    kernelType: 'r',
+    rHome: '/usr/lib/R',
+    workingDirectory: '/projects/analysis',   // where the kernel starts; must exist
+});
+```
+
+`createSession()` resolves once the kernel process has started, registered with the supervisor (up to 60 s) and the session's WebSocket is open. It rejects with the underlying reason if not (see [Troubleshooting](../troubleshooting.md)). Each session is its own OS process — create as many as you like; they share only the supervisor.
+
+## Stopping
+
+`await session.stop()`:
+
+1. The client asks the supervisor to stop the session (`DELETE /sessions/:id`).
+2. The supervisor sends the kernel a real Jupyter `shutdown_request` with `restart: false` on the control channel.
+3. It waits up to about **2 seconds** for the kernel to exit by itself, force-killing it if it does not (a kernel busy inside `shiny::runApp()` never gets to process the request — force-kill is the normal path there, not a fault).
+4. The kernel's `shutdown_reply` (and its iopub `shutdown` message) reach your `Session` and are emitted as the `'shutdown_reply'` event; then `'stopped'` fires.
+
+An orderly stop is **never** reported as a crash (`'exit'` is not emitted). A stopped session is gone from the supervisor; `restart()` on it throws.
+
+`SessionManager.stopAll()` stops every session and then kills the supervisor. Call it before exiting — otherwise the process hangs, held open by the supervisor's pipes. `killAll()` is the hard fallback.
+
+## Restarting
+
+```typescript
+await session.restart();                                    // same options, fresh interpreter
+await session.restart({ rHome: '/opt/R/4.6.0' });          // switch R installations in place
+```
+
+- The kernel gets `shutdown_request` with `restart: true` (observe it as `'shutdown_reply'` with `restart === true`), a new kernel starts under the **same session id**, and the WebSocket reconnects to the same URL. `'restarted'` fires when it is ready.
+- **Options are merged.** `restart(options)` sends `{ ...session.options, ...options }`: you only pass what changes; `workingDirectory`, `rLibs`, `heraSrcPath`, … are kept. `session.options` reflects the new values afterwards. (The supervisor's own restart endpoint replaces options wholesale; the merge happens in the client.) To clear a field, pass `undefined` or `''`.
+- The interpreter is brand new: variables, loaded packages and the kernel's history are gone. `session.getHistory()` (client-side) and your event listeners are kept.
+- Queued and running `execute()` calls reject with `Queue cleared`; pending requests with `Session is restarting`; open comms emit `'close'` (`reason: 'kernel restarted'`).
+- Restarting a session while another `restart()` for the same id is in flight waits its turn (the supervisor serialises operations per session id) — clicking "Restart" twice does not leak a kernel process.
+
+## Crashes
+
+A kernel that dies — killed from Task Manager, a segfault (e.g. `STATUS_ACCESS_VIOLATION` in a compiled R package), `os._exit()` / `quit()` — is detected by the supervisor within milliseconds (it watches the OS process handle) and you get:
+
+- the `'exit'` event with `{ reason }` — e.g. `kernel process exited unexpectedly (process exited with code 0xc0000005 (STATUS_ACCESS_VIOLATION -- a native crash, e.g. in a compiled R package))`;
+- every queued and running `execute()` rejects with `Queue cleared`, pending requests reject with `Session process exited: <reason>`, open comms emit `'close'` (`reason: 'kernel exited'`).
+
+A kernel that is *alive but unresponsive* (deadlocked, stuck in a native call) is caught later by the ZMQ heartbeat — three missed 20-second pings, so roughly a minute — and reported the same way with a `heartbeat gave up waiting for a response` reason.
+
+Recover with `await session.restart()`: same session id, fresh kernel. An `'exit'` is also emitted if the WebSocket to the supervisor drops (`WebSocket connection to the supervisor closed unexpectedly`) — that usually means the supervisor process itself ended, in which case you need a new `SessionManager`.
+
+## Timeouts vs. kernel state
+
+`execute()`'s `timeout` only stops *your promise* — the kernel keeps running the code. After a timeout, call `interrupt()` (see [Interrupting](interrupting.md)) or `restart()` before sending more work, or the next `execute()` queues behind the still-running one.
+
+## Multiple sessions
+
+Sessions are isolated processes; the supervisor multiplexes them. One session blocked in a long call does not delay another (see `examples/advanced/two-sessions.js`). Each has its own kernel state, working directory, R library path, Python interpreter and environment.

package/docs/kernels.md

@@ -0,0 +1,117 @@
+# The kernels: Elara (R) and Carpo (Python)
+
+Both kernels are the same program shape — `main()` parses a few flags, sets up environment variables, loads the language runtime dynamically, starts an Adrastea `Kernel` with a language-specific `Interpreter`, and runs its request loop on the main thread — and differ in how the interpreter turns requests into language calls. This page covers what a *user* of a session can observe and where the two differ. For the shared machinery see [Architecture](architecture/overview.md); for adding a third kernel see [C++ usage](cpp-usage.md).
+
+## Launch modes
+
+| Mode | Command line | Used by |
+|---|---|---|
+| **Supervised** | `--registration-ip <ip> --registration-port <port> --key <key>` plus the environment flags below | Themisto (what `SessionManager` uses) — the kernel binds its ZMQ ports and reports them to the supervisor's registration socket. |
+| **Connection file** | `-f <connection_file>` / `--connection-file <file>` | A Jupyter frontend (`jupyter lab`, `jupyter console`) via a kernelspec: `npm run jupyter:kernelspec` writes `kernelspec/elara/kernel.json` and `kernelspec/carpo/kernel.json` (`interrupt_mode: "message"`); install with `jupyter kernelspec install <dir> --user --name elara`. The paths to R/Python are baked into the file at generation time. Elara's connection-file mode has been exercised; Carpo's spec is generated but has not been tried against a real Jupyter install. |
+
+Environment flags — Elara: `--r-home`, `--r-path`, `--r-libs`, `--pandoc-path`, `--hera-src-path`. Carpo: `--python-home`, `--python-path`, `--venv-path`. The kernel translates them into environment variables (`R_HOME`, `PATH`, `R_LIBS`/`R_LIBS_USER`, `RSTUDIO_PANDOC`, `ELARA_HERA_SRC`; `PYTHONHOME`, `PYTHONPATH`, `CARPO_VENV_PATH`) before loading the runtime. `workingDirectory` is not a kernel flag: Themisto starts the kernel *process* in that directory (`chdir` between `fork` and `exec` on POSIX, `lpCurrentDirectory` on Windows) after checking it exists.
+
+Kernel stdout/stderr (start-up notes such as `[R Interpreter] .libPaths() …`) is captured by Themisto and re-printed on its own stderr, prefixed `[elara]` / `[carpo]`.
+
+## Elara (R)
+
+### Start-up
+
+1. `R_HOME`, `PATH` (R's `bin` directory) and `R_LIBS*` are set; R's shared library is loaded (`R.dll` / `libR.so` / `libR.dylib` — see [Architecture](architecture/overview.md#dynamic-loading-of-r-and-python)).
+2. R is started. On Windows (R ≥ 4.2) with the documented `Rstart` embedding sequence and an Elara `ReadConsole` callback, so `readline()` works over the stdin channel and R is marked interactive; on older R via plain `Rf_initEmbeddedR` (no `readline()` support). Elara also calls `GA_initapp` on Windows, without which the first `plot()` crashes the process. On Linux/macOS it sets `R_Interactive` for the same `readline()` reason.
+3. The locale is switched to UTF-8 on Windows.
+4. `hera` is loaded — installed first from `heraSrcPath` if missing or stale (see [Environments](guides/environments.md#the-hera-package-required)). If it cannot be loaded the kernel still starts and only logs a warning.
+
+### Executing code (`hera`)
+
+`RInterpreter::executeRequestImpl` calls `hera:::hera_call("execute", code, count, silent)`, which:
+
+- parses the code (a parse failure is an `error` with `ename` `PARSE ERROR`);
+- runs it with `evaluate::evaluate(…, stop_on_error = 1)` — evaluation stops at the first error in a cell;
+- **stdout** → `stream` `stdout`; **messages** and **warnings** → `stream` `stderr` (warnings formatted as `Warning message in <call>:` followed by the text); **errors** → an `error` message whose `traceback` is a coloured (cli) trace, and the failed `execute_reply`;
+- **plots** are drawn to a null device and, per cell, sent as one `display_data` with `text/plain` and `image/png` (`options(jupyter.plot_mimetypes)`; `ggplot` objects are printed);
+- the **visible value** becomes an `execute_result` whose `data` is a mime bundle from `IRdisplay::prepare_mimebundle` — `text/plain` normally, plus `text/html` for htmlwidgets, Shiny tags and R help pages;
+- `user_expressions` are evaluated afterwards with `eval(parse(text = expr))` + `print()` capture in the global environment; each fails independently.
+
+Code runs in the **global environment**. R output that goes through R's console writer is streamed as it happens (`WriteConsoleEx` hook), not batched — subject to the ~50 ms / 16 KB coalescing every kernel applies (see [Protocol](protocol.md#1-jupyter-messages-the-kernels-support)).
+
+**Live output inside one long expression.** `evaluate` captures stdout into a temporary file and normally hands it over only when a top-level expression ends (or a message/warning/plot happens), so `for (i in 1:1e6) print(i)` would show nothing until it was over. `hera` (`patch_evaluate_sink()` in `packages/hera/R/execute.R`) wraps the one internal `evaluate` function that creates that sink so it is a *split* sink (`sink(con, split = TRUE)`): the same output is also teed to R's console, and the console hook publishes it immediately; `try()` output is pointed at the same place, and `evaluate`'s own stdout handler is then silenced so nothing arrives twice. A silent execution is not teed. This needs **`hera` >= 0.6.0.9001**; with an older installed `hera` everything still works but such output arrives when the expression ends. Update it with `npm run hera:install`.
+
+Rich output from R code: `hera::display_data(list("text/html" = "<b>hi</b>"))` publishes a `display_data`; `hera::clear_output(wait = FALSE)` publishes `clear_output`; `hera:::update_display_data()` publishes `update_display_data` (not exported).
+
+### Completion, inspection, is_complete
+
+- `complete_request` → `hera::complete()` (uses `utils:::.completeToken`); `matches` are R names (`print`, not `print(`).
+- `inspect_request` → the token at the cursor is evaluated; for a function its help page (HTML + text) is returned, otherwise sections *Class attribute*, *Printed form*, *Help document*.
+- `is_complete_request` → `R_ParseVector` status: `complete` / `incomplete` / `invalid`.
+
+### Comms
+
+The full comm API is available through `hera` (`CommManager`, `Comm`) — see [Comms](guides/comms.md). `hera` depends on `jsonlite`, `R6`, `glue` and `cli`.
+
+### Interrupt
+
+Sets R's user-break flag (`R_interrupts_pending` on POSIX, `UserBreak` on Windows) from the control-watcher thread; R unwinds at its next interrupt check. `kernel_info` reports `language_info.name` `R` and R's `major.minor` version (`implementation` is reported as `xr`).
+
+### Shiny
+
+`session.createShiny({ appDir })` runs `shiny::runApp()` in the session. It blocks the kernel for as long as the app runs (so nothing else can execute in that session), and the supervisor's stop path force-kills it because the interpreter never returns to process `shutdown_request`. Use `interrupt()` to stop the app without killing the session.
+
+## Carpo (Python)
+
+### Start-up
+
+`PYTHONHOME` / `PYTHONPATH` / `CARPO_VENV_PATH` are set, `libpython` is located and loaded (see [Architecture](architecture/overview.md#dynamic-loading-of-r-and-python)) and `Py_Initialize()` runs on the kernel's main thread. Then a **bootstrap module** — Python source embedded in `interpreter_py.cpp` (`kBootstrapSource`) — is executed once in its own private namespace, so none of its helpers appear in the user's `globals()`. It defines `__carpo_run`, `__carpo_is_complete`, `__carpo_complete`, `__carpo_inspect`, `__carpo_eval_expr`, activates the venv's `site-packages`, replaces `builtins.input`, and installs `signal.default_int_handler` for SIGINT (see [Interrupting](guides/interrupting.md)).
+
+### Executing code
+
+`PyInterpreter::executeRequestImpl` calls `__carpo_run(code, __main__.__dict__)`:
+
+- the code is parsed with `ast`; if the last statement is a bare expression it is evaluated separately and, if the value is not `None`, published as an `execute_result` with `text/plain` = `repr(value)` — exactly what a REPL shows;
+- `sys.stdout` / `sys.stderr` are redirected to a stream that calls a native callback on every `write()`, so output streams **as it is written**, not at the end (published through the same ~50 ms / 16 KB coalescing as R, so a flood of `print()` calls becomes far fewer messages);
+- any exception is caught (`BaseException`, so `KeyboardInterrupt` too) and returned as `ename` (type name), `evalue` (`str(e)`) and `traceback` (formatted lines);
+- `user_expressions` are `eval`'d in the same globals and `repr`'d.
+
+Code runs in the `__main__` namespace and state persists across executions and across `execute()` calls of the same session.
+
+### Completion, inspection, is_complete
+
+- `complete` → `rlcompleter` on the token before the cursor; matches include the call paren for functions (`print(`).
+- `inspect` → `eval`s the token: signature (when available), `Type: <name>`, then the docstring or `repr`. `text/plain` only.
+- `is_complete` → `codeop.compile_command`.
+
+### `input()`
+
+`builtins.input` is replaced with a function that goes through the stdin channel. Without `allow_stdin` it raises `RuntimeError` with the message quoted in [Interactive input](guides/interactive-input.md).
+
+### venvs
+
+`pythonHome` is the **base** installation; `venvPath` adds that venv's `site-packages` to `sys.path`. See [Environments](guides/environments.md#python).
+
+### Interrupt
+
+A real SIGINT delivered to the interpreter thread; Python raises `KeyboardInterrupt` (also waking a blocked `time.sleep()`), which `__carpo_run` returns as an ordinary `error`.
+
+## Differences at a glance
+
+| | Elara (R) | Carpo (Python) |
+|---|---|---|
+| Runtime dependency | R (+ the `hera` R package, + CRAN deps) | CPython 3 with its shared library |
+| Global scope | `.GlobalEnv` | `__main__` |
+| Rich output | `display_data`, plots (`image/png`), `text/html`, `update_display_data`, `clear_output` (via `hera`) | `execute_result` `text/plain` only — no `display_data` / plots yet |
+| stderr stream | messages and warnings | `sys.stderr` |
+| Completions | R names | `rlcompleter` (with `(` for callables) |
+| Inspect | help pages (HTML + text) | signature / docstring (text) |
+| Comms | yes (`hera::CommManager`) | no targets can be registered |
+| Interrupt | user-break flag | SIGINT → `KeyboardInterrupt` |
+| `kernel_info` | `R`, `implementation: "xr"` | `python`, `implementation: "carpo"`, banner `carpo (Python x.y.z)` |
+| Extra environments | `rLibs` | `venvPath`, `pythonPath` |
+
+## Known limits (both kernels)
+
+- **One thing at a time.** A kernel is single-threaded: requests other than `interrupt_request` wait for a running execution. Use separate sessions for concurrency.
+- **Interrupts cannot break native code** that never returns to the interpreter, nor a read blocked on `input()`.
+- **History is in-memory**, holds inputs only, and is lost on restart.
+- **No debugger protocol**, no `input_reply` password prompts (`password` is always `false`).
+- **Shiny / long calls block the session**; the supervisor's stop path force-kills such a kernel after ~2 s.
+- **`inspect_request.detail_level`** is accepted but ignored.

package/docs/protocol.md

@@ -0,0 +1,135 @@
+# Protocol reference
+
+Two protocols matter in Jovian:
+
+1. **Jupyter messaging over ZMQ** between Themisto and each kernel (`elara` / `carpo`) — the standard protocol, version **5.6** (what `kernel_info_reply.protocol_version` reports).
+2. **Themisto's HTTP + WebSocket API** between the TypeScript client (`lib/`) and Themisto — Jovian's own, thin, JSON API. You normally never touch it directly (`Session` does), but it is stable enough to script against.
+
+## 1. Jupyter messages the kernels support
+
+Handlers are registered in `KernelCore` (`native/src/adrastea/core/kernel/kernel_core.cpp`) and are the same for Elara and Carpo; the language-specific part is delegated to the interpreter.
+
+| Message | Channel | Reply | Notes |
+|---|---|---|---|
+| `execute_request` | shell | `execute_reply` | Fields honoured: `code`, `silent`, `store_history`, `allow_stdin`, `stop_on_error`, `user_expressions`. Output arrives on iopub while it runs. |
+| `complete_request` | shell | `complete_reply` | `matches`, `cursor_start`, `cursor_end`. |
+| `inspect_request` | shell | `inspect_reply` | `found`, `data` (mime bundle). `detail_level` is accepted but ignored by both kernels. |
+| `is_complete_request` | shell | `is_complete_reply` | `status`: `complete` / `incomplete` / `invalid` (Carpo may answer `unknown` if its helper fails). |
+| `kernel_info_request` | shell | `kernel_info_reply` | Adds `protocol_version`. |
+| `history_request` | shell | `history_reply` | `hist_access_type` = `tail` (default) / `range` / `search`; see [History](guides/history.md). |
+| `comm_info_request` | shell | `comm_info_reply` | Optional `target_name` filter. |
+| `comm_open` / `comm_msg` / `comm_close` | shell (client → kernel); iopub (kernel → client) | — | See [Comms](guides/comms.md). |
+| `interrupt_request` | control | `interrupt_reply` | Serviced **while code runs** (see [Architecture](architecture/overview.md#inside-a-kernel-process-elara--carpo)). |
+| `shutdown_request` | control | `shutdown_reply` | Content `{restart}`; the reply echoes it. The kernel also publishes an iopub `shutdown` message. |
+| `input_request` (kernel → client) / `input_reply` | stdin | — | Only sent when the `execute_request` had `allow_stdin: true`. |
+
+iopub messages the kernels publish: `status` (`busy` / `idle` around **every** request the kernel handles, not just executes; `starting` once at start-up), `execute_input`, `stream` (`stdout` / `stderr`), `display_data`, `update_display_data`, `clear_output`, `execute_result`, `error`, `comm_open` / `comm_msg` / `comm_close`, `shutdown`, `interrupt`, and an `iopub_welcome`.
+
+**`stream` messages are coalesced.** A `print()` in a loop is one or two writes per line, and publishing each write on its own would flood ZMQ, the relay and the client far faster than they drain. The interpreter base class (`adrastea::Interpreter::publishStream`) therefore buffers stdout/stderr text and publishes it as **one `stream` message at most every ~50 ms, or as soon as 16 KB have accumulated** (the first write after a quiet period goes out at once). A different stream (stdout → stderr) starts a new message. Ordering is preserved: the buffer is flushed before any other output (`execute_result`, `display_data`, `update_display_data`, `error`, `clear_output`), before an `input_request` (so a prompt printed just before it comes first), and before the `execute_reply`; a small flusher thread flushes stale text while the interpreter is busy computing without writing. Chunk boundaries are therefore up to the kernel: a token such as `tick 3` may be split across two messages, so match on accumulated text.
+
+### `execute_request` details
+
+- `stop_on_error: true` — if this request fails, requests already queued behind it on the kernel's shell socket get a reply with `status: "aborted"` instead of running. (The TypeScript `ExecutionQueue` is single-flight, so it additionally aborts *its own* queued items — see [the API reference](api/session.md).)
+- `user_expressions` — `{name: "expression"}`, evaluated **after** the code and **only if it succeeded**; the reply's `user_expressions` holds `{name: {status: "ok", data: {"text/plain": …}, metadata: {}}}` or `{name: {status: "error", ename, evalue, traceback: []}}` per expression. R evaluates in the global environment via `print()` capture; Python via `eval()` + `repr()`.
+- `silent: true` — no `execute_input`, no execution-count increment, no history entry.
+- Kernel replies use `status`: `ok`, `error` (with `ename`, `evalue`, `traceback`), or `aborted`.
+
+### How each kernel implements the language-specific requests
+
+| Request | Elara (R) | Carpo (Python) |
+|---|---|---|
+| execute | `hera:::hera_call("execute", …)` — `evaluate` with output handlers | `__carpo_run(code, globals)` in the bootstrap module (`ast` split; last expression auto-displayed) |
+| complete | `hera` → `utils:::.completeToken` | `rlcompleter` |
+| inspect | `hera` — class, printed form and help (HTML + text) | `inspect.signature`, `Type:`, docstring or `repr` (text/plain only) |
+| is_complete | `R_ParseVector` status | `codeop.compile_command` |
+| `kernel_info` | `implementation: "xr"`, `language_info.name: "R"`, version = R's | `implementation: "carpo"`, `language_info.name: "python"`, version = Python's, `banner: "carpo (Python x.y.z)"` |
+| interrupt | sets `R_interrupts_pending` / `UserBreak` | real SIGINT → `KeyboardInterrupt` |
+| comms | full: `hera::CommManager`, `Comm` | none — nothing in the bootstrap registers comm targets, so every `comm_open` is answered with a `comm_close` and `comm_info` is empty |
+
+## 2. Themisto's HTTP API
+
+Base URL `http://127.0.0.1:<httpPort>`; both ports are announced on Themisto's stdout at start-up as one JSON line:
+
+```json
+{"type":"supervisorReady","httpPort":51662,"wsPort":58722}
+```
+
+Bodies are JSON. Errors are `{"error": "<message>"}`.
+
+### `POST /sessions` — create
+
+Request body (every field optional; unset means "empty"):
+
+| Field | Type | Meaning |
+|---|---|---|
+| `kernelType` | `"r"` \| `"python"` | Default `"r"`. A type with no kernel executable available fails just this call. |
+| `rHome`, `rPath`, `rLibs` | string | R installation (`R_HOME`), directory containing `R.dll` (Windows), extra library path (`R_LIBS`). |
+| `pandocPath` | string | Directory of a pandoc binary (`RSTUDIO_PANDOC`, added to `PATH`). |
+| `heraSrcPath` | string | Source of the `hera` package, installed on demand (`ELARA_HERA_SRC`). |
+| `pythonHome`, `pythonPath`, `venvPath` | string | Python prefix (`PYTHONHOME`), extra `PYTHONPATH`, venv whose `site-packages` is added to `sys.path`. |
+| `workingDirectory` | string | Directory the kernel process starts in. Must exist. |
+
+Responses:
+
+- **200** — the session object: `{"sessionId", "status", "kernelType", "workingDirectory", "pid", "memoryBytes", "heartbeat"}`. `status` is `starting` \| `ready` \| `stopped` \| `crashed`; `memoryBytes` is `null` when unavailable; `heartbeat` is `{"hasPong", "rttMs", "sinceLastPongMs", "misses"}` (see below) or `null` when the session has no client.
+- **400** — `{"error":"invalid JSON body"}`.
+- **500** — `{"error": …}`: no kernel executable for that `kernelType`; `workingDirectory does not exist or is not a directory: <path>`; the kernel failed to register (R not found, …).
+
+### `GET /sessions` and `GET /sessions/:id`
+
+`{"sessions":[<session object>, …]}` and `<session object>`; **404** `{"error":"session not found"}` for an unknown id. `Session.status()` is a `GET /sessions/:id`.
+
+The session object's `heartbeat` reports the supervisor's ping to the kernel's heartbeat channel: `hasPong` (false until the first ping is answered), `rttMs` (round trip of the last answered ping), `sinceLastPongMs` (age of that answer) and `misses` (consecutive unanswered pings; `0` is healthy). The kernel answers from a thread separate from the one running code, so it stays fresh while a cell runs; it is a liveness signal, not a "kernel is free" signal.
+
+### `DELETE /sessions/:id` — stop
+
+Sends the kernel a `shutdown_request` with `restart: false`, waits up to ~2 s for a clean exit, then force-kills, and removes the session. **200** `{"sessionId","status":"stopped"}`; **404** if unknown. A stopped session id is gone for good.
+
+### `POST /sessions/:id/restart` — restart
+
+Optional body, same shape as create. **Without a body** the session's original options are reused; **with one** it replaces them *wholesale* — unset fields become empty, they are *not* merged with the old values. (`Session.restart(options)` in `lib/` does the merge for you before sending.) The old kernel gets `shutdown_request{restart:true}`; a new kernel starts under the **same session id**. **200** `{"sessionId","status":"ready"}`; **500** `{"error"}`.
+
+## 3. Themisto's WebSocket API
+
+Connect to `ws://127.0.0.1:<wsPort>/sessions/<sessionId>/messages`. An unknown session id closes the socket with code **1008** (`unknown session`). At most one connection per session is relayed to at a time: the most recent one owns the kernel's output stream. All frames are JSON text.
+
+### Client → Themisto
+
+| Frame | Fields | Effect |
+|---|---|---|
+| `execute` | `id`, `code`, `options` (`silent`, `storeHistory`, `allowStdin`, `stopOnError`, `userExpressions`) | Sends `execute_request` on shell with `msg_id = id`. Other option keys (e.g. `timeout`) are ignored natively. |
+| `inputReply` | `value` | Sends `input_reply` on the stdin channel — the only thing that unblocks a pending `input()` / `readline()`. |
+| `request` | `id`, `channel` (`"shell"` \| `"control"`), `msgType`, `content` | Sends any **whitelisted** request with `msg_id = id`. |
+
+`request` whitelist — anything else is refused:
+
+| Channel | Allowed `msgType` |
+|---|---|
+| shell | `complete_request`, `inspect_request`, `is_complete_request`, `kernel_info_request`, `history_request`, `comm_info_request`, `comm_open`, `comm_msg`, `comm_close` |
+| control | `interrupt_request` |
+
+`execute_request` (own frame: it owns stdin/history semantics), `input_reply` (own frame) and `shutdown_request` (a lifecycle operation — a raw one would kill the kernel behind the supervisor's back; use `DELETE` / `restart`) are deliberately excluded. Malformed frames are ignored.
+
+### Themisto → client
+
+| Frame | Fields | Meaning |
+|---|---|---|
+| `ready` | — | Sent right after the connection opens and the session is bound. |
+| `message` | `channel`, `topic`, `msg_type`, `parent_msg_id`, `content` | Every kernel message on iopub, shell, control or stdin. `channel` is `"iopub"` \| `"shell"` \| `"control"` \| `"stdin"`. `parent_msg_id` is the `id` of the frame that caused it, which is how replies are correlated. |
+| `kernelExit` | `reason` | The kernel process died unexpectedly (`kernel process exited unexpectedly (process exited with code 0x…)`) or its heartbeat gave up. Never sent for a requested stop/restart. |
+| `requestError` | `id`, `error` | A `request` frame could not be sent: `'<type>' is not an allowed request on the '<channel>' channel`, or `session not found`. No reply will follow. |
+| `log` | `level`, `message`, `data` | Handled by the TypeScript client (replayed through its logger); the native supervisor does not currently emit it. |
+
+Example — `complete_request` round trip:
+
+```json
+→ {"type":"request","id":"7f3…","channel":"shell","msgType":"complete_request","content":{"code":"pri","cursor_pos":3}}
+← {"type":"message","channel":"iopub","topic":"kernel_core.<kernel>.status","msg_type":"status","parent_msg_id":"7f3…","content":{"execution_state":"busy"}}
+← {"type":"message","channel":"shell","topic":"complete_reply","msg_type":"complete_reply","parent_msg_id":"7f3…","content":{"status":"ok","matches":["print"],"cursor_start":0,"cursor_end":3,"metadata":{}}}
+← {"type":"message","channel":"iopub","topic":"kernel_core.<kernel>.status","msg_type":"status","parent_msg_id":"7f3…","content":{"execution_state":"idle"}}
+```
+
+## 4. Ordering guarantees (and their absence)
+
+- Messages on **one** ZMQ channel arrive in order. Across channels there is **no** ordering guarantee: iopub output published before an `execute_reply` can be relayed *after* it. The TypeScript `ExecutionQueue` compensates with a short (50 ms) grace wait when an `ok` reply arrives with no output collected yet.
+- Requests other than `interrupt_request` are answered by the kernel's main thread, so they queue behind a running execution.