@damurka/jovian 0.1.0

package/docs/architecture/overview.md

@@ -0,0 +1,218 @@
+# Jovian Architecture Overview
+
+Jovian runs language kernels as supervised Jupyter kernels that a Node.js or Electron process can drive. It ships two kernels — R (**Elara**) and Python (**Carpo**) — selected per session with `EngineOptions.kernelType` (`'r'` | `'python'`, default `'r'`).
+
+## Components
+
+The parts are named after moons of Jupiter:
+
+| Name | Role | Where |
+|---|---|---|
+| **Jovian** | The umbrella product and npm package (`jovian`): a TypeScript client over the native binaries below | `lib/` |
+| **Adrastea** | Language-neutral Jupyter kernel framework: wire protocol, ZMQ transport, kernel request loop, the abstract interpreter interface (`adrastea::`). Built as a **static library** shared by Elara, Carpo and Themisto | `native/src/adrastea`, `native/include/adrastea` |
+| **Elara** | The R kernel: embeds R on top of Adrastea, loading R's shared library at runtime (`elara::`, the `elara` executable) | `native/src/elara` |
+| **Carpo** | The Python kernel: embeds CPython on top of Adrastea the same way (`carpo::`, the `carpo` executable) | `native/src/carpo` |
+| **Themisto** | The kernel supervisor: spawns and monitors one kernel process per session, speaks ZMQ to each, and re-exposes sessions over HTTP + WebSocket (`themisto::`, the `themisto` executable) | `native/src/themisto` |
+| [hera](../../packages/hera) | The R companion package loaded inside every Elara session | `packages/hera` |
+
+There is no Node-API addon and no in-process engine. `lib/` talks to Themisto over plain HTTP (session lifecycle) and WebSocket (execute, requests, message streaming), and Themisto spawns one Elara or Carpo process per session. That is deliberate: a session blocked in a long call (a Shiny app, a long Python loop) cannot starve another, because they are different processes with different interpreters, and only Themisto ever links a native ZMQ binding — the process that embeds `lib/` (an Electron main process, a VS Code extension host) needs no native dependency at all.
+
+## Process model
+
+```
+┌─────────────────────────────────────────────────────────────┐
+│  Your Node.js / Electron process                            │
+│  lib/  SessionManager ─ Session ─ ExecutionQueue ─ Comm     │
+└───────────────┬──────────────────────────┬──────────────────┘
+      HTTP (create / stop / restart)   WebSocket, one per Session
+                │                          │        (127.0.0.1 only)
+┌───────────────┴──────────────────────────┴──────────────────┐
+│  themisto  (one per SessionManager, spawned on first session)│
+│  HttpApi · WsRelay · SessionRegistry                        │
+│  per session: KernelProcess, ClientZmq, poll thread         │
+└───────┬─────────────────────────────────────────┬───────────┘
+        │ ZMQ: shell, control, stdin,             │ ZMQ (same, other session)
+        │      iopub, heartbeat (127.0.0.1)       │
+┌───────┴─────────────────┐              ┌────────┴─────────────────┐
+│ elara  (R session)      │              │ carpo  (Python session)  │
+│ Adrastea + RInterpreter │              │ Adrastea + PyInterpreter │
+└───────┬─────────────────┘              └────────┬─────────────────┘
+        │ R C API (dlopen / LoadLibrary)          │ Python C API (dynamic)
+   R + packages/hera                         CPython + bootstrap module
+```
+
+- **Session = process.** `POST /sessions` makes Themisto spawn `elara` or `carpo` (whichever `kernelType` names), wait for it to register, and connect a ZMQ client to it.
+- **Registration handshake.** Themisto binds one registration `ROUTER` socket and passes its address and a shared HMAC key on the kernel's command line (`--registration-ip`, `--registration-port`, `--key`). The kernel binds its five sockets on free ports and sends the port numbers to that registration socket (signed with the key). Themisto waits up to 60 s, aborting early if the kernel process dies first.
+- **Authentication.** Every ZMQ message is signed (`hmac-sha256`) with that key; the key is shared by all kernels one Themisto spawns.
+- **Loopback only.** Themisto's HTTP and WebSocket servers bind `127.0.0.1` on OS-assigned ports; kernel ports are `127.0.0.1` too.
+- **Lifetime.** On Windows every kernel is assigned to a job object with `KILL_ON_JOB_CLOSE`, so kernels die with Themisto however Themisto exits. `SupervisorClient.kill()` (called by `SessionManager.stopAll()/killAll()` and an `exit` handler) ends Themisto.
+
+## Channels and threads
+
+### Jupyter channels
+
+| Channel | ZMQ pattern (kernel side) | Used for |
+|---|---|---|
+| shell | ROUTER | `execute`, `complete`, `inspect`, `is_complete`, `kernel_info`, `history`, `comm_*` requests and their replies |
+| control | ROUTER | `interrupt_request`, `shutdown_request` and their replies |
+| stdin | ROUTER | `input_request` from the kernel, `input_reply` from the client |
+| iopub | XPUB, owned by a publisher thread (the main thread hands it messages over an in-process PUB) | `status`, `execute_input`, `stream`, `display_data`, `update_display_data`, `clear_output`, `execute_result`, `error`, kernel-initiated `comm_*` |
+| heartbeat | ROUTER (the supervisor's client uses REQ) | liveness pings |
+
+Themisto's client side uses DEALER sockets for shell/control/stdin, a SUB for iopub and a REQ for the heartbeat. The three DEALERs share one explicit ZMQ routing id: the kernel addresses an `input_request` on the stdin ROUTER using the identity it captured from the *shell* message that started the execution, which only reaches the client if its stdin socket presents the same identity.
+
+### Inside a kernel process (Elara / Carpo)
+
+| Thread | Job |
+|---|---|
+| **Main thread** | Runs the ZMQ poll loop (shell + control) **and** the interpreter: R or Python executes on the process's own main thread (R's C-stack-bounds detection assumes it; Python's `KeyboardInterrupt` is raised on the thread that initialised it). |
+| **Publisher** | Owns the iopub XPUB socket; the main thread hands it messages. |
+| **Heartbeat** | Answers ping/pong, on its own thread — so it keeps answering while the main thread is busy running code. |
+| **Control watcher** | Started lazily; alive for the kernel's lifetime but only *active* while code is executing (see below). |
+
+Because the interpreter runs on the same thread that reads sockets, a control message sent mid-execution would normally sit unread until the execution finished — useless for interrupt. The **control watcher** fixes that:
+
+- `KernelCore::executeRequest` brackets the interpreter call with `Server::beginExecution()` / `endExecution()`.
+- Between those calls, the watcher thread polls the control socket (every 5 ms). An `interrupt_request` is dispatched immediately, on the watcher thread: the interpreter's `interruptRequestImpl()` flags the interpreter (R: sets `R_interrupts_pending` on POSIX / `UserBreak` on Windows; Python: delivers a real SIGINT — `raise(SIGINT)` on Windows, `pthread_kill` of the interpreter thread on POSIX — after the bootstrap installed `signal.default_int_handler`), and replies on the control channel. Any *other* control message (a `shutdown_request`, say) is queued and delivered by the main loop after the execution ends, in order, exactly as before.
+- Interrupting an idle kernel is a no-op — the flag is only set while an execution is actually running, so a stale break can never abort the *next* execution.
+- Locking: the control `ROUTER` is guarded by a recursive mutex (the interrupt handler, running on the watcher thread, replies through the same send path); iopub publishing is guarded by a mutex (the watcher publishes `status`/`interrupt` messages while the main thread streams output through the same publishing socket).
+- **Stream flusher.** `Interpreter::publishStream` coalesces stdout/stderr text (one message per ~50 ms or 16 KB) and lazily starts a small flusher thread that publishes stale buffered text while the interpreter is busy computing and not writing; everything else that is published (results, display data, errors, input prompts, the `execute_reply`) flushes the buffer first, so ordering is unchanged. Publishing from that thread goes through the same iopub mutex.
+
+`interruptRequestImpl()` therefore runs on a thread other than the interpreter's. Implementations must only do thread-safe things (see [cpp-usage.md](../cpp-usage.md)).
+
+### Inside Themisto
+
+| Thread | Job |
+|---|---|
+| HTTP server | cpp-httplib listener (plus its worker threads) handling session create/list/get/delete/restart |
+| WebSocket server | ixwebsocket server threads, one connection per `Session` object on the client |
+| **Poll thread, one per session** | Every ~5 ms: checks whether the kernel OS process is still alive; drains the client's iopub queue and the shell, control and stdin channels; relays each message as a JSON frame to the attached WebSocket |
+| Client iopub / heartbeat threads | Inside each `ClientZmq`: receive iopub into a queue; run the heartbeat — a ping roughly every 100 ms, up to 20 s to wait for each answer, 3 retries. Every answer updates a `HeartbeatStatus` (round trip, age of the last pong, consecutive misses) that `sessionToJson()` publishes as the session's `heartbeat` |
+| Kernel output pump, one per kernel | Reads the kernel's stdout/stderr pipe and re-prints it on Themisto's stderr with an `[elara]` / `[carpo]` prefix |
+
+Every operation targeting a session id (`sendExecute`, `sendRequest`, `stopSession`, `restartSession`, …) takes a per-id recursive mutex so a request that arrives mid-restart waits and then addresses whichever kernel is live afterwards. `DealerChannel` additionally serialises each DEALER socket (ZMQ sockets are not thread-safe, and the poll thread receives while HTTP/WS threads send); a "blocking" receive is a loop of short non-blocking attempts so it never starves a sender.
+
+### Crash detection
+
+Two independent mechanisms, fastest first:
+
+1. **OS liveness** — the poll thread checks the kernel's process handle every iteration (~5 ms). A kernel killed from outside (Task Manager, a segfault, `os._exit()`) is reported as `kernelExit` almost at once, with the exit code decoded when recognised (e.g. `STATUS_ACCESS_VIOLATION`). The final messages the kernel published are drained first.
+2. **Heartbeat** — for a kernel that is alive but stuck (deadlocked, in a native call): 3 missed pings at 20 s each ≈ 60–80 s, reported as `kernelExit` with a `heartbeat gave up waiting for a response` reason. The same channel doubles as a health readout: every answered ping records its round trip, exposed through `GET /sessions/:id` and `Session.status()`. Because the kernel replies from a dedicated thread, a kernel that is merely *busy* (a long cell) still answers — the heartbeat separates "busy" from "stuck".
+
+A session whose stop/restart was *requested* sets an "expecting exit" flag first, so the orderly exit that follows is not reported as a crash.
+
+## Dynamic loading of R and Python
+
+Neither interpreter is linked at build time.
+
+- **R** (`native/src/elara/r/r_dynlib.{hpp,cpp}`): `R.dll` / `libR.so` / `libR.dylib` is loaded with `LoadLibrary` / `dlopen` when the kernel starts (after `R_HOME` and `PATH` are set up). Switching R installations is a runtime `rHome` decision needing no rebuild, and a missing R surfaces as a clean, catchable error inside `elara` (exit code 1, an actionable message) instead of the OS refusing to start the process. On Windows, `R.dll` exports no hookable console pointers, so `RInterpreter` starts R with the documented embedding sequence (`R_DefParamsEx`, `R_SetParams`, `setup_Rmainloop`, …) and its own `ReadConsole` callback; older R (< 4.2) falls back to `Rf_initEmbeddedR`.
+- **Python** (`native/src/carpo/py/py_dynlib.{hpp,cpp}`): the shared library is found by scanning `pythonHome` for the newest `python3NN.dll` / `libpython3.*.so*` / `libpython3.*.dylib` (the name embeds the version, unlike R's). Only the documented stable-ABI subset of the C API is used and `PyObject` stays opaque; reference counting goes through the real `Py_IncRef` / `Py_DecRef` functions.
+- **POSIX library path.** A kernel's own interpreter library is `dlopen`ed by full path, but R's base packages and Python's extension modules have the shared library as an unqualified dependency. Themisto therefore sets `LD_LIBRARY_PATH` / `DYLD_LIBRARY_PATH` (`<R_HOME>/lib`, `<PYTHONHOME>/lib`) in the child *between `fork()` and `exec()`* — setting it from inside the running kernel would be too late, since the dynamic linker reads it at process start.
+
+## Message flows
+
+### Execute
+
+```
+Session.execute(code)
+  └ ExecutionQueue (single-flight, timeout)         { type:"execute", id, code, options }
+      └ WebSocket ─────────────────────────────────────────────────────────►  WsRelay
+                                                                    SessionRegistry::sendExecute
+                                                                     └ shell: execute_request ─► kernel
+kernel main thread: KernelCore::executeRequest
+   status busy (iopub) · execute_input · [beginExecution: control watcher on]
+   RInterpreter → hera:::hera_call("execute", …)   |   PyInterpreter → __carpo_run(code, globals)
+      stream (coalesced ≤ every ~50 ms / 16 KB) / display_data / execute_result / error   (iopub)
+   [endExecution]  execute_reply (shell)  ·  status idle (iopub)
+poll thread ── relays every message as {type:"message", channel, msg_type, parent_msg_id, content}
+Session ── MessageRouter emits events; ExecutionQueue collects output for this id;
+           execute_reply resolves the promise (ExecutionResult)
+```
+
+### Request / reply (`complete`, `inspect`, `is_complete`, `kernel_info`, `history`, `comm_info`)
+
+```
+Session.complete(code, pos)
+  └ request(): id = uuid; pending[id] = {replyType:"complete_reply", timer}
+      └ WS {type:"request", id, channel:"shell", msgType:"complete_request", content}
+          └ WsRelay → SessionRegistry::sendRequest (whitelist!) → shell: complete_request ─► kernel
+kernel (between executions, on the main thread) → complete_reply (shell)
+poll thread → {type:"message", channel:"shell", msg_type:"complete_reply", parent_msg_id:id, …}
+Session.settleRequest → resolves with the reply `content`; status "error"/"aborted" rejects
+If the supervisor refuses: {type:"requestError", id, error} → rejects at once
+```
+
+Requests are answered on the kernel's main thread, so they wait for a running execution to finish. Only `interrupt_request` (control) is serviced during one.
+
+### Interactive input
+
+```
+kernel: R readline() / Python input()  ──►  adrastea::blockingInputRequest()
+   (only if the execute_request had allow_stdin; otherwise fail fast — Python raises
+    RuntimeError, R reports on stderr and reads no input)
+   kernel main thread sends input_request on the stdin ROUTER and BLOCKS on the reply
+poll thread relays it: {type:"message", channel:"stdin", msg_type:"input_request", …}
+Session emits 'input_request' {prompt, password}; the execution's timeout is cleared
+caller: session.sendInputReply(value) → WS {type:"inputReply", value}
+   → SessionRegistry::sendInputReply → stdin: input_reply → the kernel unblocks
+```
+
+### Interrupt
+
+```
+Session.interrupt() → request('interrupt_request') on channel "control"
+   → control DEALER → kernel control ROUTER
+kernel control-watcher thread (active only while code runs):
+   interrupt_request → interpreter flags R / raises SIGINT for Python
+   status busy/idle (iopub, parented to the interrupt) · interrupt_reply (control)
+interpreter thread: R notices the flag at its next R_CheckUserInterrupt() /
+   Python's handler raises KeyboardInterrupt → the execution ends with an error;
+   execute_reply arrives as usual, the kernel stays alive
+poll thread relays interrupt_reply (channel "control"); interrupt() resolves true
+```
+
+### Stop and restart
+
+```
+Session.stop()                                   Session.restart(options?)
+  DELETE /sessions/:id                             POST /sessions/:id/restart   (optional new options)
+     └ SessionRegistry::stopSession(id, false)        └ restartSession → stopSession(id, true) → spawn new kernel
+                                                          under the SAME session id
+stopSession:
+  1. expectingExit = true                             (an orderly exit is not a crash)
+  2. control: shutdown_request {restart}              (if the process is alive)
+  3. poll thread keeps running ≤ 2 s and relays iopub "shutdown" + control shutdown_reply
+  4. stop poll thread, tear down the ZMQ client, force-kill if the process is still alive
+     (normal for a session blocked inside shiny::runApp())
+  5. status = stopped; the session is removed from the registry
+Session emits 'shutdown_reply' ({status, restart}); stop() then emits 'stopped',
+restart() reconnects its WebSocket to the same URL and emits 'restarted'.
+```
+
+## Repository layout
+
+```
+native/
+├── include/{adrastea,elara,carpo}/   public headers
+├── src/
+│   ├── adrastea/   core/ (kernel, execution, messaging, history) · transport/ (server, client, common)
+│   │               · platform/ · utils/
+│   ├── elara/      r/ (routine.cpp, r_dynlib, interpreter_r.cpp) · bridge/ · elara.cpp
+│   ├── carpo/      py/ (py_dynlib) · interpreter_py.cpp · bridge/ · carpo.cpp
+│   └── themisto/   main.cpp · session_registry · kernel_process · http_api · ws_relay
+└── test/           adrastea/ · elara/ · carpo/ · themisto/   (GoogleTest, one CTest entry per feature)
+lib/                session/ (SessionManager, Session, Comm, SupervisorClient) · messaging/ · handlers/
+                    · execution/ (ExecutionQueue) · middleware/ · utils/ · types/
+packages/hera/      R companion package
+test/               unit/lib (TypeScript, no processes) · integration (real themisto + kernels)
+tools/              playground/ (Next.js) · jupyter-kernelspec/
+examples/           basic/ · advanced/
+scripts/            build.js · test.js · clean.js · dev.js · format.js · coverage.js
+docs/               this documentation
+```
+
+`adrastea` is a static library, so `elara`, `carpo` and `themisto` are standalone executables with no companion library to ship. There is no installable CMake package for it; see [C++ usage](../cpp-usage.md).
+
+## Further reading
+
+[Protocol](../protocol.md) · [Kernels](../kernels.md) · [API reference](../api/README.md) · [Development](../development.md) · [Jupyter messaging protocol](https://jupyter-client.readthedocs.io/en/stable/messaging.html) · [ZeroMQ Guide](https://zguide.zeromq.org/)

package/docs/cpp-usage.md

@@ -0,0 +1,60 @@
+# Using Jovian from C++
+
+Jovian's native layer is four CMake targets, defined in [`native/CMakeLists.txt`](../native/CMakeLists.txt):
+
+- **`adrastea`** — a static library: the language-neutral Jupyter kernel framework (transport, messaging, the kernel request loop, the abstract `Interpreter` interface). Public headers under [`native/include/adrastea/`](../native/include/adrastea).
+- **`elara`** — an executable: embeds R on top of `adrastea`. Public headers under [`native/include/elara/`](../native/include/elara).
+- **`carpo`** — an executable: embeds Python (CPython) on top of `adrastea` the same way. Public headers under [`native/include/carpo/`](../native/include/carpo).
+- **`themisto`** — an executable: the supervisor that spawns/monitors `elara`/`carpo` processes, one per session, keyed by that session's `kernelType`.
+
+## Current state: in-tree only
+
+There is no installed/exported CMake package today — no `install()`, no `<Package>Config.cmake`, no pkg-config file. `adrastea` is only ever consumed via `add_subdirectory` from within this same repository (see how `elara`/`themisto` link it in `native/CMakeLists.txt`). If you want to link against `adrastea` from a separate CMake project right now, the only supported path is:
+
+```cmake
+add_subdirectory(path/to/jovian/native adrastea-build)
+target_link_libraries(your_target PRIVATE adrastea)
+```
+
+This pulls in `adrastea`'s `PUBLIC` include directories, compile definitions (`ADRASTEA_STATIC_LIB`), and link libraries (nlohmann_json, cppzmq, OpenSSL::Crypto) automatically, the same way `elara`/`themisto` get them.
+
+Building a real `find_package(adrastea)`-style exported package (an `install(TARGETS ... EXPORT ...)` + generated config/version files) is a reasonable follow-up if an external consumer actually needs one — it hasn't been built yet because nothing outside this repo currently needs it.
+
+## Writing a new interpreter (a new language kernel)
+
+`adrastea::Interpreter` ([`native/include/adrastea/interpreter.hpp`](../native/include/adrastea/interpreter.hpp)) is the extension point. A new language kernel:
+
+1. Subclasses `Interpreter` and implements its `*Impl()` virtual methods (`configureImpl`, `executeRequestImpl`, `completeRequestImpl`, `kernelInfoRequestImpl`, etc.) — see [`native/src/elara/r/interpreter_r.cpp`](../native/src/elara/r/interpreter_r.cpp)'s `RInterpreter` for a complete example.
+2. Calls `adrastea::registerInterpreter(this)` once constructed, before anything calls `adrastea::getInterpreter()`.
+3. Gets embedded into its own executable the way `elara.cpp` does — `main()` parses CLI args, builds an `adrastea::KernelConfiguration`, constructs the interpreter, and runs `adrastea::Kernel::start()` (see [`native/src/elara/bridge/engine.cpp`](../native/src/elara/bridge/engine.cpp)'s `Server::start()`).
+
+This is exactly the shape both Elara and Carpo have (`native/src/carpo/`, built by default via `JOVIAN_BUILD_CARPO`) -- two concrete, working examples proving the extension point genuinely generalizes, not just in theory. `PyInterpreter` (`native/src/carpo/interpreter_py.cpp`) implements the full `Interpreter` interface for real: `executeRequestImpl`/`isCompleteRequestImpl`/`completeRequestImpl`/`inspectRequestImpl` all embed and drive a real CPython interpreter, the same way `RInterpreter` drives R. Its own C API is loaded dynamically at runtime (`native/src/carpo/py/py_dynlib.hpp`, mirroring `native/src/elara/r/r_dynlib.hpp`) rather than linked at build time, for the same version-switching/clean-failure reasons Elara does it for R. A third language kernel would follow the exact same three steps above; nothing about the surrounding scaffold (registration, the executable, the CMake target, its own test suite) needs to change shape to add one.
+
+### Threading contract for interpreter authors
+
+The kernel executes code and reads its sockets on **one thread** (the process's main thread), and every `Interpreter` virtual is called from it — with **one exception**:
+
+- **`interruptRequestImpl()` is called from a different thread**, the *control watcher*, while `executeRequestImpl()` is still running (`KernelCore::executeRequest` brackets the interpreter call with `Server::beginExecution()` / `endExecution()`; between them a watcher thread services `interrupt_request` on the control channel — see [Architecture](architecture/overview.md#inside-a-kernel-process-elara--carpo)). It must therefore only do thread-safe things: flag the runtime to break out of what it is doing, and return `createInterruptReply()`. Do not touch the interpreter's state or call its API from it.
+- Only flag an interrupt **while an execution is actually running** (keep an `std::atomic<bool>` set by `executeRequestImpl`, as `RInterpreter` and `PyInterpreter` do). An interrupt with nothing to interrupt would otherwise linger and abort the *next* execution.
+- Make the runtime check that flag *and* wake blocking calls. R: set `R_interrupts_pending` (POSIX) / `UserBreak` (Windows); R polls it. Python: a real SIGINT to the interpreter thread (`raise(SIGINT)` on Windows, `pthread_kill` on POSIX) with `signal.default_int_handler` installed, because `PyErr_SetInterrupt()` alone does not wake `time.sleep()`.
+- Publishing from the watcher is safe (`ServerZmqImpl` serialises iopub publishing and control replies); everything else, including anything reached through `getInterpreter()`, is not.
+- All other control messages (`shutdown_request`, …) sent during an execution are queued and delivered on the main thread afterwards.
+
+Also part of the contract: `executeRequestImpl` must call its reply callback exactly once, honour `ExecuteRequestConfig::allow_stdin` for blocking reads (`adrastea::blockingInputRequest`, which throws when stdin is not allowed), evaluate `user_expressions` after a successful execution and return them in `createSuccessfulReply(payload, user_expressions)`, and report `restart` back in `shutdownRequestImpl`'s `createShutdownReply(restart)`.
+
+## Building and testing just the C++ side
+
+```sh
+cmake -S . -B dist/native -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake"
+cmake --build dist/native --config Release
+
+# Tests (a separate build tree, so JOVIAN_BUILD_TESTS=ON doesn't stick around
+# in dist/native's cache for future plain builds):
+cmake -S . -B dist/native-test -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake" -DJOVIAN_BUILD_TESTS=ON
+cmake --build dist/native-test --config Release
+ctest --test-dir dist/native-test -C Release --output-on-failure --timeout 180
+```
+
+See [Development](development.md) for what each CTest entry covers and the platform prerequisites ([README](../README.md#requirements)).
+
+Requires an R installation (only its headers, at build time — `elara` loads R's shared library dynamically at *runtime*, see [`native/src/elara/r/r_dynlib.hpp`](../native/src/elara/r/r_dynlib.hpp)) and [vcpkg](https://github.com/microsoft/vcpkg) with `VCPKG_ROOT` set. No Python installation is needed to *build* `carpo` at all -- unlike R, Carpo doesn't even include Python's headers at compile time (see `py_dynlib.hpp`'s file comment for why); a Python install is only needed at runtime, and only to actually run a Python session (`CarpoTest` also needs one, to embed and exercise for real -- it skips itself via `GTEST_SKIP` if `native/test/CMakeLists.txt`'s `find_package(Python3)` doesn't find one at configure time).

package/docs/development.md

@@ -0,0 +1,105 @@
+# Development
+
+## Repository layout
+
+```
+CMakeLists.txt, native/CMakeLists.txt, native/test/CMakeLists.txt   the build (see below)
+cmake/                  FindR.cmake, FindLibUUID.cmake
+vcpkg.json              native dependencies (manifest, pinned baseline)
+native/                 C++: adrastea (static lib), elara, carpo, themisto — see architecture/overview.md
+lib/                    the TypeScript package
+packages/hera/          the R companion package
+test/unit/lib/          TypeScript unit tests (Session against a fake WebSocket, ExecutionQueue, …)
+test/integration/       end-to-end tests: real themisto + real kernels
+tools/playground/       Next.js playground (own package.json)
+tools/jupyter-kernelspec/  kernel.json generator
+examples/               runnable examples
+scripts/                build.js, test.js, clean.js, dev.js, format.js, coverage.js
+docs/                   documentation
+dist/                   ALL build output (gitignored): dist/lib, dist/native, dist/native-test, dist/ide/*
+```
+
+## Build system
+
+**CMake ≥ 3.24, C++23**, dependencies from vcpkg (`vcpkg.json`; the toolchain file is `$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake`). The root `CMakeLists.txt` resolves `nlohmann_json`, `cppzmq`, `zeromq`, `OpenSSL`, `R` (**headers only** — `cmake/FindR.cmake` runs `R RHOME`), `httplib` and `ixwebsocket`, then adds `native/`.
+
+| Option | Default | Effect |
+|---|---|---|
+| `JOVIAN_BUILD_ELARA` | ON | Build the `elara` executable |
+| `JOVIAN_BUILD_THEMISTO` | ON | Build the `themisto` executable |
+| `JOVIAN_BUILD_CARPO` | ON | Build the `carpo` executable |
+| `JOVIAN_BUILD_TESTS` | OFF | Build the GoogleTest executables and register them with CTest |
+| `JOVIAN_SANITIZE_ADDRESS` | OFF | Address sanitizer |
+
+Outputs go to `dist/native/<config>/` (`CMAKE_RUNTIME_OUTPUT_DIRECTORY`) — executables, and on Windows the vcpkg DLLs next to them. Tests use a **separate build tree** (`dist/native-test`), so `JOVIAN_BUILD_TESTS=ON` does not stick in the main cache — but the output directory is set from the source root, so the test executables land in `dist/native/<config>/` too (which is why `package.json`'s `files` excludes `*_test.exe`).
+
+```sh
+# what `npm run build` does (native part):
+cmake -S . -B dist/native -DCMAKE_TOOLCHAIN_FILE="$VCPKG_ROOT/scripts/buildsystems/vcpkg.cmake"
+cmake --build dist/native --config Release
+# TypeScript:
+npx tsc --build                # lib/ -> dist/lib
+```
+
+Platform notes baked into the build: Windows links the MSVC dynamic runtime, gives `elara` a 128 MiB stack (`/STACK:134217728`, added after a real `STATUS_STACK_OVERFLOW`) and links `bcrypt` for Themisto; Linux links `libuuid` (`find_package(LibUUID)`) and `dl`; macOS links CoreFoundation.
+
+`CMakePresets.json` has Ninja + `cl.exe` presets for Windows IDE use (output under `dist/ide/<preset>`); `npm` scripts never use them.
+
+## Test layers
+
+| Layer | Command | What it needs | What it covers |
+|---|---|---|---|
+| **Native (GoogleTest / CTest)** | `npm test` (first stage), or by hand: configure `dist/native-test` with `-DJOVIAN_BUILD_TESTS=ON`, build, `ctest --test-dir dist/native-test -C Release --output-on-failure --timeout 180` | vcpkg deps; R + `hera` for `SessionRegistryTest`; Python for `CarpoTest` | 12 CTest entries, below |
+| **TypeScript unit** | `npm run test:unit` | built `dist/lib` | `Session` against a fake WebSocket, `ExecutionQueue`, router, middleware, option bodies. No processes. |
+| **Integration** | `npm run test:integration` | built native binaries + `dist/lib`, R + `hera`; Python + `carpo` for the Python tests | Real `SessionManager` → `themisto` → `elara`/`carpo`: execute, streaming, stdin (R and Python), history, complete/inspect/is_complete/kernel_info, user expressions, `stopOnError`, interrupt (R and Python), working directory, stderr, comms (client- and kernel-initiated), `shutdown_reply` on stop/restart. Skips itself if `themisto` is not built; the Python tests skip without Python or `carpo`. |
+
+CTest entries: `MessageTest`, `MiddlewareTest`, `AuthenticationTest`, `ZmqSerializerTest`, `KernelConfigurationTest`, `ClientZmqTest`, `ClientHeartbeatTest`, `ClientHandshakeZmqTest` (transport, no R); `ElaraTest` (spawns `elara` for its start-up failure paths); `KernelProcessTest` (spawn/liveness/kill against a dummy helper); `SessionRegistryTest` (drives a real `elara` through `SessionRegistry` — create/execute/restart/stop/interrupt/requests/comms/stop-on-error/user-expressions and concurrency races; ~20 kernel starts, ~25 s locally, 180 s CTest timeout); `CarpoTest` (embeds real Python: execution, streaming, complete/inspect, venv activation).
+
+Playground tests are separate: `npm --prefix tools/playground test`.
+
+The `hera` R package is installed into your R library, not run from the repo: after changing `packages/hera/R/*`, run `npm run hera:install` (CI installs it from source on every run). Features that live in `hera` — such as live streaming of one long-running expression — only work with the updated copy installed.
+
+`npm test` (`scripts/test.js`) runs the native stage (with OpenCppCoverage if installed on Windows, otherwise plain `ctest`), then the two Node stages (`--test-force-exit`) through a wrapper that force-kills `node --test` after **20 minutes** — a backstop against a hung run, far above the couple of minutes the integration suite takes, so `npm test` is a valid one-shot check. The stages can also be run on their own: `npm run test:unit` and `npm run test:integration`.
+
+### Gotchas
+
+- **The TypeScript tests import the compiled library from `dist/lib`.** After changing `lib/`, run `npm run build:lib` first, or you are testing stale code.
+- **Windows will not overwrite a running `.exe`.** A leftover `themisto.exe` / `elara.exe` / `carpo.exe` (a crashed test run, a playground left open) makes the next native build fail at link time (`LNK1104`) and can silently slow or hang later runs. Kernels are put in a job object that dies with Themisto, but a supervisor that outlives its test runner still keeps them. Check with `tasklist | findstr /i "themisto elara carpo"` and kill strays before building. To run something while rebuilding, point `JOVIAN_NATIVE_DIR` at a *copy* of `dist/native/Release`.
+- **Linux (Ubuntu/Debian).** The suites were run on Ubuntu 26.04 under WSL (native 12/12, unit, integration). What that setup needed: `cmake ninja-build uuid-dev r-base-dev python3-venv` from apt (`python3-venv` is for CarpoTest's venv test); the **official Node** from nodejs.org, because the distribution's Node has no TypeScript type stripping (`ERR_UNKNOWN_FILE_EXTENSION` on `.ts`); and `hera`'s dependencies from CRAN in a private `R_LIBS_USER` with `R_LIBS_SITE=/nonexistent`, since the apt `r-cran-*` packages fail to load (`undefined symbol: SETLENGTH`, built for a different R ABI). Export those two variables in the shell that runs `ctest` and the Node tests so the kernel processes inherit them. Build in the WSL filesystem (rsync the tree), not under `/mnt/c`. macOS is covered only by CI.
+- **A test that holds a mutex its own `onMessage` callback needs, while calling `stopSession()`, deadlocks:** the supervisor's poll thread now keeps relaying (the kernel's `shutdown_reply`) during the stop. Scope such locks.
+- **Every real-kernel test starts a process** (~0.7 s locally, more on cold CI runners) — keep the number of sessions per test small.
+- `execute()` rejections and `'error'` events: tests that create a `Session` must attach an `'error'` listener or the Node process dies (see [API](api/README.md#events)).
+
+## CI
+
+`.github/workflows/ci.yml` runs on every push to `main`, on pull requests, and manually, on **Windows, Ubuntu and macOS** (`fail-fast: false`):
+
+1. Checkout, Node (`lts/*`), R (`release`, without Rtools on Windows), Python (`3.x`).
+2. `r-lib/actions/setup-r-dependencies` with `packages: local::packages/hera` — installs `hera` and all its CRAN imports (real code execution needs it).
+3. `uuid-dev` on Linux.
+4. Clone and bootstrap vcpkg (`VCPKG_ROOT`), with the vcpkg binary cache keyed on `vcpkg.json`.
+5. Configure + build native (Release): elara, themisto, carpo.
+6. Configure + build the native tests (`dist/native-test`, `-DJOVIAN_BUILD_TESTS=ON`) and run `ctest -C Release --output-on-failure --timeout 180`.
+7. `npm ci --legacy-peer-deps` (a known peer-dependency conflict between TypeScript 7 and the `@typescript-eslint` plugin), `npx tsc --build`, unit tests, integration tests.
+
+## Releasing
+
+The npm packages are built and published by `.github/workflows/release.yml` from a version tag; see [releasing.md](releasing.md).
+
+## Debugging
+
+- **Kernel logs.** Everything a kernel prints (`[R Interpreter] …`, `[carpo] …`, and anything R/Python writes outside an execution) appears on the *supervisor's* stderr, which `lib/` forwards to your process's stderr, prefixed `[elara]` / `[carpo]`. That is the first place to look when a session fails to start.
+- **Library logs.** Pass `logger` to `createSession()` (or read the default console output): `trace` shows queueing, request ids and timeouts.
+- **Run a kernel by hand.** Generate a kernelspec (`npm run jupyter:kernelspec`) and start it from `jupyter console --kernel elara`, or run `elara -f <connection-file> --r-home …` yourself — no supervisor involved.
+- **Talk to the supervisor directly.** Start `themisto` (it prints `{"type":"supervisorReady","httpPort":…,"wsPort":…}`) and use `curl` against [its HTTP API](protocol.md#2-themistos-http-api) and any WebSocket client against `ws://127.0.0.1:<wsPort>/sessions/<id>/messages`.
+- **One native test.** `dist/native/Release/session_registry_test.exe --gtest_filter=*Interrupt*` (kernel log noise goes to the same stdout; filter with `grep -v "^\[elara\]"`).
+- **Crashes.** A kernel crash is reported with its decoded exit code (`0xc0000005` = access violation). The Windows Event Viewer's Application log usually has the faulting module.
+- **Hangs in native tests** are almost always a leaked lock or a socket used from two threads: ZMQ sockets are not thread-safe, which is why `DealerChannel`, the control socket and iopub publishing carry mutexes.
+
+## Style and tooling
+
+`npm run format` / `npm run format:check` (clang-format) and `npm run lint` (`eslint lib/**/*.ts`, `clang-tidy native/src/**/*.cpp`) are wired in `package.json`; no ESLint or clang-format configuration file is currently checked in, so they use defaults (or your own config). Code comments in this repo explain *why* (constraints, past bugs), not what.
+
+## Adding a kernel
+
+See [C++ usage](cpp-usage.md#writing-a-new-interpreter-a-new-language-kernel): implement `adrastea::Interpreter`, register it, write a `main()`, add a CMake target, and add its executable to Themisto's `kernelExePaths` map (`native/src/themisto/main.cpp`) under a new `kernelType`.

package/docs/getting-started.md

@@ -0,0 +1,127 @@
+# Getting started
+
+This walks from a clean machine to a running R session and a running Python session, **building Jovian from source**. If you only want to use it from Node.js, `npm install @damurka/jovian` ships prebuilt binaries and you can skip to [your first R session](#4-your-first-r-session) — see [Install](../README.md#install) for what R needs. The [README](../README.md#requirements) has the precise requirements per platform; this page is the sequence.
+
+## 1. Install the prerequisites
+
+| Platform | Install |
+|---|---|
+| Windows | **Visual Studio** with the *Desktop development with C++* workload (the repo is built with Visual Studio 2026 / MSVC v145), **Git**, **Node.js** (recent LTS), **R** (4.2 or newer), optionally **Python 3**. |
+| Linux | A C++23 compiler, `cmake`, Git, Node.js (the official build from nodejs.org — see the note below), R built with a shared library (`--enable-R-shlib`; distribution packages are), `uuid-dev`, optionally Python 3. On Ubuntu/Debian: `sudo apt install cmake ninja-build uuid-dev r-base-dev python3 python3-venv`. |
+| macOS | Xcode Command Line Tools, CMake, Git, Node.js, R, optionally Python 3. |
+
+On Ubuntu/Debian (verified on Ubuntu 26.04 under WSL): Debian's split R headers are found automatically (`cmake/FindR.cmake` asks `R CMD config --cppflags`), and Carpo finds a distribution Python's `libpython` in `lib/<arch>-linux-gnu/`, so `PYTHONHOME=/usr` works. The distribution's packaged Node.js has no TypeScript type stripping — install the official Node from nodejs.org to run the `.ts` tests. macOS is covered only by CI.
+
+Then install **vcpkg** and point `VCPKG_ROOT` at it:
+
+```sh
+git clone https://github.com/microsoft/vcpkg.git
+cd vcpkg
+./bootstrap-vcpkg.sh          # Windows: .\bootstrap-vcpkg.bat
+export VCPKG_ROOT=$PWD        # PowerShell: $env:VCPKG_ROOT = (Get-Location).Path
+```
+
+## 2. Install the R side
+
+Elara needs the R package `hera` (in `packages/hera`) and its dependencies in the R library the session will use:
+
+```r
+install.packages(c("cli", "evaluate", "glue", "IRdisplay", "jsonlite", "R6", "repr", "rlang"))
+```
+```sh
+npm run hera:install               # = R CMD INSTALL packages/hera, from the repo root; re-run it after pulling to pick up hera changes
+```
+
+On Debian/Ubuntu, if `hera` fails to install with `undefined symbol: SETLENGTH`, the apt `r-cran-*` packages were built for a different R ABI. Install the dependencies from CRAN into a private library instead: `export R_LIBS_SITE=/nonexistent R_LIBS_USER=$HOME/Rlib`, `mkdir -p $R_LIBS_USER`, then `install.packages(c("cli", "evaluate", "glue", "IRdisplay", "jsonlite", "R6", "repr", "rlang"), lib = Sys.getenv("R_LIBS_USER"))` and `npm run hera:install` in that same shell (keep both variables set when running sessions).
+
+(Alternatively pass `heraSrcPath` when creating a session and let Elara install it — see [Environments](guides/environments.md#the-hera-package-required).)
+
+## 3. Build
+
+```sh
+npm install --legacy-peer-deps
+npm run build          # native (elara, carpo, themisto) -> dist/native/Release, then TypeScript -> dist/lib
+```
+
+`npm run build` uses `VCPKG_ROOT`'s toolchain file; the first build compiles ZeroMQ, OpenSSL and the other vcpkg dependencies, which takes a while (vcpkg caches them afterwards). If it succeeds you have `dist/native/Release/{themisto,elara,carpo}[.exe]` and `dist/lib/`.
+
+## 4. Your first R session
+
+Save as `first-r.mjs` in the repo root and run `node first-r.mjs`:
+
+```javascript
+import { SessionManager } from './dist/lib/index.js';
+
+const manager = new SessionManager();
+const session = await manager.createSession({
+    kernelType: 'r',
+    rHome: process.env.R_HOME,                // e.g. "C:/Program Files/R/R-4.6.0" or the output of `R RHOME`
+    rPath: process.env.R_PATH,                // Windows only: e.g. "C:/Program Files/R/R-4.6.0/bin/x64"
+    workingDirectory: process.cwd(),          // where getwd() will point
+});
+
+session.on('error', () => {});                            // required: see docs/api/README.md#events
+session.on('stdout', (text) => process.stdout.write(text));
+
+const result = await session.execute('print("hello from R"); x <- 1:10; mean(x)');
+console.log('success:', result.success);
+console.log(result.output.map((m) => `${m.msgType}: ${JSON.stringify(m.content).slice(0, 80)}`));
+
+console.log(await session.kernelInfo());                  // language_info.name === 'R'
+console.log((await session.complete('pri')).matches);     // [ 'print', … ]
+
+await manager.stopAll();                                  // always: it also ends the supervisor
+```
+
+## 5. Your first Python session
+
+```javascript
+const py = await manager.createSession({
+    kernelType: 'python',
+    pythonHome: process.env.PYTHONHOME,       // the prefix that contains libpython — python -c "import sys; print(sys.base_prefix)"
+    workingDirectory: process.cwd(),
+});
+py.on('error', () => {});
+py.on('stdout', (t) => process.stdout.write(t));
+await py.execute('import os\nprint(os.getcwd())\nsum(range(1, 11))');
+```
+
+Create both in one script, side by side — they are separate processes.
+
+## 6. Things worth trying next
+
+```javascript
+// Interactive input
+session.on('input_request', ({ prompt }) => session.sendInputReply('World'));
+await session.execute('name <- readline("name? "); cat("hello", name, "\\n")', { allowStdin: true });
+
+// Interrupt a long call
+const running = session.execute('Sys.sleep(60)', { timeout: 0 });
+setTimeout(() => session.interrupt(), 500);
+console.log((await running).success);                    // false — interrupted
+
+// Evaluate an expression after the code
+const r = await session.execute('x <- 21', { userExpressions: { double: 'x * 2' } });
+console.log(r.userExpressions.double);                    // { status: 'ok', data: { 'text/plain': '[1] 42' }, … }
+```
+
+More: [interactive input](guides/interactive-input.md), [interrupting](guides/interrupting.md), [comms](guides/comms.md), [session lifecycle](guides/sessions-lifecycle.md), and the runnable [`examples/`](../examples).
+
+## 7. The playground
+
+A browser UI on top of the same API, handy for trying an installation:
+
+```sh
+npm run playground:install    # once
+npm run playground            # http://127.0.0.1:4173
+```
+
+It pre-fills R and Python from auto-discovery, shows PID / memory / working directory per session, and completes as you type (`Tab` accepts) and inspects a word when you rest the mouse or caret on it (`Shift+Tab` asks explicitly). See [the playground guide](guides/playground.md).
+
+## 8. Run the tests
+
+```sh
+npm test                 # native ctest + TypeScript unit + integration
+```
+
+If something fails to start, see [Troubleshooting](troubleshooting.md).

package/docs/guides/comms.md

@@ -0,0 +1,70 @@
+# Comms
+
+A **comm** is a named, bidirectional message stream between your application and a *target* registered inside the kernel — the mechanism Jupyter widgets are built on. Jovian exposes it as `Comm` objects. Comms are an **R feature today**: Carpo has no API for registering comm targets, so every `comm_open` sent to a Python session is answered with a `comm_close`.
+
+## Client → kernel: `openComm`
+
+First register a target in the kernel (R, via `hera`), then open a comm to it:
+
+```typescript
+await session.execute(`
+hera::CommManager$register_comm_target("echo", function(comm, message) {
+    comm$on_message(function(msg) {
+        comm$send(list(echo = msg$content$data$text))
+    })
+})
+`);
+
+const comm = await session.openComm('echo', { hello: 'kernel' });   // data becomes the comm_open payload
+comm.on('message', (data) => console.log(data));                    // { echo: 'ping' }
+await comm.send({ text: 'ping' });
+await comm.close();
+```
+
+The callback receives the new `comm` and the `comm_open` `message`; `comm$on_message()` registers a handler for the client's `comm_msg`s, `comm$send()` / `comm$open()` / `comm$close()` send to the client (data is serialised with `jsonlite`).
+
+If the kernel has no such target, it answers with a `comm_close` and the returned `Comm` emits `'close'`. (You can also check with `await session.commInfo('echo')`.)
+
+## Kernel → client: the `'comm'` event
+
+A comm the kernel creates and opens arrives as a `'comm'` event:
+
+```typescript
+session.on('comm', (comm, data) => {
+    console.log(comm.targetName, 'opened with', data);
+    comm.on('message', (msg) => console.log('kernel says', msg));
+    comm.send({ text: 'hi' });
+});
+
+await session.execute(`
+hera::CommManager$register_comm_target("kernel_side")   # a target must exist before new_comm()
+comm <- hera::CommManager$new_comm("kernel_side")
+comm$on_message(function(msg) comm$send(list(echo = msg$content$data$text)))
+comm$open(list(greeting = "from R"))
+`);
+```
+
+Attach the `'comm'` listener **before** running the code that opens it. `new_comm()` returns `NULL` for an unregistered target, so register the target first.
+
+## The `Comm` object
+
+| | |
+|---|---|
+| `comm.id`, `comm.targetName`, `comm.closed` | Identity and state. |
+| `comm.send(data)` | Sends a `comm_msg`; rejects if the comm is closed. |
+| `comm.close(data?)` | Sends `comm_close`; emits `'close'`. |
+| `'message'` `(data)` | The kernel sent a `comm_msg`. |
+| `'close'` `(data)` | Closed by either side; or the kernel restarted / exited / the session stopped / the connection dropped, in which case `data.reason` says which. |
+
+## Listing comms
+
+`await session.commInfo()` returns `{ comms: { <commId>: { target_name } } }` for every open comm in the kernel; pass a target name to filter.
+
+## Low-level API
+
+`commOpen(targetName, data?, commId?)`, `commMsg(commId, data?)` and `commClose(commId, data?)` send the raw messages and resolve with the message id they were sent under (useful for correlating iopub replies — e.g. the `comm_close` for an unknown target has that id as its `parentMsgId`). The kernel's `comm_open` / `comm_msg` / `comm_close` are also available as plain events of those names (`session.on('comm_msg', (content) => …)` with `content.comm_id`). `openComm()` and the `'comm'` event are built on them.
+
+## Notes
+
+- Comm requests are handled on the kernel's main thread between executions, so a comm message sent while code is running waits for it to finish.
+- Comms do not survive a restart; open `Comm`s emit `'close'` with `reason: 'kernel restarted'`.