esoul-sdk 0.24.0 → 0.25.2

package/docs/10-testing.md

@@ -39,6 +39,35 @@ it("the description never claims an emptiness it could not read", () => {
 A UI test renders once with a fixed state and checks the visible chrome and the empty state.
 Keep suites fast; they run beside the live preview on the same machine.
 
+## The testing kit — `esoul-sdk/testing`
+
+Worlds your tests run in, so nothing real is touched and every run is the same:
+
+| Helper | Gives you | Chapter |
+|---|---|---|
+| `memoryDb(manifest)` + `fakeViewer(kind, opts)` | your declared tables in memory, with your REAL row rules compiled; a viewer of any kind (`owner`, `member`, `visitor`, `anonymous`, `agent`, roles and attributes) | 14 |
+| `runOp(pluginServer, "op", { viewer, args, db, apps })` | calls one op with a real-shaped `ctx`; returns its result plus what it `emit`ted and `notify`d | 14 |
+| `fakeApps({ slot: { tools } })`, `capture()` | bound apps for `ctx.apps.<slot>` (calls recorded); a recorder for any callback | 16 |
+| `memoryFiles(tree)` | the files door (`filesForOp`) over an in-memory tree: sources, list, read, write, transfers — with `crashAt` / `failItem` to prove a retry leaves no residue | 09 |
+| `scriptedModel(policy)` | `llm(ctx)` answering from your script; `calls` records every request | 19 |
+| `simGmail(opts)` | a Gmail API at the HTTP level (`fetch`, `install()`), with accounts, `deliver(...)`, personas that reply (`addPersona`), a cold-campaign `crowd(n)`, a virtual clock (`advance(ms)`), `faults` (429/5xx/401/network), `snapshot()` / restore; `sent` and `outbox` say what left | 08 |
+| `fakeCredentials(slots)` | `credentials(ctx)` without accounts: per slot a `status`, the `hosts` it may reach and a `fetch` handler; the platform's refusals (undeclared slot, plain http, a host off the list, a slot not ready) and `calls` recording every request | 08 |
+| `startMockOAuth()` | a local OAuth 2 provider for a connection's sign-in and refresh | 08 |
+| `listFailedRequests()`, `subscribeFailedRequests`, `resetFailedRequests` | what `runOptimistic` reverted and named on the banner | 05 |
+
+```ts
+import { memoryDb, fakeViewer, runOp, scriptedModel } from "esoul-sdk/testing";
+import manifest from "./plugin.json";
+import { pluginServer } from "./server";
+
+it("a customer sees only their own orders", async () => {
+  const db = memoryDb(manifest);
+  await runOp(pluginServer, "place-order", { viewer: fakeViewer("visitor", { userId: "a" }), args: { sku: "x" }, db });
+  const b = await runOp(pluginServer, "list-orders", { viewer: fakeViewer("visitor", { userId: "b" }), db });
+  expect(b.result).toEqual({ orders: [] });
+});
+```
+
 ## Tools
 
 Call `execute` with a recording `eventCallback` and assert the events it emitted and the text it

package/docs/17-editing-and-merging.md

@@ -153,3 +153,25 @@ recorded as a conflict. `baseMerge` runs in every fold: keep it pure, exactly li
   platform and needs no merge — prefer operations to whole-value saves where you can.
 - Device view state (which page is open, scroll, selection) belongs in component state or
   `sessionStorage`, never in a shared event another device would follow.
+
+## 4. Declare `resolveConcurrent` for patch-shaped events
+
+`baseMerge` is for events that write one whole value. An event that writes a PATCH — a list of sub-items it
+changed, each with the version it built on — can do better: declare `merge` with `scope` (the sub-item ids),
+and `resolveConcurrent`. The platform calls it only when the patch would land as a conflict (a writer it never
+saw changed one of its sub-items), with the state it lands on; it merges what it can against the bases it
+carries and returns the rewritten event:
+
+```ts
+merge: { item: (_d, ctx) => `${ctx.applicationId}:cad:model`, scope: (d) => d.delta.ops.map(keyOf), describe: (d) => d.label },
+resolveConcurrent: (state, eventData, { theirsSess }) => {
+  const r = resolveEdit(state, eventData);          // pure: three-way per sub-item against the base each op carries
+  if (!r) return null;                              // cannot reason about it → the plain conflict (applied and marked)
+  return r.clashes.length === 0
+    ? { eventData: r.eventData }                    // everything merged: applied, no marker
+    : { eventData: r.eventData, detail: { kind: "cad-edit", clashes: r.clashes, theirsSess } };
+},
+```
+
+What it returns is what the fold applies, on every device and on the server. Keep it pure and symmetric; a
+property test over randomised edit pairs is the cheapest proof.

package/docs/18-your-computer.md

@@ -1,5 +1,9 @@
 # 18 · Parts of your app that run on the person's computer
 
+> Building a service that runs on the computer and talks to your app? Start with the
+> walk-through, [20 · A service on your computer](20-a-service-on-your-computer.md); this page is
+> the reference.
+
 Some apps need the person's own machine: a GPU for training, files that never
 leave a laptop, a tool installed there, a local network. The **device arm**
 lets an app run commands on a computer the owner connects, with the output
@@ -59,12 +63,17 @@ The panel lists connected computers (online, last seen, Disconnect) and a
 **Connect a computer** button. It shows one command to paste in a terminal:
 
 ```
-npx -y -p esoul-sdk@0.19.0 esoul-device connect https://externalsoul.com/api/device/m/offers/off_…
+npx -y -p esoul-sdk@0.19.1 esoul-device connect https://externalsoul.com/api/device/m/offers/off_…
 ```
 
 The runtime ships inside `esoul-sdk` (the `esoul-device` command, and
 `esoul-sdk/machine` for code that runs on the computer). It uses only Node
-built-ins, so a computer needs nothing but Node 22.
+built-ins, so a computer needs nothing but Node 22. The version in the command
+is the runtime's own, pinned by the platform (it moves more slowly than the
+package). A computer without Node uses the panel's other line, which brings a
+pinned Node first, without sudo:
+`curl -fsSL https://externalsoul.com/api/device/install | sh -s -- <link>`.
+The link expires in 10 minutes.
 
 The command holds no secret. The computer makes its own key, prints a code
 (`K7Q-4XM`) and the list of commands; the page shows the same code with
@@ -134,8 +143,10 @@ const { run, finished } = useDeviceRun({ workspaceId, nodeId, commandId });
 // run.output streams in as the command prints (first 1 MB kept, then run.tail)
 ```
 
-`useDeviceActions({ nodeId })` gives the owner's buttons from the UI (run,
-cancel, disconnect, approve an update) without writing an op.
+`useDeviceActions({ nodeId })` gives the owner's buttons from the UI (`run`,
+`cancel`, `disconnect`, `approveUpdate`, `setWorkspace` for the reach, `sh`,
+and `recent` runs) without writing an op. `useMyComputers()` and
+`<YourComputers/>` list the person's computers across apps.
 
 ## 6. Local settings stay local
 
@@ -144,7 +155,7 @@ manifest declares under `device.config` are set on the computer and never sent
 to the platform:
 
 ```
-npx -y -p esoul-sdk@0.19.0 esoul-device config set "My app" HF_TOKEN
+npx -y -p esoul-sdk@0.19.1 esoul-device config set "My app" HF_TOKEN
 ```
 
 The command receives them as environment variables. `redact` patterns are
@@ -163,9 +174,12 @@ applied on the computer before any output leaves it.
 
 ## 8. Testing
 
-In a Forge preview, `devices(ctx)` answers with a clear refusal and
-`<ConnectComputer/>` says computers connect to the installed app. Mock
-`esoul-sdk/server` in unit tests, as for any host function.
+Computers connect to INSTALLED apps. In a Forge preview `devices(ctx).list()`
+answers `[]`, every other `devices()` call throws "computers connect to
+INSTALLED apps", `useDevices`/`useDeviceRun` stay empty, `useDeviceActions`
+throws, and `<ConnectComputer/>` says so. Mock `esoul-sdk/server` in unit tests,
+as for any host function, and test a program with a fake `p`
+([20 §5](20-a-service-on-your-computer.md)).
 
 ## 9. The workspace: agents working on the computer
 
@@ -181,9 +195,9 @@ dev server open — the owner gives the app a **workspace**:
 Two parties agree. **The computer** sets a ceiling when it connects:
 
 ```
-npx -y -p esoul-sdk@0.19.0 esoul-device connect <link>                    # this folder (the default)
-npx -y -p esoul-sdk@0.19.0 esoul-device connect <link> --folder ~/proj    # another folder
-npx -y -p esoul-sdk@0.19.0 esoul-device connect <link> --scope computer   # allow the whole computer
+npx -y -p esoul-sdk@0.19.1 esoul-device connect <link>                    # this folder (the default)
+npx -y -p esoul-sdk@0.19.1 esoul-device connect <link> --folder ~/proj    # another folder
+npx -y -p esoul-sdk@0.19.1 esoul-device connect <link> --scope computer   # allow the whole computer
 ```
 
 **The owner** grants each app at most that ceiling from the app's page (the
@@ -320,8 +334,16 @@ The rules that keep it safe and quiet:
 
 Where each program stands on each computer — installing, each setup step,
 self-test, running, the program's own `p.status` words, why it failed —
-comes back on `devices(ctx).list()` as `programState`, and `<ConnectComputer/>`
-shows it. The reference app is **CPU monitor** (`src/plugins/cpu-monitor`).
+comes back on `devices(ctx).list()` as `programState` (one computer:
+`devices(ctx).program(linkId).state()`), and `<ConnectComputer/>` shows it.
+
+Limits: a message (`send`) and its answer are each at most 1 MB of JSON; a
+topic matches `[a-zA-Z][a-zA-Z0-9_.:-]{0,63}`; a message for an offline
+computer expires (600 s default). `p.op` calls are resent when no answer has
+come in about 4 s (up to 3 tries) — make those ops idempotent and pass
+`{ key }`. `p.workspaceRoot` is null when the whole computer is granted.
+Nothing reaches INTO the computer: there is no tunnel or open port; a program
+reaches its own localhost and calls out. The reference app is **CPU monitor** (`src/plugins/cpu-monitor`).
 
 ### An agent on each computer
 

package/docs/20-a-service-on-your-computer.md

@@ -0,0 +1,291 @@
+# 20 · A service on your computer, connected to your app
+
+A guide, start to finish: an app built in the Forge that runs its own
+long-lived service on the person's computer — a local model server, a
+sampler, a bridge to a device on the desk, a trainer's supervisor — and
+talks to it both ways. The reference for every piece is
+[18 · Parts of your app that run on the person's computer](18-your-computer.md);
+this page is the path through it.
+
+## The shape
+
+```
+ your app on ExternalSoul                      the person's computer
+ ─────────────────────────                     ─────────────────────────────
+ server.ts  ── devices(ctx).program(linkId)    device/main.ts  (your PROGRAM,
+               .send("ask", data)  ─────────▶    p.on("ask", …)   kept running,
+                                    ◀───────── answer (≤ 1 MB)    sandboxed)
+ ops.ts     ◀── p.op("report", data) ────────   p.every / events      │
+               (the op sees viewer.device)                             ▼
+                                                     your service on localhost
+                                                     (started by the program,
+                                                      installed by setup steps)
+```
+
+- **The program is the service's keeper.** Your app ships a `device/` folder;
+  the person approves it once on their computer; the runtime installs it,
+  runs your setup steps (a Python venv, `npm ci`, a model download), checks it
+  with your self-test, and keeps it running (`"resident": true`), restarting
+  it when it exits.
+- **App → service:** your server code sends a message to the program; the
+  program's handler does the work — usually a `fetch("http://localhost:…")`
+  to the service it started — and returns the answer.
+- **Service → app:** the program calls one of your app's own ops with
+  `p.op(...)`. The op knows it came from that computer (`ctx.viewer.device`).
+- **Nothing reaches INTO the computer.** There is no tunnel and no open port:
+  the computer only calls out. The program can reach its own `localhost` and
+  the internet; your app reaches the program only through `send`.
+
+## 1. Declare the program (`plugin.json`)
+
+```json
+{
+  "ops": ["ask", "report", "computers"],
+  "device": {
+    "program": {
+      "main": "device/main.ts",
+      "resident": true,
+      "setup": [
+        { "id": "venv", "describe": "Python environment", "run": ["python3", "-m", "venv", ".venv"] },
+        {
+          "id": "deps", "describe": "Install the server",
+          "run": [".venv/bin/pip", "install", "-r", "{program}/device/requirements.txt"],
+          "inputs": ["device/requirements.txt"]
+        }
+      ],
+      "selfTest": { "run": [".venv/bin/python", "{program}/device/self_test.py"] }
+    },
+    "config": { "MODEL_DIR": { "describe": "Where the model weights are on this computer" } },
+    "onAppDeleted": "kill"
+  }
+}
+```
+
+- `setup` runs once per step and per change of its `inputs` — an app update
+  that leaves `requirements.txt` alone does not reinstall. Steps are argv, never
+  a shell line; `{program}` is the program's folder. Relative paths (`.venv`)
+  live in the program's data folder, which is kept across versions.
+- `selfTest` must exit 0 before the program counts as ready; make it the
+  thing your service needs (`claude --version`, a GPU check, an import).
+- `config` names values that stay on the computer (`esoul-device config set
+  "<app name>" MODEL_DIR ~/models`); the program reads them as `p.config`.
+
+## 2. Write the program (`device/main.ts`)
+
+```ts
+import type { Program } from "esoul-sdk/machine";      // types only — nothing is installed beside the program
+import { spawn, type ChildProcess } from "node:child_process";
+import { join } from "node:path";
+
+const PORT = 8711;
+let server: ChildProcess | null = null;
+
+async function healthy(): Promise<boolean> {
+  try { return (await fetch(`http://127.0.0.1:${PORT}/health`)).ok; } catch { return false; }
+}
+
+export default {
+  async start(p) {
+    // Start the service: argv, no shell; its output goes to the program's log.
+    server = spawn(join(p.dataDir, ".venv/bin/python"), [join(p.programDir, "device/serve.py"), "--port", String(PORT)], {
+      cwd: p.dataDir,
+      env: { ...process.env, MODEL_DIR: p.config.MODEL_DIR ?? "" },
+      stdio: ["ignore", "inherit", "inherit"],
+    });
+    for (let i = 0; i < 60 && !(await healthy()); i++) await new Promise((r) => setTimeout(r, 1000));
+    p.status((await healthy()) ? "Model server ready" : "Model server did not start", { ready: await healthy() });
+
+    // App → service: the answer of the handler is the answer of send().
+    p.on("ask", async (data) => {
+      const r = await fetch(`http://127.0.0.1:${PORT}/ask`, { method: "POST", body: JSON.stringify(data), headers: { "content-type": "application/json" } });
+      return await r.json();
+    });
+
+    // Long work: answer at once, report when done (the app's op stores it).
+    p.on("job", async (data) => {
+      const { requestId } = data as { requestId: string };
+      void (async () => {
+        const r = await fetch(`http://127.0.0.1:${PORT}/job`, { method: "POST", body: JSON.stringify(data) });
+        await p.op("report", { requestId, result: await r.json() }, { key: requestId });
+      })();
+      return { accepted: true };
+    });
+
+    // Service → app on a clock.
+    p.every(60_000, async () => p.op("report", { kind: "heartbeat", ok: await healthy() }));
+  },
+  async stop() { server?.kill(); },
+} satisfies Program;
+```
+
+What a program may import: Node built-ins (`node:fs`, `node:child_process`, …),
+files of its own `device/` folder, and `esoul-sdk/machine` **types only**. No
+packages — install what the service needs in a setup step. At most 2 MB and
+200 files; download weights and datasets in setup or at start. Node 22.6+
+runs a `.ts` entry directly (no enums, no namespaces).
+
+## 3. Talk to it from the app (`server.ts`)
+
+```ts
+import { devices, type PluginOpContext } from "esoul-sdk/server";
+
+// App → service. linkId names one connected computer (devices(ctx).list()).
+async function ask(ctx: PluginOpContext, args: { linkId: string; prompt: string }) {
+  const r = await devices(ctx).program(args.linkId).send("ask", { prompt: args.prompt }, { waitSeconds: 30 });
+  if (!r.ok) return { ok: false, error: r.error ?? "the computer did not answer" };   // offline, not approved, handler threw
+  return { ok: true, answer: r.result };
+}
+
+// Service → app. Only the program on a computer may report.
+async function report(ctx: PluginOpContext, args: { requestId?: string; result?: unknown; kind?: string }) {
+  const device = ctx.viewer.device;
+  if (!device) throw new Error("report comes from the program on a computer");
+  await ctx.emit("result_reported", { machineId: device.machineId, hostname: device.hostname, ...args });
+  return { ok: true };
+}
+
+// Which computers there are, and where each program stands.
+async function computers(ctx: PluginOpContext) {
+  return (await devices(ctx).list()).map((d) => ({
+    linkId: d.linkId, hostname: d.machine?.hostname, online: d.machine?.online ?? false,
+    program: d.programState?.phase ?? (d.program ? "approved" : "not approved"),
+    status: d.programState?.status?.text ?? null,
+  }));
+}
+```
+
+Rules that matter:
+
+- **`send` waits at most 55 s** (`waitSeconds`; `0` returns at once with the
+  `commandId`). For anything longer, use the job pattern above: answer
+  `accepted`, finish in the background, `p.op` the result.
+- **A message and an answer are each at most 1 MB of JSON.** Topics match
+  `[a-zA-Z][a-zA-Z0-9_.:-]{0,63}`. For files, hand the program a link
+  (`(await filesForOp(ctx)).readGrant(...)` + `fileGrantUrl`) and let it
+  download; for results, store them through ops or save files.
+- **A queued message for an offline computer expires** (600 s by default) —
+  a click on Monday never runs on Thursday.
+- **`p.op` must be quick and idempotent.** The computer resends a call whose
+  answer has not arrived in about 4 s (up to 3 tries). Pass `{ key }` with a
+  stable id and make the op a no-op the second time (store by `requestId`).
+- **Who can do what:** `send`, `run`, `sh` and `cancel` need a caller who may
+  change the workspace (`ctx.viewer.canWrite`). The program's `p.op` calls run
+  as the owner who approved that computer (`viewer.kind === "agent"`,
+  `viewer.device` set) through your app's own op and its own rules.
+
+`devices(ctx).program(linkId).state()` gives that program's state on that
+computer (installing, a setup step, self-test, running, its last `p.status`
+words, why it failed) — the same that `list()` returns as `programState`.
+
+## 4. The page: connect a computer
+
+```tsx
+import { ConnectComputer, useDevices } from "esoul-sdk/react";
+
+export function Computers({ workspaceId, nodeId }: { workspaceId: string; nodeId: string }) {
+  const { devices } = useDevices({ workspaceId, nodeId });   // pick a linkId for ask()
+  return <ConnectComputer workspaceId={workspaceId} nodeId={nodeId} title="Computers" />;
+}
+```
+
+`<ConnectComputer/>` lists the computers (online, last seen, the program's
+progress and words, Disconnect) and a **Connect a computer** button.
+
+## 5. Build it in the Forge — what you can prove before installing
+
+The Forge box runs your app in a preview. **Computers connect to installed
+apps only**: in the box `devices(ctx).list()` answers `[]`, every other
+`devices()` call throws "computers connect to INSTALLED apps", and
+`<ConnectComputer/>` shows a note saying so. Plan the work in that order:
+
+1. **Prove the program without a computer.** It is a plain object with
+   `start(p)`; give it a fake `p` in a test and call its handlers:
+
+   ```ts
+   import program from "../device/main";
+
+   test("ask goes to the local service and comes back", async () => {
+     const handlers: Record<string, (d: unknown) => unknown> = {};
+     const ops: Array<[string, unknown]> = [];
+     const p = {
+       computer: { hostname: "test", platform: "linux", arch: "x64", cpus: 4, memoryGb: 8 },
+       appName: "Model", programDir: __dirname, dataDir: "/tmp/model-test", workspaceRoot: null, config: {},
+       status: () => {}, log: () => {}, every: () => () => {},
+       on: (t: string, h: (d: unknown) => unknown) => { handlers[t] = h; },
+       op: async (name: string, args: unknown) => { ops.push([name, args]); return { ok: true }; },
+     };
+     // stub the service: global.fetch = jest.fn(...) answering /health and /ask
+     await program.start(p as never);
+     expect(await handlers.ask({ prompt: "hi" })).toEqual({ text: "hello" });
+   });
+   ```
+
+2. **Prove the app side** with `esoul-sdk/server` mocked (`devices` returning a
+   fake whose `program().send` answers), and call `report` with a faked
+   `viewer.device` to see the result land in the fold.
+3. **Run the service itself** in the box if it can run there (`run_in_app`:
+   `python3 device/serve.py --port 8711 &` then `curl localhost:8711/health`) —
+   the same code the program will start.
+4. `check_app` packs `device/` and refuses a program over 2 MB / 200 files or
+   one that imports a package.
+
+What only the installed app can prove: the program installed and running on
+a real computer, `send`, `p.op`, the approval card and the reach choice.
+
+## 6. Install, connect, approve
+
+1. `install_app` (the board's tool, or Install on the board). The app is now
+   on the platform.
+2. Open the app, press **Connect a computer**. The panel shows one command to
+   paste in a terminal on that computer:
+
+   ```
+   npx -y -p esoul-sdk@0.19.1 esoul-device connect https://externalsoul.com/api/device/m/offers/off_…
+   ```
+
+   No Node there? The panel's other line installs a pinned Node first, without
+   sudo: `curl -fsSL https://externalsoul.com/api/device/install | sh -s -- <link>`.
+   The link expires in 10 minutes and holds no secret.
+3. The computer prints a code (`K7Q-4XM`); the page shows the same code with
+   **Approve**, the program's size and version, its setup steps and
+   self-test, and **what the app may reach there**: commands only, its
+   folder, or the whole computer (never more than the computer allowed with
+   `--scope` / `--folder` when it connected).
+4. The runtime installs itself as a user service (systemd on Linux, a
+   LaunchAgent on macOS, a plain background service where neither exists),
+   so it comes back after a reboot. Linux and macOS; Windows via WSL2.
+5. The card shows setup progress, then the self-test, then your `p.status`
+   words. Your app's `computers` op (or `list()`) says `running`.
+
+**Updates.** A new version that changes the program (any file, the setup, the
+self-test) has a new digest and waits for **Approve** on the card; until then
+the computer keeps running the version it has. Changing what the app may
+reach there restarts the program under the new reach.
+
+**Where it runs.** Unconfined only with the whole computer granted (needed
+when the service uses the person's own logins, like Claude Code or git).
+Otherwise in an OS sandbox (bubblewrap on Linux, Seatbelt on macOS): it reads
+the system, writes only `p.dataDir` and the granted folder, and its home and
+`/tmp` are private. No sandbox on the computer → refused, never run
+unconfined. The network is not confined either way.
+
+## 7. Other ways in, and when to use them
+
+| You need | Use |
+|---|---|
+| A long-lived service your app owns, two-way | **A program** (this page) |
+| A known tool run with checked parameters (`nvidia-smi`, a training script) | **Declared commands** — `device.commands` + `devices(ctx).run` ([18 §1–4](18-your-computer.md)) |
+| Agents that work IN a project folder (read, edit, patch, run tests, keep a dev server in a terminal) | **The workspace** — `devices(ctx).sh("term new dev npm run dev", { agent })` ([18 §9](18-your-computer.md)) |
+| A process that is not your program pushes data to your app (a script, a device, a CI job) | **A token route** — declare the route `"access": "token"`, mint with `mintRouteToken(ctx, { route, ttlSeconds })`, the process POSTs with `Authorization: Bearer <token>` ([06 · Routes](06-server.md)). Works in the Forge box too (the box's public preview URL). |
+| The person already runs the My Computer app and the work is one-off shell commands | **`computer(ctx, machineNodeId)`** — shell strings the owner approves, output ≤ 8,000 characters ([06](06-server.md)); works in the box through the board's tab |
+
+## 8. When it does not work
+
+| You see | It means |
+|---|---|
+| `send` → `ok: false`, `expired` | the computer was offline for the message's whole life (600 s) |
+| `send` → error "not approved" / no program in `list()` | the person has not approved the program (or its new version) on the card |
+| `programState.phase === "failed"` | a setup step or the self-test failed, or the program exited five times in five minutes — the card shows the last log lines |
+| "refused: no sandbox" on the card | the computer has neither bubblewrap nor Seatbelt; grant the whole computer or install bubblewrap |
+| the same report stored twice | the op is not idempotent — key it by `requestId` (`p.op(..., { key })`) |
+| `send` answers `the answer was larger than 1 MB` | return a summary; put the bulk in a file and send a link or an id |