@lampdotdev/lamp 0.36.2-nightly.41 → 0.36.2-nightly.44

package/README.md

@@ -19,7 +19,7 @@ Mac app, `projectOf`), or any folder with an `index.html`; without one, the fold
 | `lamp init [--update]` | Puts Lamp's game runtime beside a page that's already there, changing none of the page's own files. `--update` writes again the Lamp files the folder has (`lamp.js` too), adding the game's only to a page that's a game already. |
 | `lamp unpack <file.lampapp> [folder]` | Unpacks a Lamp app file into a new folder beside it, named for it. A file carries none of Lamp's own files: a page gets them beside it as it's unpacked (the runtime for a game, lamp.js, the libraries its import map names), and a Mac app gets Package.swift, build.sh and LampSupport.swift from Lamp's starting files as it's first built (`writeMacStarter`), where it has none. One with a widget, a share extension or a 3D scene is left to Lamp's app, which writes the files they're built with. |
 | `lamp export [--to file] [--anyway]` | Makes the app's Lamp app file as Lamp's Share… does (`Sharing.export`, followed by `cli/lampapp.mjs`): its own files as its kind travels, Assets/, APP.md, what it declares, its credits with the app added to the chain, the licenses they name, its icon's design and lamp-app.json, never Lamp's own files, nor what the folder's `.lampignore` names. Refused when a license its credits name is gone or something in it is shaped like a key; `--anyway` sends work its credits give no license for, and lamp keeps that the person agreed (in its state), so it asks again only for work new to the app. A source credited `"live": true` (fetched while the app runs, nothing of it in the app) is never asked about. The shapes of keys, the names a license goes by and the sizes another Lamp opens are in `Resources/share-rules.json`, which Lamp's app reads too (`ShareRules`). |
-| `lamp dev [--open] [--lan] [--port N]` | Runs the page and says how each change went. Each save goes into every open copy of it, with the card Lamp's app puts on an app taking a change (`page.js`, left out of lamp's pictures; `lamp changing "<words>"` says on it what's changing). `--open` opens a window for it; `--lan` lets other devices on the network open it. |
+| `lamp dev [--open] [--lan] [--port N]` | Runs the page and says how each change went. Each save of a file a page has asked for goes into every open copy of it (a file no page has loaded, such as renders or logs a page's tests write, opens no page again), with the card Lamp's app puts on an app taking a change (`page.js`, left out of lamp's pictures; `lamp changing "<words>"` says on it what's changing). `--open` opens a window for it; `--lan` lets other devices on the network open it. |
 | `lamp open` | Shows the page in a window, running it in the background if it isn't running, and returns at once. |
 | `lamp open --in-lamp` | Gives a Mac app to Lamp's app (`cli/outside.mjs`), over the socket the person's own coding agent uses (`OutsideServer`, `OutsideWork`): `open_project` takes the folder on, with Lamp's review the first time, and from then on `look`, `reload` and `dev` call `apply_change`, so a change goes into the open app as Lamp's app puts it there (`LiveReload`) and its outcome is said: live, held, reopened, not_built, crashed. `use` calls `use_app`, in plain sight in the app Lamp's app has open, and `stop` calls `close_app`. A folder Lamp's app no longer has (its refusal says `taken_on: false`) is lamp's own again, said once. `lamp open --here` has lamp run its own copy again. `LAMP_APP` names another Lamp by its bundle id (a Lamp Dev instance). A Mac app `lamp new --mac` makes has the `lamp-app.json` Lamp's app takes a folder on by. |
 | `lamp look [--reopen] [--wait s] [--json]` | How the changes since the last look went, a picture of the page, its screen as text, new errors and the log. `--reopen` opens the page again from its beginning first. |

package/Resources/skills/web-app/SKILL.md

@@ -23,7 +23,10 @@ opening comment for the whole list. What it gives:
   heard of Lamp finds out how. Where it answers `connect: true` (a page on lamp.dev that hasn't reached this computer
   yet), also show a small "Connect AI" button that calls `LampAI.connect(capability)` with the same kind of AI you asked
   `status` about (`LampAI.connect('pictures')` for pictures) and use the answer it gives as `status`'s;
-  never call `connect()` as the page loads, since it makes the browser ask the person.
+  never call `connect()` as the page loads, since it makes the browser ask the person. For JSON the app reads, give
+  `ask` a `schema` (a JSON Schema: every property in `required`, `additionalProperties: false`), which holds the AI to
+  it, rather than only describing the shape. A long `ask` the person can stop takes a `signal` (an `AbortController`'s),
+  and its failure then has `stopped`, which needs no error shown.
 - **LampAI.act**: the person asks the app to do something in their own words, and the app's own reads and writes do
   it; "Doing things for the person" below, with **LampVoice** and **LampAI.hear** for saying it.
 - **LampStore**: what the app saves, as JSON on the device. Once something is saved, only ever add to it: a new field
@@ -102,6 +105,8 @@ whichever AI the person set up for apps' text, so don't tie it to voice: typing
   and take only the first answer, a button, what was heard or what was typed. With none, it's answered by voice or
   typing; never guess choices from the question's wording.
 - Keep what each write changed for Undo, and undo only what's still as the write left it; say which ones it left.
+- Where the app draws something the AI builds (a scene, a layout, a chart), give it a read that returns
+  `LampAI.picture(canvas)` in its result, so it can look at its work and fix it in the same turn.
 - Show each step while it runs (`onStep`), and cancel the turn (`turn.cancel()`) when the person closes it or the page
   is hidden.
 - `about` says what the app is and its rules for asking back: which person, which day and time, which of two things

package/cli/ai.mjs

@@ -9,8 +9,8 @@ import { accessSync, closeSync, constants, fstatSync, mkdtempSync, openSync, rea
 import { tmpdir } from "node:os";
 import { delimiter, join } from "node:path";
 import { randomBytes } from "node:crypto";
-import { chatBody, decidePrompt, planPrompt, planSchema, readDecision, readMessage, readPlan, readStep } from "./app-ai.mjs";
-import { DEFAULT_MODEL, IMAGE_MODELS, OpenRouterTrouble, chat as openRouterChat, image as openRouterImage, text as openRouterText } from "./openrouter.mjs";
+import { afterStream, chatBody, decidePrompt, planPrompt, planSchema, readDecision, readMessage, readPlan, readStep, stepReader } from "./app-ai.mjs";
+import { DEFAULT_MODEL, IMAGE_MODELS, OpenRouterTrouble, chat as openRouterChat, chatStream as openRouterChatStream, image as openRouterImage, text as openRouterText } from "./openrouter.mjs";
 
 
 /**
@@ -56,10 +56,12 @@ export class EngineTrouble extends Error {
  * computer is read or run whatever the prompt asks. Pictures (`images`, base64 with their `media_type`) go with the
  * request itself: to Claude Code in its input, to Codex as attachments, never through a tool that reads files.
  * Lamp's app runs it this way too (`engineAnswer` in calls.mjs, `AppAI.viaEngine`), naming the program (`tool`) and
- * what it needs set (`env`); on its own lamp finds the person's own on the PATH.
+ * what it needs set (`env`); on its own lamp finds the person's own on the PATH. `spent` hears what a call through
+ * OpenRouter cost; one through the person's own sign-in costs nothing through Lamp. `onPartial` hears the answer to a
+ * `schema` as it's written, the JSON so far, where the engine says it as it goes (Claude Code; Codex says it at the end).
  */
-export function complete({ system, prompt, schema, seconds = 60, which = engine(), tool, env = {}, model = process.env.LAMP_AI_MODEL || "", images = [], signal } = {}) {
-  if (which === "openrouter") return viaOpenRouter({ system, prompt, schema, seconds, model, images, signal });
+export function complete({ system, prompt, schema, seconds = 60, which = engine(), tool, env = {}, model = process.env.LAMP_AI_MODEL || "", images = [], signal, spent, onPartial } = {}) {
+  if (which === "openrouter") return viaOpenRouter({ system, prompt, schema, seconds, model, images, signal, spent });
   if (which !== "claude" && which !== "codex") return Promise.reject(new EngineTrouble("missing", missing));
   const folder = mkdtempSync(join(tmpdir(), "lamp-ai-"));
   const sayer = system || "You are a helpful assistant inside an app.";
@@ -72,6 +74,7 @@ export function complete({ system, prompt, schema, seconds = 60, which = engine(
       "--strict-mcp-config", "--disable-slash-commands", "--tools", "", "--system-prompt", sayer];
     if (model) args.push("--model", model);
     if (schema) args.push("--json-schema", JSON.stringify(schema));
+    if (schema && onPartial) args.push("--include-partial-messages");
     if (pictures.length) {
       args.push("--input-format", "stream-json");
       const content = [{ type: "text", text: prompt }, ...pictures.map((image) => ({ type: "image",
@@ -104,7 +107,7 @@ export function complete({ system, prompt, schema, seconds = 60, which = engine(
   delete childEnv.BUN_BE_BUN;
   return new Promise((resolve, reject) => {
     const child = spawn(tool || onPath(which) || which, args, { cwd: folder, env: childEnv, detached: true, stdio: [input ? "pipe" : "ignore", "pipe", "pipe"] });
-    let text = "", trouble = null, rest = "", said = "";
+    let text = "", trouble = null, rest = "", said = "", partial = "";
     // The whole group, since an engine may leave a child holding its output.
     const stop = (kind, words) => { if (!trouble) trouble = new EngineTrouble(kind, words); try { process.kill(-child.pid, "SIGKILL"); } catch { child.kill("SIGKILL"); } };
     const timer = setTimeout(() => stop("timeout", `${named(which)} took too long to answer.`), Math.max(1, seconds) * 1000);
@@ -118,6 +121,16 @@ export function complete({ system, prompt, schema, seconds = 60, which = engine(
         else if (codexFailure(event)) trouble ??= new EngineTrouble("failed", codexFailure(event));
         return;
       }
+      // The schema's answer as it's written: its JSON, a piece at a time.
+      if (event.type === "stream_event" && onPartial) {
+        const inner = event.event ?? {};
+        if (inner.type === "content_block_start") partial = "";
+        else if (inner.type === "content_block_delta" && inner.delta?.type === "input_json_delta" && typeof inner.delta.partial_json === "string") {
+          partial += inner.delta.partial_json;
+          try { onPartial(partial); } catch {}
+        }
+        return;
+      }
       if (event.type !== "result") return;
       if (event.is_error) trouble ??= new EngineTrouble("failed", String(event.result || "Claude Code couldn't answer."));
       else if (schema && event.structured_output !== undefined) text = JSON.stringify(event.structured_output);
@@ -156,31 +169,68 @@ function codexFailure(event) {
   return nested || words;
 }
 
+// MARK: The OpenRouter key
+
+const KEY_REFUSED = "OpenRouter didn't accept the key in OPENROUTER_API_KEY: it may have been revoked. Set a new one from openrouter.ai/keys.";
+const KEY_EMPTY = "The OpenRouter key in OPENROUTER_API_KEY has no credit left. Add credit at openrouter.ai.";
+let keyAsked = { key: "", at: 0, problem: null };
+
+/**
+ * What's wrong with the key in OPENROUTER_API_KEY, as OpenRouter says (as Lamp's app asks it, `openRouterVerdict`), or
+ * null: a key that's there isn't one that works (revoked, or out of credit), and `status` says so rather than offer
+ * what every call would then refuse. Asked at most every five minutes, and quickly, since a page waits on its status; when
+ * OpenRouter can't be reached in time, the key may be good.
+ */
+export async function keyProblem(fetcher = fetch) {
+  const key = process.env.OPENROUTER_API_KEY;
+  if (!key) return null;
+  if (keyAsked.key === key && Date.now() - keyAsked.at < 5 * 60_000) return keyAsked.problem;
+  let problem = null;
+  try {
+    const answer = await fetcher("https://openrouter.ai/api/v1/key", { headers: { Authorization: `Bearer ${key}` }, signal: AbortSignal.timeout(4000) });
+    if (answer.status === 401 || answer.status === 403) problem = KEY_REFUSED;
+    else if (answer.status === 200) {
+      const left = (await answer.json().catch(() => ({})))?.data?.limit_remaining;
+      if (typeof left === "number" && left <= 0) problem = KEY_EMPTY;
+    } else return null;
+  } catch { return null; }
+  keyAsked = { key, at: Date.now(), problem };
+  return problem;
+}
+
+/** What an OpenRouter failure says to the page: the key's trouble as the setting to fix, else what OpenRouter said. */
+function openRouterWords(error) {
+  if (error?.status === 401 || error?.status === 403) return KEY_REFUSED;
+  if (error?.status === 402) return KEY_EMPTY;
+  const words = error?.reply?.error?.message;
+  return words ? String(words) : String(error?.message || "OpenRouter didn't answer.");
+}
+
 /** An answer from OpenRouter with the person's key (openrouter.mjs, as Lamp's app sends it), its troubles said as an engine's. */
-async function viaOpenRouter({ system, prompt, schema, seconds, model, images, signal }) {
+async function viaOpenRouter({ system, prompt, schema, seconds, model, images, signal, spent }) {
   try {
-    return (await openRouterText({ key: process.env.OPENROUTER_API_KEY, prompt, images, system: system || "You are a helpful assistant inside an app.",
-      model: model || DEFAULT_MODEL, json: !!schema, schema, seconds, signal })).text;
+    const { text, cost } = await openRouterText({ key: process.env.OPENROUTER_API_KEY, prompt, images, system: system || "You are a helpful assistant inside an app.",
+      model: model || DEFAULT_MODEL, json: !!schema, schema, seconds, signal });
+    spent?.(cost);
+    return text;
   } catch (error) {
     if (!(error instanceof OpenRouterTrouble)) throw error;
-    const words = error.reply?.error?.message;
-    throw new EngineTrouble(error.kind === "refused" ? "failed" : error.kind, words ? String(words) : error.message);
+    spent?.(error.cost);
+    throw new EngineTrouble(error.kind === "refused" ? "failed" : error.kind, error.kind === "refused" ? openRouterWords(error) : error.message);
   }
 }
 
 // MARK: Status and asking
 
-/** `/ai/status`: what can answer here. Text, decisions and actions with an engine; pictures as `pictureRoute` says. */
-export function status(page) {
-  const which = engine();
-  const route = which ? named(which) : null;
-  const available = (kind) => route ? { available: true, route, where: "remote" } : { available: false, reason: missing, setup: "lamp" };
-  const drawer = pictureRoute(page);
-  return {
-    text: !!route, decisions: !!route, pictures: !!drawer, act: !!route,
-    details: { text: available("text"), decisions: available("decisions"), act: available("act"),
-      pictures: drawer ? { available: true, route: named(drawer), where: "remote" } : { available: false, reason: noPictures(page), setup: "lamp" } },
-  };
+/** `/ai/status`: what can answer here. Text, decisions and actions with an engine; pictures as `pictureRoute` says.
+ *  What goes through OpenRouter answers only while it takes the key (`keyProblem`). */
+export async function status(page, problem = keyProblem) {
+  const which = engine(), drawer = pictureRoute(page);
+  const broken = which === "openrouter" || drawer === "openrouter" ? await problem() : null;
+  const one = (by, reason) => !by ? { available: false, reason, setup: "lamp" }
+    : by === "openrouter" && broken ? { available: false, reason: broken, setup: "lamp" } : { available: true, route: named(by), where: "remote" };
+  const details = { text: one(which, missing), decisions: one(which, missing), act: one(which, missing), pictures: one(drawer, noPictures(page)) };
+  return { text: details.text.available, decisions: details.decisions.available, pictures: details.pictures.available, act: details.act.available, details };
 }
 
 // MARK: Models (`LampAI.models`, and `model` on `ask` and `act`)
@@ -281,16 +331,35 @@ const GUARD_SECONDS = 600;
 const secondsFor = (body) => (typeof body?.seconds === "number" && body.seconds > 0 ? Math.min(body.seconds, GUARD_SECONDS) : GUARD_SECONDS);
 
 /** `/ai`: a text answer, or JSON when asked for it. */
-export async function ask(body, here = engines()) {
+export async function ask(body, here = engines(), signal) {
   const prompt = typeof body?.prompt === "string" ? body.prompt.trim() : "";
   if (!prompt) return [400, { error: "Missing prompt." }];
+  // A JSON Schema the answer must fit (`LampAI.ask`'s `schema`): every engine holds the model to it.
+  const schema = body.schema && typeof body.schema === "object" && !Array.isArray(body.schema) ? body.schema : undefined;
+  const json = !!body.json || !!schema;
   let system = typeof body.system === "string" ? body.system : "";
-  if (body.json) system += (system ? "\n\n" : "") + "Answer with JSON only, nothing around it.";
+  if (json) system += (system ? "\n\n" : "") + "Answer with JSON only, nothing around it.";
   const { which, model } = routeFor(body.model, here);
-  try { return [200, { text: await complete({ system, prompt, images: body.images, which, model, seconds: secondsFor(body) }) }]; }
-  catch (error) { return [502, { error: error.message }]; }
+  let cost = 0;
+  const spent = (more) => { cost += Number(more) || 0; };
+  // One time for the whole request, its second try included, as Lamp's app gives it (`AppAI.complete`).
+  const deadline = Date.now() + secondsFor(body) * 1000;
+  const once = (asked) => complete({ system, prompt: asked, schema, images: body.images, which, model, seconds: Math.max(1, (deadline - Date.now()) / 1000), signal, spent });
+  try {
+    let text = await once(prompt);
+    // JSON that doesn't read (cut off, or with words around it) is no use to the page: asked once more, saying so.
+    if (json && !readsAsJSON(text)) {
+      if (deadline - Date.now() < 5000) return [502, { error: "The AI's reply wasn't valid JSON, and there was no time left to ask again.", cost }];
+      text = await once(`${prompt}\n\nYour previous reply was not valid JSON (it may have been cut off). Reply with the complete JSON only.`);
+      if (!readsAsJSON(text)) return [502, { error: "The AI's reply wasn't valid JSON, even on a second try. Ask for less at once.", cost }];
+    }
+    return [200, { text: json ? unfenced(text) : text, cost }];
+  } catch (error) { return [502, { error: error.message, cost }]; }
 }
 
+const unfenced = (text) => String(text).trim().replace(/^```(?:json)?\s*|\s*```$/g, "").trim();
+function readsAsJSON(text) { try { JSON.parse(unfenced(text)); return true; } catch { return false; } }
+
 // MARK: Decisions
 
 /** `/ai/decide`: each question answered by a text model, told and read as Lamp's app does it (app-ai.mjs). */
@@ -309,9 +378,34 @@ export async function decide(body) {
 // MARK: Actions (`LampAI.act`)
 
 /** `/ai/act`: one step of the page's agent loop, told and checked as Lamp's app does it (app-ai.mjs). */
+// A step being streamed (`stream` on lamp.js's act), by its id: its calls so far, and its answer once it's done. The page
+// asks for what's new (`follow`) until it has the answer; a step nobody follows for a few minutes is let go.
+const live = new Map();
+const FOLLOW_WAIT = 50_000;
+
+/** What a streamed step has that the page hasn't (`have` calls): waits for something new, or the end, a while at most. */
+async function following(id, have) {
+  const step = live.get(id);
+  if (!step) return [404, { error: "This step is over." }];
+  step.seen = Date.now();
+  if (step.calls.length <= have && !step.done) {
+    await new Promise((woken) => { const timer = setTimeout(woken, FOLLOW_WAIT); step.waiting.add(() => { clearTimeout(timer); woken(); }); });
+  }
+  const calls = step.calls.slice(have);
+  if (!step.done) return [200, { stream: id, more: true, calls }];
+  live.delete(id);
+  const [status, json] = step.done;
+  return [status, { ...json, stream: id, more: false, calls: [...calls, ...(Array.isArray(json.calls) ? json.calls : [])] }];
+}
+
 export async function act(body, here = engines()) {
   const first = here[0] ?? null;
-  if (body?.probe === true) return [200, { act: !!first, route: first || "", why: first ? "" : missing }];
+  if (typeof body?.follow === "string") return following(body.follow, Math.max(0, Math.floor(Number(body.have)) || 0));
+  if (body?.probe === true) {
+    // A key OpenRouter refuses can't answer a step, as `/ai/status` says too.
+    const refused = first === "openrouter" ? await keyProblem() : null;
+    return [200, { act: !!first && !refused, route: first || "", why: refused || (first ? "" : missing) }];
+  }
   if (!first) return [501, { error: missing }];
   let r;
   try { r = readStep(JSON.stringify(body ?? null)); } catch (refused) { return [refused.status ?? 400, { error: refused.message }]; }
@@ -321,26 +415,52 @@ export async function act(body, here = engines()) {
   const seconds = 600;
   const deadline = Date.now() + Math.min(r.seconds, seconds) * 1000;
   const tag = "outside-text-" + randomBytes(4).toString("hex");
-  let retry = null;
+  // Streamed: each call goes to the page as soon as it's whole, and the page follows the step for the rest.
+  if (body?.stream === true) {
+    const id = "st" + randomBytes(6).toString("hex");
+    const step = { calls: [], done: null, waiting: new Set(), seen: Date.now() };
+    const wake = () => { for (const woken of step.waiting) woken(); step.waiting.clear(); };
+    live.set(id, step);
+    for (const [other, kept] of live) if (Date.now() - kept.seen > 5 * 60_000) live.delete(other);
+    runStep(r, { which, model, tag, deadline, seconds, release: (call) => { step.calls.push(call); wake(); } })
+      .then((done) => { step.done = done; wake(); });
+    return following(id, 0);
+  }
+  return runStep(r, { which, model, tag, deadline, seconds });
+}
+
+/** A step asked of its route, asked again once when its answer can't be used: [status, json]. With `release`, calls go
+ *  out as they're written, and once one has gone the answer isn't asked again. */
+async function runStep(r, { which, model, tag, deadline, seconds, release }) {
+  let retry = null, cost = 0;
   for (let attempt = 0; attempt < 2; attempt++) {
     const left = (deadline - Date.now()) / 1000;
-    if (left < 1) return [504, { error: "This step ran out of time." }];
+    if (left < 1) return [504, { error: "This step ran out of time.", cost }];
     let read;
+    const reader = release ? stepReader(r, release) : null;
     try {
       if (which === "openrouter") {
         const body = chatBody(r, tag, retry, model);
-        read = readMessage((await openRouterChat({ key: process.env.OPENROUTER_API_KEY, body, seconds: left })).message, r);
+        const answered = reader
+          ? await openRouterChatStream({ key: process.env.OPENROUTER_API_KEY, body, seconds: left, onCall: (name, args) => reader.call(name, args) })
+          : await openRouterChat({ key: process.env.OPENROUTER_API_KEY, body, seconds: left });
+        cost += answered.cost;
+        read = readMessage(answered.message, r);
       } else {
-        const { system, prompt } = planPrompt(r, tag, retry);
-        read = readPlan(await complete({ system, prompt, schema: planSchema(r), seconds: left, which, model }), r);
+        const { system, prompt, images } = planPrompt(r, tag, retry);
+        const text = await complete({ system, prompt, images, schema: planSchema(r), seconds: left, which, model, onPartial: reader ? (so) => reader.planned(so) : undefined });
+        reader?.planned(text);
+        read = readPlan(text, r);
       }
+      if (reader) read = afterStream(reader, read);
     } catch (error) {
-      const words = error?.reply?.error?.message || error.message;
-      return Date.now() >= deadline || error?.kind === "timeout" ? [504, { error: "This step ran out of time." }] : [502, { error: String(words) }];
+      cost += Number(error?.cost) || 0;
+      const words = error instanceof OpenRouterTrouble && error.kind === "refused" ? openRouterWords(error) : error?.reply?.error?.message || error.message;
+      return Date.now() >= deadline || error?.kind === "timeout" ? [504, { error: "This step ran out of time.", cost }] : [502, { error: String(words), cost }];
     }
-    if (read.answer) return [200, { ...read.answer, route: which, seconds }];
+    if (read.answer) return [200, { ...read.answer, route: which, seconds, cost }];
     if (attempt === 0) { retry = read.problem; continue; }
-    return [200, { reply: "I couldn't work out how to do that.", ask: false, route: which, seconds, trouble: read.problem }];
+    return [200, { reply: "I couldn't work out how to do that.", ask: false, route: which, seconds, trouble: read.problem, cost }];
   }
   return [502, { error: "The AI didn't answer." }];
 }
@@ -404,7 +524,7 @@ export async function makeImage(body, page, app = {}) {
     }
   } catch (error) {
     if (error?.kind === "timeout") return [504, { error: "The picture took too long to make." }];
-    return [502, { error: String(error?.reply?.error?.message || error.message) }];
+    return [502, { error: openRouterWords(error) }];
   }
   const type = pictureType(made);
   if (!type) return [502, { error: route === "codex" ? "Codex didn't make the picture. Check it's signed in." : "The image model didn't return a picture." }];
@@ -483,11 +603,11 @@ export function codexImage({ prompt, references = [], aspectRatio, seconds = PIC
 }
 
 /** A page's AI request, by its route, as Lamp's app answers it: [status, JSON]. `app` is what the app chose for itself
- *  (`imageModel`), when that's known. */
-export async function answer(path, body, page, app = {}) {
-  if (path === "/ai/status") return [200, status(page)];
+ *  (`imageModel`), when that's known; `signal` stops an answer the page no longer waits for. */
+export async function answer(path, body, page, app = {}, signal) {
+  if (path === "/ai/status") return [200, await status(page)];
   if (path === "/ai/models") return [200, models(enginesFor(page))];
-  if (path === "/ai") return ask(body, enginesFor(page));
+  if (path === "/ai") return ask(body, enginesFor(page), signal);
   if (path === "/ai/act") return act(body, enginesFor(page));
   if (path === "/ai/decide") return decide(body);
   if (path === "/ai/hear") return body?.probe === true ? [200, { hear: false, why: "Hearing needs Lamp's app." }] : [501, { error: "Hearing needs Lamp's app: open the app in Lamp." }];

package/cli/app-ai.mjs

@@ -17,7 +17,11 @@
 // `MAX_STEPS` is the most steps an app may raise a turn to (`ACT_LIMITS_MAX` in lamp.js, `LampActSession.mostSteps`):
 // room to build something real a few calls at a time, while a turn that goes round in circles still ends. The most
 // writes follows from it, since each step holds at most `MAX_CALLS`.
-export const MAX_TOOLS = 24, MAX_CALLS = 8, MAX_STEPS = 40, MAX_BODY = 2 * 1024 * 1024, MAX_CHOICES = 4, MAX_CHOICE = 24;
+export const MAX_TOOLS = 24, MAX_CALLS = 8, MAX_STEPS = 40, MAX_BODY = 3.5 * 1024 * 1024, MAX_CHOICES = 4, MAX_CHOICE = 24;
+/** The most pictures a step carries from its tools' results (`LampAI.picture`), the newest; lamp.js sends no more, and
+ *  keeps them small enough (`ACT_PICTURE_BYTES`) for a step with its text results to stay under `MAX_BODY`. */
+export const MAX_PICTURES = 6;
+const PICTURE_TYPES = ["image/jpeg", "image/png", "image/webp", "image/gif"];
 /** The longest a step may take, whatever its route, before its call is taken to have stopped answering (`ACT_STEP_SECONDS`
  *  in lamp.js, `AppAct.turnSeconds`). A turn has no time of its own unless the app gives it one. */
 export const MAX_TURN_SECONDS = 600;
@@ -69,7 +73,7 @@ function parseArg(raw, path) {
  * steps, route, seconds, final, model?}`, each tool's `args` as [name, shape] pairs by name. Throws `Refused`.
  */
 export function readStep(text) {
-  if (Buffer.byteLength(text) > MAX_BODY) throw new Refused(413, "A step can carry at most 2 MB.");
+  if (Buffer.byteLength(text) > MAX_BODY) throw new Refused(413, "A step can carry at most 3.5 MB.");
   let body;
   try { body = JSON.parse(text); } catch {}
   if (!body || typeof body !== "object" || Array.isArray(body)) throw new Refused(400, "A step needs a JSON object.");
@@ -95,6 +99,8 @@ export function readStep(text) {
   const rawSteps = Array.isArray(body.steps) ? body.steps.filter((s) => s && typeof s === "object") : [];
   if (rawSteps.length >= MAX_STEPS) throw new Refused(400, `A turn has at most ${MAX_STEPS} steps.`);
   const seen = new Set(), steps = [];
+  // The pictures the results carry, numbered in the order the turn made them; only the newest MAX_PICTURES are shown.
+  const pictures = [];
   for (const raw of rawSteps) {
     const calls = [];
     for (const call of Array.isArray(raw.calls) ? raw.calls.filter((c) => c && typeof c === "object") : []) {
@@ -104,10 +110,30 @@ export function readStep(text) {
       calls.push({ id: call.id, name: call.name, args: call.args && typeof call.args === "object" && !Array.isArray(call.args) ? call.args : {} });
     }
     if (!calls.length || calls.length > MAX_CALLS) throw new Refused(400, `Every earlier step has 1 to ${MAX_CALLS} calls.`);
-    const results = {};
-    for (const result of Array.isArray(raw.results) ? raw.results : []) if (typeof result?.id === "string") results[result.id] = result;
-    steps.push({ calls, results });
+    const results = {}, shown = [];
+    for (const result of Array.isArray(raw.results) ? raw.results : []) {
+      if (typeof result?.id !== "string") continue;
+      const { pictures: given, ...rest } = result;
+      const call = calls.find((c) => c.id === result.id);
+      // A result's pictures go to the model beside it as pictures, never as text: the result names each one in its place.
+      for (const picture of Array.isArray(given) ? given : []) {
+        // `data` null is one the page no longer sends, which the model has seen.
+        if (!call || !PICTURE_TYPES.includes(picture?.media_type) || (picture.data !== null && (typeof picture.data !== "string" || !/^[A-Za-z0-9+/]+=*$/.test(picture.data)))) {
+          throw new Refused(400, "A result's pictures are each a `media_type` (JPEG, PNG, WebP or GIF) and its `data` in base64.");
+        }
+        // Numbered by the page, which names it so in the result ("[picture 2]"); a number that isn't one of its own is replaced.
+        const taken = new Set(pictures.map((p) => p.number));
+        const number = Number.isInteger(picture.number) && picture.number > 0 && !taken.has(picture.number) ? picture.number : Math.max(0, ...taken) + 1;
+        const one = { number, call: call.name, media_type: picture.media_type, data: picture.data };
+        pictures.push(one);
+        shown.push(one);
+      }
+      results[result.id] = rest;
+    }
+    steps.push({ calls, results, pictures: shown });
   }
+  // Older pictures than the newest few are left out: the model has seen them, and the step says they're no longer shown.
+  for (const old of [...pictures].sort((a, b) => a.number - b.number).slice(0, -MAX_PICTURES)) old.data = null;
   const history = (Array.isArray(body.history) ? body.history : []).filter((h) => h && typeof h === "object").slice(-3)
     .map((h) => ({ said: typeof h.said === "string" ? h.said : "", heard: typeof h.heard === "string" ? h.heard : "" }));
   const seconds = typeof body.seconds === "number" ? body.seconds : 60;
@@ -289,12 +315,102 @@ export function readPlan(text, r) {
   return readMessage({ tool_calls: [{ function: { name: REPLY, arguments: JSON.stringify(args) } }] }, r);
 }
 
+// MARK: Calls as they're written
+
+/**
+ * The calls finished so far in an answer in `planSchema`'s shape that's still being written: each element of its `calls`
+ * whose closing brace has come, parsed, in order. What's after them (an element half written, the reply) waits.
+ */
+export function plannedCalls(text) {
+  const found = [];
+  const key = text.search(/"calls"\s*:\s*\[/);
+  if (key < 0) return found;
+  let i = text.indexOf("[", key) + 1, depth = 0, start = -1, inString = false, escaped = false;
+  for (; i < text.length; i++) {
+    const c = text[i];
+    if (inString) {
+      if (escaped) escaped = false;
+      else if (c === "\\") escaped = true;
+      else if (c === '"') inString = false;
+      continue;
+    }
+    if (c === '"') inString = true;
+    else if (c === "{" || c === "[") { if (depth === 0) start = i; depth++; }
+    else if (c === "}" || c === "]") {
+      if (depth === 0) break;
+      depth--;
+      if (depth === 0 && start >= 0) {
+        try { found.push(JSON.parse(text.slice(start, i + 1))); } catch { break; }
+        start = -1;
+      }
+    }
+  }
+  return found;
+}
+
+/**
+ * Reads a step's calls one at a time as the model writes them (`stream` on `LampAI.act`), with the checks `readMessage`
+ * gives a whole answer: `release` gets each call that fits as soon as it's whole, with the id it would have had. One that
+ * doesn't fit is held, with why (`problem`), until one that fits comes after it, which it goes before, in its place, as
+ * a call that failed: the page never runs it, and a change that failed stops the changes after it. One left held at the
+ * end goes last when calls were released (`afterStream`), or, when none was, the whole answer is read and asked again
+ * as before. Calls past `MAX_CALLS` are left out.
+ */
+export function stepReader(r, release) {
+  const step = r.steps.length + 1;
+  let given = 0, seen = 0;
+  const reader = {
+    released: 0,
+    held: [],
+    /** One whole call, `args` its arguments as an object or as the JSON text the model wrote. */
+    call(name, raw) {
+      if (name === REPLY) return;
+      const id = `s${step}c${++given}`;
+      let args = raw;
+      if (typeof raw === "string") { try { args = JSON.parse(raw || "{}"); } catch { args = undefined; } }
+      const hold = (problem) => reader.held.push({ id, name: String(name), args: args && typeof args === "object" && !Array.isArray(args) ? args : {}, problem });
+      // Past the most a step holds, a call is left out: the AI sees what ran, and writes the rest next.
+      if (given > MAX_CALLS) return;
+      if (typeof name !== "string") return hold("A call had no name.");
+      if (!args || typeof args !== "object" || Array.isArray(args)) return hold(`${name}'s arguments weren't a JSON object.`);
+      if (r.final) return hold("Only `reply` is possible now.");
+      const tool = r.tools.find((t) => t.name === name);
+      if (!tool) return hold(`There's no tool called ${name}.`);
+      const bad = toolProblem(tool, args);
+      if (bad) return hold(bad);
+      reader.released++;
+      for (const held of reader.held.splice(0)) release(held);
+      release({ id, name, args });
+    },
+    /** The answer in `planSchema`'s shape as written so far: calls finished since last time are read. An optional
+     *  argument left null is left out, as `readPlan` leaves it. */
+    planned(text) {
+      const found = plannedCalls(text);
+      for (const call of found.slice(seen)) {
+        const args = call?.args && typeof call.args === "object" && !Array.isArray(call.args) ? Object.fromEntries(Object.entries(call.args).filter(([, v]) => v !== null)) : call?.args;
+        reader.call(call?.name, args);
+      }
+      seen = found.length;
+    },
+  };
+  return reader;
+}
+
+/**
+ * A streamed step's answer once the model is done: when calls were released, only what's left (the calls that didn't
+ * fit, as failures); else the whole answer read as before (`read`, from `readMessage` or `readPlan`).
+ */
+export function afterStream(reader, read) {
+  if (!reader.released) return read;
+  return { answer: { calls: reader.held.map((c) => ({ id: c.id, name: c.name, args: c.args, problem: c.problem })) } };
+}
+
 // MARK: What the model is told
 
 const RULES = `You act for the person inside this app, only through its tools.
 - Use a tool to look something up before acting on it, and use ids exactly as the tools gave them. Never invent an id.
 - Change only what the person asked for. When the request is unclear (which one, who, what day or time), ask back in the reply instead of guessing.
-- Several look-ups can go in one answer. Changes run one after another, each checked right before it runs; a change that fails stops the ones after it in that answer.
+- Several calls can go in one answer, and they run in the order you write them: look-ups side by side run together, changes run one after another, each checked right before it runs, and a change that fails stops the changes after it in that answer. A look-up after a change sees it.
 - Never say something was done unless its result says it was. If something failed or was skipped, say so.
 - When the request is met, or you need to ask, finish with \`reply\`: one or two short sentences that can be spoken aloud, no lists, no emoji, no markdown. Set \`ask\` to true only when the reply is a question for the person.
 - When your question has a few short answers (yes or no, which of two or three), give them in \`choices\`: two to four, each a word or a few, plain text, in the order to offer them. Leave \`choices\` out when any answer could do.`;
@@ -336,6 +452,16 @@ export function chatBody(r, tag, retry, model) {
   for (const step of r.steps) {
     messages.push({ role: "assistant", content: "", tool_calls: step.calls.map((call) => ({ id: call.id, type: "function", function: { name: call.name, arguments: json(call.args) } })) });
     for (const call of step.calls) messages.push({ role: "tool", tool_call_id: call.id, content: fence(tag, json(step.results[call.id] ?? skipped)) });
+    // A tool's result can't hold a picture on every provider, so the step's pictures follow its results as the person's
+    // message, each named as its result names it.
+    if (step.pictures.length) {
+      const content = [{ type: "text", text: picturesSaid(step.pictures) }];
+      for (const picture of step.pictures.filter((p) => p.data)) {
+        content.push({ type: "text", text: `[picture ${picture.number}], from ${picture.call}:` });
+        content.push({ type: "image_url", image_url: { url: `data:${picture.media_type};base64,${picture.data}` } });
+      }
+      messages.push({ role: "user", content });
+    }
   }
   if (retry) messages.push({ role: "user", content: `Your last answer couldn't be used: ${retry} Answer again.` });
   const tools = r.tools.map((tool) => ({ type: "function", function: { name: tool.name,
@@ -381,12 +507,24 @@ export function planPrompt(r, tag, retry) {
   system += "\n\n" + PLAN;
   if (r.final) system += '\nOnly a reply is possible now: answer {"reply": {...}}.';
   else system += "\n\nThe app's tools:\n" + r.tools.map((tool) => `- ${tool.name}: ${tool.reads ? "Looks up, changes nothing. " : "Changes the app's data. "}${tool.about}\n  Arguments: ${json(toolSchema(tool))}`).join("\n");
+  const images = [];
   r.steps.forEach((step, i) => {
     prompt += `\n\nYour answer ${i + 1} called:\n` + json(step.calls.map((c) => ({ name: c.name, args: c.args })));
     prompt += "\nWhat happened, in that order:\n" + fence(tag, json(step.calls.map((c) => ({ ...(step.results[c.id] ?? skipped), name: c.name }))));
+    if (step.pictures.length) prompt += "\n" + picturesSaid(step.pictures, true);
+    for (const picture of step.pictures) if (picture.data) images.push({ media_type: picture.media_type, data: picture.data });
   });
   if (retry) prompt += `\n\nYour last answer couldn't be used: ${retry} Answer again.`;
-  return { system, prompt };
+  return { system, prompt, images };
+}
+
+/** What's said of a step's pictures: which come with the message, in order, and which are no longer shown. */
+function picturesSaid(pictures, attached = false) {
+  const named = (list) => list.map((p) => `[picture ${p.number}] (from ${p.call})`).join(", ");
+  const sent = pictures.filter((p) => p.data), gone = pictures.filter((p) => !p.data);
+  let said = sent.length ? `${attached ? "Attached to this message, in this order" : "The pictures those results name"}: ${named(sent)}. Look at them before you answer.` : "";
+  if (gone.length) said += `${said ? " " : ""}No longer shown, as newer pictures came since: ${named(gone)}.`;
+  return said;
 }
 
 // MARK: Decisions (`LampAI.decide`) answered by a text model