@womp/spark-agent 6.1.0 → 6.2.0

package/CHANGELOG.md

@@ -4,6 +4,31 @@ Notable changes to `@womp/spark-agent`, newest first. Versions follow
 [Semantic Versioning](https://semver.org/). Add an entry with each version bump; describe what
 changed for hosts and users, and how to migrate across breaking changes.
 
+## 6.2.0 — 2026-10-09
+
+### Added
+
+- `intent-bench` CLI: measures which route Spark takes (scene, OpenSCAD, Geometry Nodes, image,
+  image edit, mesh, image-to-mesh, Flow, import, or a reply) for the build-intent QA requests
+  under the current creation rules, preview follow-ups and attached-photo cases, in parallel over
+  fixtures, with gpt-6-luna by default. See the README.
+
+### Changed
+
+- An attached image without a request gets a question about what to make of it, not a generation.
+  Turning an attached image into 3D generates the mesh from it directly, with no new preview.
+- Characters, creatures, people, food, sculptures, specific vehicles and branded items preview
+  first even when simple or stylized, unless the request names a method or has a source image.
+- `flow-graphs` (v5) builds the requested graph with the catalog's nodes when an input, such as a
+  conversation upload, cannot be expressed in source, and tells the user which input to set.
+
+## 6.1.1 — 2026-10-08
+
+### Fixed
+
+- Image-to-mesh and other generation run inline unless the user asks for Flow or nodes. Spark no
+  longer asks the user to open Flow for an attached photo.
+
 ## 6.1.0 — 2026-10-08
 
 ### Fixed

package/README.md

@@ -262,6 +262,26 @@ Progress goes to stderr; one JSON result goes to stdout. Use `yarn --silent agen
 stdout. Exit codes: 0 completed, 1 run/transport/export failure, 2 invalid arguments, 3 requires
 user action, 130 interrupted.
 
+## Measure intent routing
+
+`yarn intent-bench` checks which route Spark takes for the requests in
+`src/intent-bench-cases.ts`: editable scene, OpenSCAD, Geometry Nodes, image, image edit, mesh,
+image-to-mesh, Flow, import, or a reply without acting. It needs no server or renderer. Each run
+gets Swan's skills over fixtures: an empty scene, the case's attachments and earlier turns, and
+Flow open or closed. Photo cases download their image from Womp's media host. The first call that would change the scene, generate, write a graph or
+import is recorded, and the run stops before it takes effect. Material edits do not decide a
+route, since Geometry Nodes prepares its materials in the scene first.
+
+```sh
+yarn intent-bench                                  # every case 3 times on gpt-6-luna, 24 at once
+yarn intent-bench --only photo-only,F1 --repeat 10 --model gpt-6-astra
+yarn intent-bench --list
+```
+
+It prints each case's routes and pass rate and writes `report.json`, with every run's skills,
+calls, deciding call and reply, under `.spark/intent-bench/`. It exits 1 when any run took a route
+the case does not expect. Model calls are paid; the full set costs about $0.10 on gpt-6-luna.
+
 ## Run cost estimates
 
 The chat displays estimated API token cost in USD after every completed model step, from the

package/dist/capabilities/flow.js

@@ -11,14 +11,13 @@ export function createFlowCapability(host, catalog) {
         skills: [
             {
                 name: "flow-graphs",
-                version: "3",
+                version: "5",
                 description: "Read and edit the open Workbench Flow pipeline: image/mesh generation, processing, and viewers.",
                 instructions: `Use womp.flow for the open Workbench pipeline.
-Use Flow for image/mesh generation explicitly requested through Flow/nodes or continued edits to that Flow workflow, unless the user asks for an inline result.
-An open Flow canvas alone does not redirect ordinary scene creation or inline generation.
 Read the current graph and flow-graphs-reference before writing. Supply the complete edited s-expression document and the latest revision. Preserve unrelated nodes and stable node IDs. Omitting a node deletes it; empty source clears the graph.
 Existing node configuration, edge identity, layout, runtime metadata, and generated outputs are retained for unchanged node identities/types. Scalar inputs represented in source are replaced by your document. The host retains preservedInputs for those same nodes. Non-private preserved inputs can be explicitly replaced in the document; never recreate asset objects or cached values as JSON strings. Private settings remain editor-owned: preserve existing private connections unchanged between retained nodes and do not introduce new private literals or connections.
 If read returns source null, explain the diagnostics. Do not replace an unreadable graph as though it were empty.
+Build the requested graph from the catalog's nodes even when an input cannot be expressed in source, such as a conversation upload or an editor-owned image: leave that input unconnected and tell the user which node input to set in the editor.
 The Flow canvas must be open. If unavailable, tell the user to open Flow or Split in a project, or Nodes in the sidebar, and wait for their next message. Never claim a graph was created after a failed call.
 Writes validate types, preserve the runner lifecycle, and await commit. They edit the graph; they do not report that generation jobs completed. Repair compiler errors in this same conversation with a fresh read. On committed/unknown receipts after failure, inspect the graph before retrying; do not replay blindly.`,
                 declarations: WOMP_FLOW_DECLARATIONS,

package/dist/intent-bench-cases.d.ts

@@ -0,0 +1,42 @@
+import type { OpenScadTarget } from "./capabilities.js";
+/** The first action a run commits to, or how it ended without one. */
+export declare const INTENT_ROUTES: readonly ["scene", "openscad", "geometry-nodes", "image", "image-edit", "mesh", "image-to-mesh", "flow", "import", "reply", "undecided", "error"];
+export type IntentRoute = (typeof INTENT_ROUTES)[number];
+/** A photo downloaded from `url`, or a generated solid color. */
+export type IntentImage = {
+    url: string;
+} | {
+    name: string;
+    color: [number, number, number];
+};
+export type IntentTurn = {
+    role: "user";
+    text?: string;
+    images?: IntentImage[];
+}
+/** Earlier assistant text; `generated` images are listed as that turn's generation outputs. */
+ | {
+    role: "assistant";
+    text: string;
+    generated?: IntentImage[];
+};
+export type IntentCase = {
+    id: string;
+    group: "core" | "workspace" | "conversation" | "photo";
+    /** Earlier turns; the last user turn is the request under test. */
+    turns: IntentTurn[];
+    /** Flow is listed whether or not a canvas is open, as in Swan; "unavailable" omits it. */
+    flow?: "open" | "closed" | "unavailable";
+    /** false omits the 3D stage capabilities (scene, materials, geometry, OpenSCAD, imports). */
+    stage?: boolean;
+    openScadParts?: OpenScadTarget[];
+    /** Acceptable routes. More than one only where the shared prompt allows either. */
+    expect: readonly IntentRoute[];
+    note?: string;
+};
+/**
+ * The build-intent QA requests (22 Sep 2026), with expectations updated to the creation
+ * rules restored in 6.0 (image previews for ambiguous and likeness-dependent requests),
+ * plus preview follow-ups and attached-photo cases.
+ */
+export declare const INTENT_CASES: IntentCase[];

package/dist/intent-bench-cases.js

@@ -0,0 +1,149 @@
+/** The first action a run commits to, or how it ended without one. */
+export const INTENT_ROUTES = [
+    "scene",
+    "openscad",
+    "geometry-nodes",
+    "image",
+    "image-edit",
+    "mesh",
+    "image-to-mesh",
+    "flow",
+    "import",
+    "reply",
+    "undecided",
+    "error",
+];
+const user = (text, images) => ({ role: "user", text, images });
+// A takeaway coffee cup on a table, from Womp's media host.
+const cup = { url: "https://media.womp.com/staging-kkp/thumbnails/ma_IMG_3417_1791494968752.webp" };
+const washer = "difference() { cylinder(h = thickness, d = outer_diameter, $fn = 96); translate([0, 0, -1]) cylinder(h = thickness + 2, d = hole_diameter, $fn = 96); }";
+/**
+ * The build-intent QA requests (22 Sep 2026), with expectations updated to the creation
+ * rules restored in 6.0 (image previews for ambiguous and likeness-dependent requests),
+ * plus preview follow-ups and attached-photo cases.
+ */
+export const INTENT_CASES = [
+    { id: "P1", group: "core", turns: [user("Make a washer: 20 mm outer diameter, 8 mm hole, 2 mm thick.")], expect: ["openscad"] },
+    { id: "P2", group: "core", turns: [user("Make a parametric phone stand where I can adjust the width and tilt angle.")], expect: ["openscad"] },
+    { id: "P3", group: "core", turns: [user("Make a storage tray, 120 × 80 × 25 mm.")], expect: ["openscad"] },
+    { id: "G1", group: "core", turns: [user("Make a branching coral with adjustable branch density.")], expect: ["geometry-nodes"] },
+    { id: "G2", group: "core", turns: [user("Make a rippling surface with adjustable wave speed.")], expect: ["geometry-nodes"] },
+    { id: "G3", group: "core", turns: [user("Make an organic vine where I can adjust the curl and leaf spacing.")], expect: ["geometry-nodes"] },
+    {
+        id: "S1", group: "core", turns: [user("A car")], expect: ["image"],
+        note: "Ambiguous creative request: one image preview, then a choice. The 4.3 QA expected scene.",
+    },
+    { id: "S2", group: "core", turns: [user("Make a cup of water in the scene.")], expect: ["scene"] },
+    {
+        id: "S3", group: "core", turns: [user("Make a friendly little mushroom in 3D.")], expect: ["image"],
+        note: "A character depends on likeness, so it previews first. The 4.3 QA expected scene.",
+    },
+    { id: "S4", group: "core", turns: [user("Make a simple wooden stool in 3D.")], expect: ["scene"], note: "Plain functional objects build directly." },
+    { id: "S5", group: "core", turns: [user("Make a 3D corgi.")], expect: ["image"], note: "Creatures preview first." },
+    { id: "S6", group: "core", turns: [user("Make a 3D model of a Ferrari F40.")], expect: ["image"], note: "Specific vehicles preview first." },
+    { id: "S7", group: "core", turns: [user("a cozy cabin in the woods")], expect: ["image"], note: "Ambiguous creative request." },
+    { id: "I1", group: "core", turns: [user("Generate an image of a red toy car on a white background.")], expect: ["image"] },
+    { id: "I2", group: "core", turns: [user("Make a studio product photo of a blue ceramic cup.")], expect: ["image"] },
+    { id: "I3", group: "core", turns: [user("Make a picture of a small fantasy mushroom house.")], expect: ["image"] },
+    { id: "I4", group: "core", turns: [user("Design a logo for a coffee shop called Bean There.")], expect: ["image"] },
+    { id: "M1", group: "core", turns: [user("Generate a red toy car mesh.")], expect: ["mesh"] },
+    { id: "M2", group: "core", turns: [user("Generate a mesh of a blue ceramic water cup.")], expect: ["mesh"] },
+    { id: "M3", group: "core", turns: [user("Generate a fantasy mushroom house mesh.")], expect: ["mesh"] },
+    { id: "F1", group: "core", flow: "open", turns: [user("Make an image of a red toy car using Flow nodes.")], expect: ["flow"] },
+    { id: "F2", group: "core", flow: "open", turns: [user("Make a ceramic cup mesh using Flow.")], expect: ["flow"] },
+    { id: "F3", group: "core", flow: "open", turns: [user("Set up a nodes workflow that makes an image of a mushroom house.")], expect: ["flow"] },
+    {
+        id: "I1-flow-open", group: "workspace", flow: "open",
+        turns: [user("Generate an image of a red toy car on a white background.")], expect: ["image"],
+        note: "An open Flow canvas does not redirect ordinary generation.",
+    },
+    {
+        id: "F1-flow-closed", group: "workspace", flow: "closed",
+        turns: [user("Make an image of a red toy car using Flow nodes.")], expect: ["reply"],
+        note: "Explains that Flow must be open instead of generating inline.",
+    },
+    {
+        id: "S2-no-stage", group: "workspace", stage: false,
+        turns: [user("Make a cup of water in the scene.")], expect: ["reply"],
+        note: "Explains that the 3D stage is unavailable instead of switching route.",
+    },
+    {
+        id: "P1-follow-up", group: "conversation",
+        turns: [
+            user("Make a washer: 20 mm outer diameter, 8 mm hole, 2 mm thick."),
+            { role: "assistant", text: "I made the washer as an adjustable part (object 12): 20 mm outer diameter, 8 mm hole and 2 mm thick." },
+            user("Make the hole 10 mm."),
+        ],
+        openScadParts: [{
+                kind: "part", objectId: 12, parentId: 0, title: "Washer", source: washer,
+                params: { outer_diameter: 20, hole_diameter: 8, thickness: 2 },
+            }],
+        expect: ["openscad"],
+    },
+    {
+        id: "I1-follow-up", group: "conversation",
+        turns: [
+            user("Generate an image of a red toy car on a white background."),
+            { role: "assistant", text: "Here's your red toy car on a white background.", generated: [{ name: "red-toy-car.png", color: [200, 30, 30] }] },
+            user("Make it blue."),
+        ],
+        expect: ["image-edit"],
+    },
+    {
+        id: "ambiguous-yes", group: "conversation",
+        turns: [
+            user("Generate an image of a red toy car on a white background."),
+            {
+                role: "assistant",
+                text: "Here's your red toy car. I can change its colors, or turn it into a sculpted 3D model.",
+                generated: [{ name: "red-toy-car.png", color: [200, 30, 30] }],
+            },
+            user("Yes"),
+        ],
+        expect: ["reply"],
+        note: "Two options were offered, so a bare yes needs clarification.",
+    },
+    {
+        id: "preview-to-mesh", group: "conversation",
+        turns: [
+            user("A car"),
+            {
+                role: "assistant",
+                text: "Here's a quick preview. Should I refine the image, turn it into a sculpted 3D piece, or build it from individually editable shapes?",
+                generated: [{ name: "car-preview.png", color: [40, 90, 200] }],
+            },
+            user("Turn it into a sculpted 3D piece."),
+        ],
+        expect: ["image-to-mesh"],
+        note: "Reuses the preview as the image-to-mesh input.",
+    },
+    {
+        id: "preview-to-shapes", group: "conversation",
+        turns: [
+            user("A car"),
+            {
+                role: "assistant",
+                text: "Here's a quick preview. Should I refine the image, turn it into a sculpted 3D piece, or build it from individually editable shapes?",
+                generated: [{ name: "car-preview.png", color: [40, 90, 200] }],
+            },
+            user("Build it from editable shapes."),
+        ],
+        expect: ["scene"],
+    },
+    { id: "photo-3d", group: "photo", turns: [user("make this 3D", [cup])], expect: ["image-to-mesh"] },
+    {
+        id: "photo-3d-flow-open", group: "photo", flow: "open", turns: [user("make this 3D", [cup])], expect: ["image-to-mesh"],
+        note: "Without an explicit Flow request, an attached photo is standard image-to-mesh.",
+    },
+    { id: "photo-shapes", group: "photo", turns: [user("build this with editable shapes", [cup])], expect: ["scene"] },
+    {
+        id: "photo-only", group: "photo", turns: [user(undefined, [cup])], expect: ["reply"],
+        note: "A photo alone does not say what to make, so ask. dev0 thread 145108 asked to open Flow instead.",
+    },
+    { id: "photo-only-flow-open", group: "photo", flow: "open", turns: [user(undefined, [cup])], expect: ["reply"] },
+    { id: "photo-flow-closed", group: "photo", flow: "closed", turns: [user("turn this into a 3D model using Flow", [cup])], expect: ["reply"] },
+    {
+        id: "photo-flow-open", group: "photo", flow: "open", turns: [user("turn this into a 3D model using Flow", [cup])], expect: ["flow"],
+        note: "Flow has no node for a chat photo; build the graph with what it has and leave the image input to the user.",
+    },
+];

package/dist/intent-bench-cli.d.ts

@@ -0,0 +1,2 @@
+#!/usr/bin/env node
+export {};

package/dist/intent-bench-cli.js

@@ -0,0 +1,122 @@
+#!/usr/bin/env node
+import { existsSync } from "node:fs";
+import { mkdir, writeFile } from "node:fs/promises";
+import path from "node:path";
+import { loadEnvFile } from "node:process";
+import { parseArgs } from "node:util";
+import { INTENT_CASES } from "./intent-bench-cases.js";
+import { intentBenchModel, runIntentBench, summarizeIntentRuns } from "./intent-bench.js";
+import { SPARK_MODELS } from "./models.js";
+function positiveInteger(value, name) {
+    const number = Number(value);
+    if (!Number.isSafeInteger(number) || number < 1)
+        throw new Error(`${name} must be a positive integer.`);
+    return number;
+}
+async function main() {
+    const { values } = parseArgs({ options: {
+            model: { type: "string", default: "gpt-6-luna" }, reasoning: { type: "string", default: "medium" },
+            repeat: { type: "string", default: "3" }, concurrency: { type: "string", default: "24" },
+            only: { type: "string" }, group: { type: "string" }, "max-steps": { type: "string", default: "12" },
+            "timeout-seconds": { type: "string", default: "240" }, output: { type: "string" },
+            list: { type: "boolean" }, help: { type: "boolean" },
+        } });
+    if (values.help) {
+        console.log(`intent-bench [options]
+
+Measures which route Spark takes for ${INTENT_CASES.length} requests: editable scene, OpenSCAD, Geometry Nodes,
+image, image edit, mesh, image-to-mesh, Flow, import, or a reply without acting.
+Runs use Swan's skills over fixtures: no renderer, no generation, no scene changes. The first
+call that would change something is recorded and the run stops there. Makes paid model calls.
+
+--model <id>              Default: gpt-6-luna. Models: ${SPARK_MODELS.map((model) => model.id).join(", ")}
+--reasoning <effort>      Default: medium
+--repeat <n>              Runs per case; default: 3
+--concurrency <n>         Runs at once; default: 24
+--only <ids>              Comma-separated case IDs (--list shows them)
+--group <names>           core, workspace, conversation, photo
+--max-steps <n>           Model steps per run; default: 12
+--timeout-seconds <n>     Per run; default: 240
+--output <directory>      New directory only; default: .spark/intent-bench/<timestamp>
+--list                    List the cases and expected routes`);
+        return;
+    }
+    if (values.list) {
+        for (const testCase of INTENT_CASES) {
+            const prompt = testCase.turns.at(-1);
+            const text = prompt.role === "user" ? `${prompt.images?.length ? "[photo] " : ""}${prompt.text ?? ""}` : "";
+            console.log(`${testCase.id.padEnd(22)} ${testCase.group.padEnd(13)} ${testCase.expect.join("|").padEnd(22)} ${text}`);
+        }
+        return;
+    }
+    const model = SPARK_MODELS.find((item) => item.id === values.model);
+    if (!model)
+        throw new Error("Select a supported model using --model. See --help.");
+    const reasoning = values.reasoning;
+    if (!model.reasoningEfforts.includes(reasoning))
+        throw new Error("Unsupported reasoning effort for this model.");
+    const only = values.only?.split(",").map((id) => id.trim()).filter(Boolean);
+    const groups = values.group?.split(",").map((group) => group.trim()).filter(Boolean);
+    const unknown = only?.filter((id) => !INTENT_CASES.some((testCase) => testCase.id === id)) ?? [];
+    if (unknown.length)
+        throw new Error(`Unknown case IDs: ${unknown.join(", ")}. Use --list.`);
+    const cases = INTENT_CASES.filter((testCase) => (!only || only.includes(testCase.id)) && (!groups || groups.includes(testCase.group)));
+    if (!cases.length)
+        throw new Error("No cases selected.");
+    const repeat = positiveInteger(values.repeat, "--repeat");
+    const concurrency = positiveInteger(values.concurrency, "--concurrency");
+    const maxSteps = positiveInteger(values["max-steps"], "--max-steps");
+    const timeoutMs = positiveInteger(values["timeout-seconds"], "--timeout-seconds") * 1000;
+    if (existsSync(".env"))
+        loadEnvFile();
+    const resolved = intentBenchModel({ modelId: model.id, reasoningEffort: reasoning });
+    const output = path.resolve(values.output ?? `.spark/intent-bench/${new Date().toISOString().replaceAll(":", "-")}`);
+    // Never reuse a result directory or overwrite a previous benchmark.
+    await mkdir(path.dirname(output), { recursive: true });
+    await mkdir(output);
+    const controller = new AbortController();
+    const stop = () => controller.abort(new Error("Benchmark interrupted."));
+    process.once("SIGINT", stop);
+    process.once("SIGTERM", stop);
+    const total = cases.length * repeat;
+    console.error(`${total} runs (${cases.length} cases × ${repeat}) on ${model.id}/${reasoning}, ${Math.min(concurrency, total)} at once. Results: ${output}`);
+    const startedAt = Date.now();
+    let done = 0;
+    const line = (run) => `${run.pass ? "pass" : "FAIL"}  ${run.caseId.padEnd(22)} #${run.repeat}  ${run.route.padEnd(14)} ${run.error ?? run.decision?.call ?? ""}`;
+    try {
+        const runs = await runIntentBench({
+            cases, repeat, concurrency, model: resolved, maxSteps, timeoutMs, signal: controller.signal,
+            onRun: (run) => console.error(`[${String(++done).padStart(String(total).length)}/${total}] ${line(run)}`),
+        });
+        const summary = summarizeIntentRuns(cases, runs);
+        const passed = runs.filter((run) => run.pass).length;
+        const priced = runs.filter((run) => run.costUsd !== null);
+        const costUsd = priced.reduce((sum, run) => sum + run.costUsd, 0);
+        const durationMs = Date.now() - startedAt;
+        await writeFile(path.join(output, "report.json"), `${JSON.stringify({
+            model: model.id, reasoning, repeat, maxSteps, startedAt: new Date(startedAt).toISOString(), durationMs,
+            passed, runs: runs.length, costUsd: priced.length ? costUsd : null, costStatus: priced.length === runs.length ? "estimated" : priced.length ? "partial" : "unavailable",
+            summary, results: runs,
+        }, null, 2)}\n`);
+        console.log(`\n${"case".padEnd(22)} ${"expected".padEnd(22)} pass   routes`);
+        for (const item of summary) {
+            const routes = Object.entries(item.routes).map(([route, count]) => `${route}×${count}`).join(" ");
+            const mark = item.passed === item.runs ? " " : "!";
+            console.log(`${mark}${item.id.padEnd(21)} ${item.expect.join("|").padEnd(22)} ${`${item.passed}/${item.runs}`.padEnd(6)} ${routes}`);
+        }
+        const reliable = summary.filter((item) => item.passed === item.runs).length;
+        const gaps = runs.filter((run) => run.fixtureErrors.length).length;
+        console.log(`\n${passed}/${runs.length} runs on the expected route (${Math.round((100 * passed) / runs.length)}%), ` +
+            `${reliable}/${summary.length} cases on it every time. ${(durationMs / 1000).toFixed(0)} s, ` +
+            `${priced.length ? `$${costUsd.toFixed(3)} estimated` : "cost unavailable"}.` +
+            (gaps ? `\n${gaps} runs hit a read the fixtures cannot answer (fixtureErrors in the report).` : ""));
+        console.log(`Report: ${path.join(output, "report.json")}`);
+        if (passed < runs.length)
+            process.exitCode = 1;
+    }
+    finally {
+        process.removeListener("SIGINT", stop);
+        process.removeListener("SIGTERM", stop);
+    }
+}
+main().catch((error) => { console.error(error instanceof Error ? error.message : String(error)); process.exitCode = 1; });

package/dist/intent-bench-flow-catalog.d.ts

@@ -0,0 +1 @@
+export declare const INTENT_BENCH_FLOW_CATALOG: string;

package/dist/intent-bench-flow-catalog.js

@@ -0,0 +1,4 @@
+// Swan's Flow node catalog (`flowNodeCatalog()` in swan-frontend src/llm/flowbot/editableDocument.ts),
+// copied from crow-backend src/utils/agent/flow-node-catalog.generated.ts on 2026-10-08 for
+// intent-bench runs. Refresh it from there when the Flow node surface changes.
+export const INTENT_BENCH_FLOW_CATALOG = "Connect only the listed input and output handles. Set literal-only settings with values, never refs. generateImage exposes Image1 through ImageN according to :num_images (default 1); use Image1, not its hidden array outputs.\n\nai.llm\nConnectable inputs:\n  System: base.string\n  Prompt: base.string\n  Image1: res.bitmap\n  Image2: res.bitmap\n  Image3: res.bitmap\n  Image4: res.bitmap\nConnectable outputs:\n  Result: base.string\nLiteral-only settings (no connections):\n  Model: enum:ai.llmModel\n\nai.llmText\nConnectable inputs:\n  System: base.string\n  Prompt1: base.string\n  Prompt2: base.string\n  Prompt3: base.string\n  Prompt4: base.string\nConnectable outputs:\n  Result: base.string\nLiteral-only settings (no connections):\n  Model: enum:ai.llmModel\n\nai.removeImageBackground\nConnectable inputs:\n  Image: res.image\n  Model: enum:ai.processImage.removeBackgroundModel\n  Sensitivity: base.float\nConnectable outputs:\n  Image: res.bitmap\nLiteral-only settings (no connections):\n  Temporary: base.bool\n\nai.upscaleImage\nConnectable inputs:\n  Image: res.bitmap\n  Model: enum:ai.upscaleModel\n  Scale: base.float\nConnectable outputs:\n  Image: res.bitmap\nLiteral-only settings (no connections):\n  Temporary: base.bool\n\nbase.input.string\nConnectable inputs:\n  Value: base.string\nConnectable outputs:\n  Result: base.string\nLiteral-only settings (no connections):\n\n\ndb.scriptInstance\nConnectable inputs:\n\nConnectable outputs:\n  Result: db.scriptInstance\nLiteral-only settings (no connections):\n\n\next.ai.generateImage\nConnectable inputs:\n  Prompt: base.string\n  Image1: res.bitmap\n  Image2: res.bitmap\n  Image3: res.bitmap\n  Image4: res.bitmap\nConnectable outputs:\n  Image1: res.bitmap\n  Image2: res.bitmap - only with :num_images 2 or more\n  Image3: res.bitmap - only with :num_images 3 or more\n  Image4: res.bitmap - only with :num_images 4 or more\nLiteral-only settings (no connections):\n  PromptPreset: base.string\n  AiWorkflow: enum:db.aiWorkflowId\n  Seed: base.float\n  RandomSeed: base.bool\n  NumImages: base.float\n  Temporary: base.bool\n\next.ai.generateMesh\nConnectable inputs:\n  Image: res.bitmap\n  Back: res.bitmap\n  Left: res.bitmap\n  Right: res.bitmap\nConnectable outputs:\n  Mesh: res.mesh\nLiteral-only settings (no connections):\n  AiWorkflow: enum:db.aiWorkflowId\n  Seed: base.float\n  RandomSeed: base.bool\n\next.edit.paper\nConnectable inputs:\n  Background: res.image\n  Overlays: res.image[]\n  Width: base.float\n  Height: base.float\n  Title: base.string\nConnectable outputs:\n  Image: res.bitmap\nLiteral-only settings (no connections):\n\n\next.scene.area\nConnectable inputs:\n  Script: db.scriptInstance\nConnectable outputs:\n  AreaId: enum:scene.areaId\nLiteral-only settings (no connections):\n  ProjectId: enum:db.projectId\n  AreaId: enum:scene.areaId\n  Hide: base.bool\n\nscene.importAsset\nConnectable inputs:\n  Image: res.bitmap\n  Mesh: res.mesh\n  AreaId: enum:scene.areaId\nConnectable outputs:\n  AreaId: enum:scene.areaId\nLiteral-only settings (no connections):\n  ProjectId: enum:db.projectId\n  AssetType: enum:db.assetType\n  LibraryOnly: base.bool\n\nscene.printEstimate\nConnectable inputs:\n  Target: enum:scene.areaId | res.mesh[]\n  Material: base.string\n  Color: base.string\n  Scale: base.float\n  Thickness: base.float\n  DisableAIPreview: base.bool\nConnectable outputs:\n  Preview: res.image\nLiteral-only settings (no connections):\n  ProjectId: enum:db.projectId\n\nviewer.images\nConnectable inputs:\n  Images: res.bitmap[]\nConnectable outputs:\n  Image: res.bitmap\nLiteral-only settings (no connections):\n  ImageIndex: base.float\n\nviewer.meshes\nConnectable inputs:\n  Meshes: res.mesh[]\nConnectable outputs:\n  Mesh: res.mesh\nLiteral-only settings (no connections):\n  MeshIndex: base.float";

package/dist/intent-bench.d.ts

@@ -0,0 +1,66 @@
+import { type SparkAgentOptions } from "./agent.js";
+import type { IntentCase, IntentRoute } from "./intent-bench-cases.js";
+import type { SparkRunSettings } from "./server/types.js";
+export type IntentDecision = {
+    route: IntentRoute;
+    /** The first call that would have changed something; the run stopped there. */
+    call: string;
+    detail?: unknown;
+    /** The JavaScript it ran in. */
+    code?: string;
+};
+/** A scene edit made in the same call as another route's write belongs to that route. */
+export declare function intentRoute(decision: IntentDecision): IntentRoute;
+export type IntentRun = {
+    caseId: string;
+    repeat: number;
+    route: IntentRoute;
+    pass: boolean;
+    decision?: IntentDecision;
+    /** Skills the model loaded, in order. */
+    skills: string[];
+    /** Agent API calls the fixtures served, in order. */
+    calls: string[];
+    /** Fixture reads that could not answer; the run may not reflect production. */
+    fixtureErrors: string[];
+    text: string;
+    /** Thinking, skill loads, code and its outcome, and replies, in order. */
+    transcript: string[];
+    steps: number;
+    costUsd: number | null;
+    durationMs: number;
+    error?: string;
+};
+/** A solid-color PNG, so conversation cases need no image files. */
+export declare function solidPng([r, g, b]: [number, number, number], size?: number): Uint8Array;
+export type IntentModel = Pick<SparkAgentOptions, "model" | "modelId" | "providerOptions" | "maxOutputTokens">;
+/** The standalone server's provider settings for a Spark model. */
+export declare function intentBenchModel(settings: SparkRunSettings): IntentModel;
+export declare function runIntentCase(options: {
+    testCase: IntentCase;
+    repeat: number;
+    model: IntentModel;
+    maxSteps: number;
+    timeoutMs: number;
+    signal?: AbortSignal;
+}): Promise<IntentRun>;
+export type IntentCaseSummary = {
+    id: string;
+    expect: readonly IntentRoute[];
+    runs: number;
+    passed: number;
+    routes: Partial<Record<IntentRoute, number>>;
+    note?: string;
+};
+export declare function summarizeIntentRuns(cases: readonly IntentCase[], runs: readonly IntentRun[]): IntentCaseSummary[];
+/** Runs every case `repeat` times, at most `concurrency` at once. */
+export declare function runIntentBench(options: {
+    cases: readonly IntentCase[];
+    repeat: number;
+    concurrency: number;
+    model: IntentModel;
+    maxSteps: number;
+    timeoutMs: number;
+    signal?: AbortSignal;
+    onRun?(run: IntentRun): void;
+}): Promise<IntentRun[]>;

package/dist/intent-bench.js

@@ -0,0 +1,555 @@
+import { crc32, deflateSync } from "node:zlib";
+import { createAnthropic } from "@ai-sdk/anthropic";
+import { createGoogleGenerativeAI } from "@ai-sdk/google";
+import { createOpenAI } from "@ai-sdk/openai";
+import { runSparkAgent } from "./agent.js";
+import { composeCapabilities, createAccountCapability, createAssetsCapability, createFlowCapability, createGeometryCapability, createKnowledgeCapability, createMaterialsCapability, createOpenScadCapability, createOperationsCapability, createSceneCapability, } from "./capabilities.js";
+import { geometryReference } from "./geometry.js";
+import { INTENT_BENCH_FLOW_CATALOG } from "./intent-bench-flow-catalog.js";
+import { sparkModelOptions } from "./model-options.js";
+import { getSparkModel } from "./models.js";
+import { QuickJSJavaScriptExecutor } from "./quickjs-executor.js";
+import { createBenchMetrics } from "./run-metrics.js";
+import { createWompSceneApi } from "./womp-scene-api.js";
+/**
+ * Skills that edit the scene to prepare their own target in the same call: Geometry Nodes
+ * creates an Area in scene.access and reads it as its target, for example.
+ */
+const PREPARED_WRITES = [
+    [/\bwomp\.geometry\.(?:read|write)\s*\(/u, "geometry-nodes"],
+    [/\bwomp\.openscad\.(?:create|update)\s*\(/u, "openscad"],
+];
+/** A scene edit made in the same call as another route's write belongs to that route. */
+export function intentRoute(decision) {
+    const { route, code } = decision;
+    if (route !== "scene" || code === undefined)
+        return route;
+    return PREPARED_WRITES.find(([pattern]) => pattern.test(code))?.[1] ?? "scene";
+}
+const FLOW_CLOSED = "Open the Flow canvas before using flow-graphs.";
+const GENERATION_ROUTES = {
+    "text-to-image": "image",
+    "image-to-image": "image-edit",
+    "upscale-image": "image-edit",
+    "remove-image-background": "image-edit",
+    "text-to-mesh": "mesh",
+    "image-to-mesh": "image-to-mesh",
+};
+/** A solid-color PNG, so conversation cases need no image files. */
+export function solidPng([r, g, b], size = 64) {
+    const chunk = (type, data) => {
+        const body = Buffer.concat([Buffer.from(type, "ascii"), data]);
+        const length = Buffer.alloc(4);
+        length.writeUInt32BE(data.length);
+        const crc = Buffer.alloc(4);
+        crc.writeUInt32BE(crc32(body));
+        return Buffer.concat([length, body, crc]);
+    };
+    const header = Buffer.alloc(13);
+    header.writeUInt32BE(size, 0);
+    header.writeUInt32BE(size, 4);
+    header.set([8, 2, 0, 0, 0], 8);
+    const row = Buffer.concat([Buffer.from([0]), Buffer.from(Array.from({ length: size }, () => [r, g, b]).flat())]);
+    return new Uint8Array(Buffer.concat([
+        Buffer.from([137, 80, 78, 71, 13, 10, 26, 10]),
+        chunk("IHDR", header),
+        chunk("IDAT", deflateSync(Buffer.concat(Array.from({ length: size }, () => row)))),
+        chunk("IEND", Buffer.alloc(0)),
+    ]));
+}
+// Each photo is downloaded once per process and shared by every run that attaches it.
+const downloads = new Map();
+async function download(url) {
+    const response = await fetch(url, { signal: AbortSignal.timeout(30_000) });
+    const mediaType = response.headers.get("content-type")?.split(";")[0]?.trim() ?? "";
+    if (!response.ok || !mediaType.startsWith("image/")) {
+        throw new Error(`Cannot download the case image ${url}: HTTP ${response.status} ${mediaType}`.trim());
+    }
+    return { name: decodeURIComponent(new URL(url).pathname.split("/").pop()), mediaType, data: new Uint8Array(await response.arrayBuffer()) };
+}
+async function loadImage(image) {
+    if ("color" in image)
+        return { name: image.name, mediaType: "image/png", data: solidPng(image.color) };
+    let loaded = downloads.get(image.url);
+    if (!loaded) {
+        loaded = download(image.url);
+        downloads.set(image.url, loaded);
+        loaded.catch(() => downloads.delete(image.url));
+    }
+    return loaded;
+}
+class IntentDecided extends Error {
+    decision;
+    constructor(decision) {
+        super(`Intent recorded: ${decision.route}`);
+        this.decision = decision;
+        this.name = "IntentDecided";
+    }
+}
+/**
+ * Swan's capability set over fixtures. Reads answer from an empty scene and the case's
+ * conversation; the first call that would change the scene, generate, write a graph or
+ * import is the decision, and stops the run before it takes effect.
+ */
+async function intentFixtures(testCase, controller) {
+    const calls = [];
+    const fixtureErrors = [];
+    let decision;
+    // Spark runs one execute_javascript call at a time.
+    let code;
+    const note = (call) => calls.push(call);
+    const decide = (route, call, detail) => {
+        note(call);
+        decision ??= { route, call, detail, ...(code === undefined ? {} : { code }) };
+        controller.abort(new IntentDecided(decision));
+        throw new IntentDecided(decision);
+    };
+    /** An error production would also return, such as a node that does not exist. */
+    const refuse = (call, message) => {
+        note(call);
+        throw new Error(message);
+    };
+    /** A read these fixtures cannot answer; the run may then differ from production. */
+    const missing = (call, message) => {
+        fixtureErrors.push(call);
+        return refuse(call, message);
+    };
+    const files = new Map();
+    const assets = [];
+    const messages = [];
+    const addFile = async (image, messageId, source) => {
+        const loaded = await loadImage(image);
+        const id = `${source}-${messageId}-${files.size + 1}`;
+        files.set(id, { artifact: { id, url: `fixture://${id}`, mediaType: loaded.mediaType, name: loaded.name }, data: loaded.data });
+        assets.push({ artifact: { id, mediaType: loaded.mediaType, name: loaded.name }, messageId, kind: "image", source });
+        return loaded;
+    };
+    for (const [index, turn] of testCase.turns.entries()) {
+        const messageId = 1001 + index;
+        if (turn.role === "assistant") {
+            for (const image of turn.generated ?? [])
+                await addFile(image, messageId, "generated");
+            messages.push({ role: "assistant", content: turn.text });
+            continue;
+        }
+        const content = [];
+        for (const image of turn.images ?? []) {
+            const loaded = await addFile(image, messageId, "uploaded");
+            content.push({ type: "file", data: loaded.data, mediaType: loaded.mediaType, filename: loaded.name });
+        }
+        if (turn.text)
+            content.push({ type: "text", text: turn.text });
+        messages.push({ role: "user", content });
+    }
+    const screenshot = { id: "scene-screenshot", mediaType: "image/png", name: "scene.png" };
+    files.set(screenshot.id, { artifact: { ...screenshot, url: "fixture://scene-screenshot" }, data: solidPng([228, 228, 228]) });
+    // An empty scene: the root group and the default material. Materials are prepared for
+    // whichever route follows (Geometry Nodes assigns them too), so material edits apply here
+    // and do not decide the route; a run that only edits materials is a scene edit.
+    const transform = { position: { x: 0, y: 0, z: 0 }, rotation: { x: 0, y: 0, z: 0 }, scale: { x: 1, y: 1, z: 1 } };
+    // The root, plus the case's OpenSCAD parts from earlier turns.
+    const nodes = [
+        { id: 0, kind: "group", name: "", parentId: null, pickable: true, transform },
+        ...(testCase.openScadParts ?? []).map((part) => ({
+            id: part.objectId, kind: "openScad", name: part.title, parentId: part.parentId, pickable: true, transform,
+        })),
+    ];
+    const node = (id) => nodes.find((item) => item.id === id) ?? refuse(`scene.getNode(${id})`, `Node ${id} does not exist.`);
+    const materials = new Map([
+        [1, { id: 1, data: { color: { x: 0.74, y: 0.74, z: 0.74 }, roughness: 0.22, metalness: 0 }, shaders: [] }],
+    ]);
+    const ext = { materialsExt: { 1: { name: "Default Material", tier: 0 } } };
+    const materialEdits = [];
+    const material = (id) => materials.get(id) ?? refuse(`scene.getMaterial(${id})`, `Material ${id} does not exist.`);
+    const sceneReads = {
+        getNode: (id) => structuredClone(node(id)),
+        getNodeChildren: (id) => nodes.filter((item) => item.id !== 0 && item.parentId === id).map((item) => structuredClone(item)),
+        listNodes: () => nodes.map((item) => structuredClone(item)),
+        listNodeSummaries: () => nodes.map(({ id, name, kind, parentId }) => ({ id, name, kind, parentId })),
+        getSceneExt: () => structuredClone(ext),
+        getSceneProperties: () => ({}),
+        listMaterials: () => [...materials.values()].map((item) => structuredClone(item)),
+        getMaterial: (id) => structuredClone(material(id)),
+        createMaterial: (data) => {
+            const id = Math.max(...materials.keys()) + 1;
+            materials.set(id, { id, data: structuredClone(data), shaders: [] });
+            return structuredClone(material(id));
+        },
+        updateMaterial: (id, patch) => {
+            const item = material(id);
+            item.data = { ...item.data, ...structuredClone(patch) };
+            return structuredClone(item);
+        },
+        deleteMaterial: (id) => {
+            const item = material(id);
+            materials.delete(id);
+            return item;
+        },
+        // Metadata: material names here; node metadata needs a node, which only a decision creates.
+        patchSceneExt: (operations) => {
+            for (const { op, path, value } of operations) {
+                const keys = path.slice(1).split("/");
+                const last = keys.pop();
+                let parent = ext;
+                for (const key of keys)
+                    parent = (parent[key] ??= {});
+                if (op === "remove")
+                    delete parent[last];
+                else
+                    parent[last] = structuredClone(value);
+            }
+            return undefined;
+        },
+    };
+    const scene = new Proxy(sceneReads, {
+        get: (target, key) => typeof key !== "string" ? undefined : target[key] ?? ((...args) => decide("scene", `scene.${key}`, args)),
+    });
+    const MATERIAL_EDITS = new Set(["createMaterial", "updateMaterial", "deleteMaterial", "materialImport"]);
+    const sceneCall = (method, args, action) => {
+        if (method === "getNodeBoundingBox")
+            return action();
+        if (!MATERIAL_EDITS.has(method))
+            return decide("scene", `scene.${method}`, args);
+        note(`scene.${method}`);
+        materialEdits.push({ call: `scene.${method}`, args });
+        return action();
+    };
+    const bounds = { min: { x: 0, y: 0, z: 0 }, max: { x: 0, y: 0, z: 0 }, center: { x: 0, y: 0, z: 0 }, size: { x: 0, y: 0, z: 0 } };
+    const draft = () => createWompSceneApi({
+        scene: scene,
+        engine: { getNodeBoundingBox: async () => bounds },
+        rootId: 0,
+        getDefaultMaterialId: () => 1,
+        fonts: [],
+        requireNode: (id) => {
+            node(id);
+        },
+        importLibraryMaterial: async (name) => {
+            const id = Math.max(...materials.keys()) + 1;
+            materials.set(id, { id, data: { name }, shaders: [] });
+            return id;
+        },
+        trackNode: () => undefined,
+        trackMaterial: () => undefined,
+        untrackMaterial: () => undefined,
+        listMaterialIds: () => [...materials.keys()],
+        requireMaterial: (id) => {
+            material(id);
+        },
+        call: sceneCall,
+        callAsync: sceneCall,
+        print: () => undefined,
+    });
+    const flow = testCase.flow ?? "closed";
+    const stage = testCase.stage ?? true;
+    const bindings = {
+        assets: createAssetsCapability({
+            list: async (options) => {
+                note("assets.list");
+                const items = assets.filter((item) => (options.kind === undefined || item.kind === options.kind) &&
+                    (options.source === undefined || item.source === options.source) &&
+                    (options.messageId === undefined || item.messageId === options.messageId));
+                return { items: items.slice(options.from, options.from + options.count), nextFrom: null };
+            },
+        }, stage
+            ? {
+                importMesh: async (request) => decide("import", "assets.importMesh", request),
+                importImage: async (request) => decide("import", "assets.importImage", request),
+                retryMeshImport: async (request) => decide("import", "assets.retryMeshImport", request),
+                readImport: async (key) => refuse("assets.readImport", `No import with key ${key}.`),
+                finishImport: async (key) => refuse("assets.finishImport", `No import with key ${key}.`),
+            }
+            : undefined, stage
+            ? {
+                importTexture: async (request) => decide("import", "assets.importTexture", request),
+                readTextureImport: async (key) => refuse("assets.readTextureImport", `No texture import with key ${key}.`),
+            }
+            : undefined),
+        operations: createOperationsCapability({
+            catalog: async () => {
+                note("operations.catalog");
+                return {
+                    models: [
+                        { id: "womp-image", name: "Womp Image", requiresSubscription: false, operations: ["text-to-image", "image-to-image", "upscale-image", "remove-image-background"] },
+                        { id: "womp-mesh", name: "Womp Mesh", requiresSubscription: false, operations: ["text-to-mesh", "image-to-mesh"] },
+                    ],
+                    defaults: { image: "womp-image", editing: "womp-image", mesh: "womp-mesh" },
+                    confirmationRequired: false,
+                };
+            },
+            list: async () => {
+                note("operations.list");
+                return { jobs: [], nextFrom: null };
+            },
+            start: async (request) => decide(GENERATION_ROUTES[request.type], `operations.generate:${request.type}`, request),
+            read: async (messageId) => missing("operations.read", `No job ${messageId}.`),
+            wait: async (messageId) => missing("operations.wait", `No job ${messageId}.`),
+        }),
+        account: createAccountCapability({ read: async () => missing("account.read", "Account information is unavailable right now.") }),
+        knowledge: createKnowledgeCapability({ search: async () => missing("knowledge.search", "Help search is unavailable right now.") }),
+    };
+    if (flow !== "unavailable") {
+        bindings.flow = createFlowCapability({
+            read: async () => {
+                if (flow === "closed")
+                    return refuse("flow.read", FLOW_CLOSED);
+                note("flow.read");
+                return { flowId: "intent-bench", source: "", revision: "1", preservedInputs: {}, diagnostics: [] };
+            },
+            write: async (options) => (flow === "closed" ? refuse("flow.write", FLOW_CLOSED) : decide("flow", "flow.write", options)),
+        }, INTENT_BENCH_FLOW_CATALOG);
+    }
+    if (stage) {
+        Object.assign(bindings, {
+            scene: createSceneCapability({
+                health: async () => {
+                    note("scene.health");
+                    return { status: "within_limits", freshness: "snapshot", revisionVerified: false, affectedAreaIds: [] };
+                },
+                access: async (callback) => {
+                    note("scene.access");
+                    return callback(draft());
+                },
+                screenshot: async () => {
+                    note("scene.screenshot");
+                    return screenshot;
+                },
+            }),
+            materials: createMaterialsCapability({
+                // An empty node catalog: enough to plan plain colors, not procedural layers.
+                catalog: async () => {
+                    note("materials.catalog");
+                    fixtureErrors.push("materials.catalog (no shader nodes)");
+                    return { operations: [], nodes: {}, textureCount: 0 };
+                },
+                read: async () => {
+                    note("materials.read");
+                    return {
+                        materials: [...materials.keys()].map((id) => ({ id, shaderIds: [], objectIds: [] })),
+                        material: { id: 1, data: structuredClone(material(1).data), layers: [] },
+                        diagnostics: [],
+                    };
+                },
+                write: async (options) => decide("scene", "materials.write", options),
+                status: async () => {
+                    note("materials.status");
+                    return { remaining: 0, pendingResources: {}, rendering: "unverified", explanation: "No material work is pending." };
+                },
+            }),
+            geometry: createGeometryCapability({
+                // The scene root is the creation container.
+                read: async (objectId) => {
+                    if (objectId !== 0)
+                        return refuse("geometry.read", `Object ${objectId} has no Geometry Nodes graph.`);
+                    note("geometry.read");
+                    return { kind: "container", objectId: 0, title: "Scene", source: null, instances: [], diagnostics: [] };
+                },
+                write: async (options) => decide("geometry-nodes", "geometry.write", options),
+            }, geometryReference()),
+            openscad: createOpenScadCapability({
+                read: async (objectId) => {
+                    const part = objectId === 0
+                        ? { kind: "container", objectId: 0, parentId: 0, title: "Scene", source: null, params: {} }
+                        : testCase.openScadParts?.find((item) => item.objectId === objectId);
+                    if (!part)
+                        return refuse("openscad.read", `Object ${objectId} is not an OpenSCAD part.`);
+                    note("openscad.read");
+                    return structuredClone(part);
+                },
+                create: async (options) => decide("openscad", "openscad.create", options),
+                update: async (options) => decide("openscad", "openscad.update", options),
+            }),
+        });
+    }
+    const executor = new QuickJSJavaScriptExecutor();
+    return {
+        capabilities: composeCapabilities(bindings),
+        artifacts: {
+            resolve: async (reference) => {
+                const file = files.get(reference.id);
+                if (!file)
+                    throw new Error(`Artifact unavailable: ${reference.id}`);
+                return file;
+            },
+        },
+        messages,
+        messageId: String(1000 + testCase.turns.length),
+        executor: {
+            execute: (...args) => {
+                code = args[0];
+                return executor.execute(...args);
+            },
+        },
+        result: () => ({
+            calls,
+            fixtureErrors,
+            decision: decision ?? (materialEdits[0] && { route: "scene", call: materialEdits[0].call, detail: materialEdits[0].args }),
+        }),
+    };
+}
+/** The standalone server's provider settings for a Spark model. */
+export function intentBenchModel(settings) {
+    const selected = getSparkModel(settings.modelId);
+    const key = { openai: "OPENAI_API_KEY", anthropic: "ANTHROPIC_API_KEY", google: "GEMINI_API_KEY" }[selected.provider];
+    const apiKey = process.env[key]?.trim();
+    if (!apiKey)
+        throw new Error(`${key} is required in the environment or .env.`);
+    const model = selected.provider === "anthropic" ? createAnthropic({ apiKey })(selected.id)
+        : selected.provider === "google" ? createGoogleGenerativeAI({ apiKey })(selected.id)
+            : createOpenAI({ apiKey }).responses(selected.id);
+    return {
+        model,
+        modelId: selected.id,
+        providerOptions: sparkModelOptions(selected.id, selected.provider, settings.reasoningEffort),
+        maxOutputTokens: 48_000,
+    };
+}
+export async function runIntentCase(options) {
+    const { testCase } = options;
+    const startedAt = Date.now();
+    const controller = new AbortController();
+    const onAbort = () => controller.abort(options.signal?.reason);
+    options.signal?.addEventListener("abort", onAbort, { once: true });
+    const timer = setTimeout(() => controller.abort(new Error(`Timed out after ${options.timeoutMs / 1000} s.`)), options.timeoutMs);
+    const fixtures = await intentFixtures(testCase, controller);
+    const metrics = createBenchMetrics();
+    const skills = [];
+    let streamed = "";
+    const transcript = [];
+    let open;
+    const append = (kind, delta) => {
+        if (open?.kind !== kind) {
+            if (open)
+                transcript.push(`[${open.kind}] ${open.text.trim()}`);
+            open = { kind, text: "" };
+        }
+        open.text += delta;
+    };
+    const entry = (line) => {
+        if (open)
+            transcript.push(`[${open.kind}] ${open.text.trim()}`);
+        open = undefined;
+        transcript.push(line);
+    };
+    let completed;
+    let error;
+    try {
+        const run = runSparkAgent({
+            ...options.model,
+            messages: fixtures.messages,
+            capabilities: fixtures.capabilities,
+            executor: fixtures.executor,
+            artifacts: fixtures.artifacts,
+            runId: `intent-${testCase.id}-${options.repeat}`,
+            messageId: fixtures.messageId,
+            signal: controller.signal,
+            maxSteps: options.maxSteps,
+        });
+        const reader = run.stream.getReader();
+        try {
+            for (;;) {
+                const { done, value } = await reader.read();
+                if (done)
+                    break;
+                const chunk = value;
+                metrics.accept(chunk);
+                if (chunk.type === "text-delta") {
+                    streamed += chunk.delta;
+                    append("reply", chunk.delta);
+                }
+                if (chunk.type === "reasoning-delta")
+                    append("thinking", chunk.delta);
+                if (chunk.type === "tool-input-available") {
+                    const input = chunk.input;
+                    entry(chunk.toolName === "execute_javascript"
+                        ? `[run] ${String(input.purpose ?? "")}
+${String(input.code ?? "")}`
+                        : `[${chunk.toolName}] ${JSON.stringify(input)}`);
+                }
+                if (chunk.type === "tool-output-available") {
+                    const output = chunk.output;
+                    entry(`[result] ${output?.ok === false ? `error: ${String(output.error)}` : JSON.stringify(output?.logs ?? output).slice(0, 600)}`);
+                }
+                if (chunk.type === "tool-output-error")
+                    entry(`[result] error: ${chunk.errorText}`);
+                if (chunk.type === "tool-input-available" && chunk.toolName === "read_skill") {
+                    const names = chunk.input.names;
+                    if (Array.isArray(names))
+                        skills.push(...names.filter((name) => typeof name === "string"));
+                }
+            }
+        }
+        catch {
+            // The run's result reports why the stream stopped.
+        }
+        if (open)
+            transcript.push(`[${open.kind}] ${open.text.trim()}`);
+        const result = await run.result;
+        if (result.status === "completed")
+            completed = result.text;
+    }
+    catch (caught) {
+        if (!(controller.signal.reason instanceof IntentDecided)) {
+            const reason = controller.signal.aborted ? controller.signal.reason : caught;
+            error = reason instanceof Error ? reason.message : String(reason);
+        }
+    }
+    finally {
+        clearTimeout(timer);
+        options.signal?.removeEventListener("abort", onAbort);
+    }
+    const { calls, fixtureErrors, decision } = fixtures.result();
+    const route = decision
+        ? intentRoute(decision)
+        : error ? "error" : completed !== undefined ? "reply" : "undecided";
+    const snapshot = metrics.snapshot();
+    return {
+        caseId: testCase.id,
+        repeat: options.repeat,
+        route,
+        pass: testCase.expect.includes(route),
+        ...(decision ? { decision } : {}),
+        skills,
+        calls,
+        fixtureErrors,
+        text: (completed ?? streamed).trim(),
+        transcript,
+        steps: snapshot.steps,
+        costUsd: snapshot.costUsd,
+        durationMs: Date.now() - startedAt,
+        ...(error ? { error } : {}),
+    };
+}
+export function summarizeIntentRuns(cases, runs) {
+    return cases.map((testCase) => {
+        const own = runs.filter((run) => run.caseId === testCase.id);
+        const routes = {};
+        for (const run of own)
+            routes[run.route] = (routes[run.route] ?? 0) + 1;
+        return {
+            id: testCase.id,
+            expect: testCase.expect,
+            runs: own.length,
+            passed: own.filter((run) => run.pass).length,
+            routes,
+            ...(testCase.note ? { note: testCase.note } : {}),
+        };
+    });
+}
+/** Runs every case `repeat` times, at most `concurrency` at once. */
+export async function runIntentBench(options) {
+    const jobs = options.cases.flatMap((testCase) => Array.from({ length: options.repeat }, (_, repeat) => ({ testCase, repeat })));
+    const runs = [];
+    let next = 0;
+    const worker = async () => {
+        while (next < jobs.length && !options.signal?.aborted) {
+            const job = jobs[next++];
+            const run = await runIntentCase({ ...job, model: options.model, maxSteps: options.maxSteps, timeoutMs: options.timeoutMs, signal: options.signal });
+            runs.push(run);
+            options.onRun?.(run);
+        }
+    };
+    await Promise.all(Array.from({ length: Math.min(options.concurrency, jobs.length) }, worker));
+    const order = new Map(options.cases.map((testCase, index) => [testCase.id, index]));
+    return runs.sort((a, b) => order.get(a.caseId) - order.get(b.caseId) || a.repeat - b.repeat);
+}

package/dist/prompt.d.ts

@@ -1,5 +1,5 @@
 import type { SparkSkill } from "./capabilities.js";
-export declare const SPARK_AGENT_PROMPT = "You are Spark, Womp's unified assistant.\nUse read_skill({names:[...]}) to load the required APIs together. Skills provide TypeScript\ndeclarations for documentation; execute_javascript runs JavaScript, with top-level await.\nOnly the listed capabilities are available. Reading a skill does not grant additional permissions.\nUse only the supplied womp APIs and output helpers to access application services and state.\nIf a capability is unavailable, explain what is missing and ask the user to enable it when applicable.\nDo not work around missing capabilities by inspecting page globals, DOM or framework internals,\nor making direct network requests. Native execution is not a security sandbox; it does not expand\nthe supported API or the user's granted capabilities.\nGive each execute_javascript call a concise present-progressive purpose, such as \"Generating a car image\", \"Inspecting the scene\", or \"Importing the model\". Write purpose before code.\nEach execution has fresh local variables. Keep stable object, job, and artifact references.\nUse print for concise, filtered observations; await emitImage or emitFile with a host-issued artifact\nreference to show actual output. emitImage inspects by default; use { inspect: false } to display without adding image bytes to model context.\nCompleted generations are shown automatically to the user; image bytes are intentionally omitted from your context. Acknowledge completion without listing, re-emitting, or inspecting outputs unless the task requires it. Use returned artifact references for follow-up work.\nDo not claim visual verification without inspecting the image. Missing artifact references must be recovered or reported; omitted image bytes alone are not a missing result.\nInspect the scene before editing and verify the committed result.\nPreserve existing content unless the user asks to replace or delete it. Use the latest target revision\nwhen a write requires one. Capabilities own transactions: an entire script is not atomic.\nRecover ordinary tool errors yourself: use confirmed receipts and fresh reads, retry only unresolved work,\npreserve successful changes, and never replay an interrupted script. Do not narrate internal failures\nunless recovery cannot continue. Never claim an uncertain commit rolled back. Never resubmit a job just because waiting timed out.\nComplete related work in this conversation without delegating to another agent. Report limits,\nunsupported actions, cancellation, and incomplete work truthfully. You cannot see what your work\ncosts: model use can be billed, not only generations. Never estimate credits used or call work free;\nreport only credit figures a host message or tool provides. Finish with a concise explanation\nof the result in the user's language; keep implementation details out of that explanation.\n\nCreation intent\n\nResolve the requested outcome from the current message and preceding dialogue, not keyword matches.\nExplicit method choices take precedence. Continue edits in the existing representation unless the user\nasks to change it. Creation defaults do not apply to help, discussion, imports, or account questions.\n\n- Build directly with editable scene shapes for ordinary 3D requests, including \"3D X\" and\n  \"make X in 3D\", and when the user wants individually editable shapes or geometry that blends\n  with existing objects. Organic appearance alone does not require Geometry Nodes.\n\n- Use OpenSCAD for parametric, measured, mechanical, printable, or real-world-fit objects,\n  except when another method is explicit (including primitives with dimensions), or when\n  parametric organic/procedural geometry, animation, or Geometry Nodes is requested:\n  use Geometry Nodes for those, not Workbench Flow. Do not claim print readiness without validation.\n\n- For mesh wompification, read mesh-wompify and scene-editing together and reconstruct only the specified reference using primitives and curves.\n\n- Use mesh generation when the user explicitly requests a mesh, asks to turn an existing or\n  attached image into 3D, or chooses a sculpted result rather than individually editable shapes.\n\n- Use image generation for pictures, logos, concept art, wallpapers, and other clearly visual output.\n\n- For an ambiguous creative request without explicit 3D or printing intent, generate one image\n  preview first. Then offer one short, outcome-based choice: refine the image, turn it into a\n  sculpted 3D piece, or build it from individually editable shapes. Offer adjustable CAD instead\n  of editable shapes when the subject is measured or must fit something real.\n\n- Also preview first when a 3D request without a named method depends on likeness or surface\n  detail that shapes cannot carry (characters, creatures, people, food, sculptures, specific\n  vehicles or branded items). Plain functional objects still build directly.\n\n- Generating a result does not authorize importing, converting, or building it in the scene.\n\n- Use Workbench Flow for image/mesh generation explicitly requested through Flow/nodes or\n  continued edits to that Flow workflow, unless the user asks for an inline result. Flow must be\n  open; explain that prerequisite if closed instead of silently generating inline. Merely opening\n  Flow does not redirect ordinary scene requests or inline generation to it.\n\n- After a preview, offer the choice above. After other completed generation, you may briefly\n  suggest a relevant next step using available capabilities. Describe its outcome, not internal tools. Suggestions are optional,\n  not a mandatory menu, and never authorize actions. Wait for a clear choice and follow existing\n  confirmations. Reuse the existing artifact/reference for the chosen follow-up, including as the\n  image-to-mesh input, instead of regenerating the source. Preserve its defining features when building it.\n\n- Choose construction methods by the user's requested workflow and the\n  available capabilities, not by assuming editable shapes require lower\n  fidelity. Outside the preview defaults above, a sculpted appearance does not by itself\n  require mesh generation.\n  Respect an explicit method choice and pursue the reference's visible shape\n  and detail with that method. Offer alternatives only for a specific\n  demonstrated limitation; ask before switching or simplifying the requested\n  result. No method guarantees exact reconstruction from a single image.\n\n- Ask at most one focused question at a time, only when a sensible default\n  would likely produce the wrong outcome. Do not repeat resolved questions.\n  Interpret replies using the preceding offer; if several options were\n  offered and \u201Cyes\u201D does not identify one, clarify briefly.\n\n- Treat a clear answer to your question as an instruction to proceed.\n  Continue from existing results rather than restarting the workflow.\n\nOffer only workflows available in this host. Follow their authorization\nand confirmation requirements before generation or scene changes. If a requested workflow is unavailable,\nexplain the limitation rather than silently substituting another.";
+export declare const SPARK_AGENT_PROMPT = "You are Spark, Womp's unified assistant.\nUse read_skill({names:[...]}) to load the required APIs together. Skills provide TypeScript\ndeclarations for documentation; execute_javascript runs JavaScript, with top-level await.\nOnly the listed capabilities are available. Reading a skill does not grant additional permissions.\nUse only the supplied womp APIs and output helpers to access application services and state.\nIf a capability is unavailable, explain what is missing and ask the user to enable it when applicable.\nDo not work around missing capabilities by inspecting page globals, DOM or framework internals,\nor making direct network requests. Native execution is not a security sandbox; it does not expand\nthe supported API or the user's granted capabilities.\nGive each execute_javascript call a concise present-progressive purpose, such as \"Generating a car image\", \"Inspecting the scene\", or \"Importing the model\". Write purpose before code.\nEach execution has fresh local variables. Keep stable object, job, and artifact references.\nUse print for concise, filtered observations; await emitImage or emitFile with a host-issued artifact\nreference to show actual output. emitImage inspects by default; use { inspect: false } to display without adding image bytes to model context.\nCompleted generations are shown automatically to the user; image bytes are intentionally omitted from your context. Acknowledge completion without listing, re-emitting, or inspecting outputs unless the task requires it. Use returned artifact references for follow-up work.\nDo not claim visual verification without inspecting the image. Missing artifact references must be recovered or reported; omitted image bytes alone are not a missing result.\nInspect the scene before editing and verify the committed result.\nPreserve existing content unless the user asks to replace or delete it. Use the latest target revision\nwhen a write requires one. Capabilities own transactions: an entire script is not atomic.\nRecover ordinary tool errors yourself: use confirmed receipts and fresh reads, retry only unresolved work,\npreserve successful changes, and never replay an interrupted script. Do not narrate internal failures\nunless recovery cannot continue. Never claim an uncertain commit rolled back. Never resubmit a job just because waiting timed out.\nComplete related work in this conversation without delegating to another agent. Report limits,\nunsupported actions, cancellation, and incomplete work truthfully. You cannot see what your work\ncosts: model use can be billed, not only generations. Never estimate credits used or call work free;\nreport only credit figures a host message or tool provides. Finish with a concise explanation\nof the result in the user's language; keep implementation details out of that explanation.\n\nCreation intent\n\nResolve the requested outcome from the current message and preceding dialogue, not keyword matches.\nExplicit method choices take precedence. Continue edits in the existing representation unless the user\nasks to change it. Creation defaults do not apply to help, discussion, imports, or account questions.\n\n- Build directly with editable scene shapes for ordinary 3D requests, including \"3D X\" and\n  \"make X in 3D\", and when the user wants individually editable shapes or geometry that blends\n  with existing objects. Organic appearance alone does not require Geometry Nodes.\n\n- Use OpenSCAD for parametric, measured, mechanical, printable, or real-world-fit objects,\n  except when another method is explicit (including primitives with dimensions), or when\n  parametric organic/procedural geometry, animation, or Geometry Nodes is requested:\n  use Geometry Nodes for those, not Workbench Flow. Do not claim print readiness without validation.\n\n- For mesh wompification, read mesh-wompify and scene-editing together and reconstruct only the specified reference using primitives and curves.\n\n- Use mesh generation when the user explicitly requests a mesh, asks to turn an existing or\n  attached image into 3D, or chooses a sculpted result rather than individually editable shapes.\n\n- Use image generation for pictures, logos, concept art, wallpapers, and other clearly visual output.\n\n- For an ambiguous creative request without explicit 3D or printing intent, generate one image\n  preview first. Then offer one short, outcome-based choice: refine the image, turn it into a\n  sculpted 3D piece, or build it from individually editable shapes. Offer adjustable CAD instead\n  of editable shapes when the subject is measured or must fit something real.\n\n- Also preview first when a 3D request names no method (such as a mesh, editable shapes or CAD),\n  has no source image, and is a character, creature, person, food, sculpture, specific vehicle or\n  branded item, even a simple or stylized one: its likeness and surface detail decide the result.\n  Plain functional objects still build directly.\n\n- An attached image is already the reference. Turning it into 3D generates the mesh from it\n  directly, with no new preview. An attached image without a request does not ask for anything\n  yet: ask what to make of it before generating or building.\n\n- Generating a result does not authorize importing, converting, or building it in the scene.\n\n- Use Workbench Flow only when the user names Flow or nodes for the request, or continues edits\n  to an existing Flow workflow. Otherwise run image and mesh generation, including image-to-mesh\n  from an attached photo, inline, whether or not Flow is open; never ask the user to open Flow\n  for it. If the user requested Flow and it is closed, explain that it must be open instead of\n  silently generating inline.\n\n- After a preview, offer the choice above. After other completed generation, you may briefly\n  suggest a relevant next step using available capabilities. Describe its outcome, not internal tools. Suggestions are optional,\n  not a mandatory menu, and never authorize actions. Wait for a clear choice and follow existing\n  confirmations. Reuse the existing artifact/reference for the chosen follow-up, including as the\n  image-to-mesh input, instead of regenerating the source. Preserve its defining features when building it.\n\n- Choose construction methods by the user's requested workflow and the\n  available capabilities, not by assuming editable shapes require lower\n  fidelity. Outside the preview defaults above, a sculpted appearance does not by itself\n  require mesh generation.\n  Respect an explicit method choice and pursue the reference's visible shape\n  and detail with that method. Offer alternatives only for a specific\n  demonstrated limitation; ask before switching or simplifying the requested\n  result. No method guarantees exact reconstruction from a single image.\n\n- Ask at most one focused question at a time, only when a sensible default\n  would likely produce the wrong outcome. Do not repeat resolved questions.\n  Interpret replies using the preceding offer; if several options were\n  offered and \u201Cyes\u201D does not identify one, clarify briefly.\n\n- Treat a clear answer to your question as an instruction to proceed.\n  Continue from existing results rather than restarting the workflow.\n\nOffer only workflows available in this host. Follow their authorization\nand confirmation requirements before generation or scene changes. If a requested workflow is unavailable,\nexplain the limitation rather than silently substituting another.";
 /**
  * The system prompt every host sends: the agent prompt and the skills directory.
  * It must stay identical across the turns and steps of a conversation, or the

package/dist/prompt.js

@@ -52,16 +52,22 @@ asks to change it. Creation defaults do not apply to help, discussion, imports,
   sculpted 3D piece, or build it from individually editable shapes. Offer adjustable CAD instead
   of editable shapes when the subject is measured or must fit something real.
 
-- Also preview first when a 3D request without a named method depends on likeness or surface
-  detail that shapes cannot carry (characters, creatures, people, food, sculptures, specific
-  vehicles or branded items). Plain functional objects still build directly.
+- Also preview first when a 3D request names no method (such as a mesh, editable shapes or CAD),
+  has no source image, and is a character, creature, person, food, sculpture, specific vehicle or
+  branded item, even a simple or stylized one: its likeness and surface detail decide the result.
+  Plain functional objects still build directly.
+
+- An attached image is already the reference. Turning it into 3D generates the mesh from it
+  directly, with no new preview. An attached image without a request does not ask for anything
+  yet: ask what to make of it before generating or building.
 
 - Generating a result does not authorize importing, converting, or building it in the scene.
 
-- Use Workbench Flow for image/mesh generation explicitly requested through Flow/nodes or
-  continued edits to that Flow workflow, unless the user asks for an inline result. Flow must be
-  open; explain that prerequisite if closed instead of silently generating inline. Merely opening
-  Flow does not redirect ordinary scene requests or inline generation to it.
+- Use Workbench Flow only when the user names Flow or nodes for the request, or continues edits
+  to an existing Flow workflow. Otherwise run image and mesh generation, including image-to-mesh
+  from an attached photo, inline, whether or not Flow is open; never ask the user to open Flow
+  for it. If the user requested Flow and it is closed, explain that it must be open instead of
+  silently generating inline.
 
 - After a preview, offer the choice above. After other completed generation, you may briefly
   suggest a relevant next step using available capabilities. Describe its outcome, not internal tools. Suggestions are optional,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@womp/spark-agent",
-  "version": "6.1.0",
+  "version": "6.2.0",
   "description": "Portable skills-based AI agent with local and durable chat transports for Womp",
   "type": "module",
   "main": "./dist/index.js",
@@ -8,7 +8,8 @@
   "bin": {
     "spark-agent": "dist/cli.js",
     "simple-bench": "dist/simple-bench-cli.js",
-    "agent-run": "dist/agent-run-cli.js"
+    "agent-run": "dist/agent-run-cli.js",
+    "intent-bench": "dist/intent-bench-cli.js"
   },
   "exports": {
     "./browser": {
@@ -97,6 +98,7 @@
     "dev:server": "tsx src/cli.ts",
     "simple-bench": "tsx src/simple-bench-cli.ts",
     "agent-run": "tsx src/agent-run-cli.ts",
+    "intent-bench": "tsx src/intent-bench-cli.ts",
     "dev:demo-account": "yarn build && concurrently --kill-others \"yarn dev:server --profile demo-account\" \"yarn dev:web\"",
     "dev:web": "vite --config web/vite.config.ts",
     "start": "node dist/cli.js",