@columenlabs/lumen-ai 0.1.216 → 0.1.218

package/README.md

@@ -99,7 +99,7 @@ app.all('/api/assistant/*', (c) => routes(c.req.raw));
 - **ChatGPT sign-in.** In place of their own OpenAI key a person may sign in with their ChatGPT plan (`POST /my-signin`, then `POST /my-signin/finish {url}`): the sign-in opens in a new tab and ends on `http://127.0.0.1:1455/auth/callback`, a page that does not load, whose address the person pastes back. The tokens are sealed as their own key and refreshed before they run out; answers on the plan count tokens and no dollars.
 - **Who pays.** The person's pick (the shared keys by default) while it has a key for the tier's provider and money left this month under its cap (an admin sets the shared keys' cap per person, `PUT /app`), else the other; with neither the panel says why.
 - **Tools.** Each group is Off, Read, Read + draft or Read + write per person. A tool is a read when its MCP `readOnlyHint` is set. On Read + draft a write only drafts a card, and runs as the person when they press Save, after access is checked again. A group with `readOnly: true` (read tools only) is offered Off or Read alone. Read + write is offered only for a group with `writes: true`: its writes run at once, and with `undo` set each one's line in the chat has Undo, which calls `undo(caller, { tool, args, result })`; the model is told what was undone. Cards drafted one after another show as one stack with Save all. Use `mcpServerTools` (an SDK `McpServer`), `mcpFetchTools` (a server served as a fetch handler, any SDK), `remoteMcpTools` or `plainTools`.
-- **Web.** `fetch` is a plain GET from the Worker (10 s, 2 MB, never the app's own hosts or private addresses). ScrapeQL's own tools join over its MCP API once an admin pastes a Cloudflare Access service token (`PUT /web/scrapeql`). A Worker on ScrapeQL's own Cloudflare account passes its service binding instead, `scrapeql: { url: 'https://scrapeql/mcp', fetch: env.SCRAPEQL.fetch.bind(env.SCRAPEQL) }`: no token is stored, the tools are always on and Settings shows the link with a Test.
+- **Web.** `fetch` is a plain GET from the Worker (10 s, 2 MB, never the app's own hosts or private addresses). ScrapeQL's own tools join over its MCP API once an admin pastes a Cloudflare Access service token (`PUT /web/scrapeql`). A Worker on ScrapeQL's own Cloudflare account passes its service binding instead, `scrapeql: { url: 'https://scrapeql/mcp', fetch: env.SCRAPEQL.fetch.bind(env.SCRAPEQL) }`: no token is stored, the tools are always on and Settings shows the link with a Test. Each web step carries `web` (`StepWeb`: a search's `query` and `results`, or the page it `read`), so the panel folds a search's results under its line and numbers each page read under the answer as Sources chips; the model marks facts with those numbers ([1]).
 - **Hand off and advice.** The `escalate` tool hands one task to a higher tier (the person sets next, any or only when asked); Anthropic's advisor tool lets a higher tier send a plan, at most two an answer.
 - **Chats** run on the agent (`./agent`, `LumenAgent`, below): the app routes `POST /chat` and each chat's reads, cards and Undo to its own agent, and `assistantRoutes` answers `POST /chat` with 410 `moved`. Chats saved by the old turn loop stay readable (`GET /chats`, `GET|DELETE /chats/:id`) until `assistant.purge()` drops them 30 days after their last message.
 - **Screen moves**: while the person's Settings ▸ Screen is Move (the default), the panel sends its page map (`AssistantMessage.screen`: routes, nav pages, Settings sections, cards on screen, rows to open) and `LumenAgent` gives the model a `screen` tool (`./agent` `screen.ts`). Each call's moves (go, settings, chart, open, point) are checked against the map, kept in the chat and streamed as a `screen` event; the panel runs them through `AssistantScreenProvider` (mounted by `LumenShell`) with an Undo chip, and their outcomes go back with the next message (`outcomes`). Nothing is saved by a move.
@@ -241,6 +241,8 @@ Drafts and writes are cards in the chat's own storage (`lumen_cards` in the agen
 
 **The panel.** `LumenAssistant` reads a Think chat with no change. `ask(input, sink)` runs one answer and sends the panel's own events (`AssistantEvent`: the chat, the text, each tool line, cards, hand-off sections, advice, the use) to `sink`. The app's `POST /chat` returns `askResponse((sink) => chat.ask(input, sink), { signal, stop: () => chat.stop() })`, the same server-sent events `assistantRoutes` sends. `view()` gives a stored chat as the panel's items, and `usage()` gives the chat's use so far. A card's id is its tool call's, so the panel's Save, Dismiss and Undo go to `saveCard`, `dismissCard` and `undoCard`. The app keeps the list of chats, and each chat is its own agent, such as a facet of the person's store. `chunkEvents` and `itemsOf` are the mapping, for an app that streams some other way.
 
+**Steps.** One answer makes at most `plan.maxSteps` model calls (`MAX_STEPS`, 8). The last one runs with no tools and `LAST_STEP_NOTE`, so an answer always ends in words: what it found, and what is left when the task did not fit.
+
 **Steer and queue.** While an answer runs, the panel's box stays open: Enter steers the message into the answer (`steer(message)` on the agent; the app routes `POST /chats/:id/steer` `{ message }` to it and answers `{ steered }`), Alt+Enter queues it for after. A steer joins at the answer's next step, after the tool it is on, with `STEER_NOTE` before it for the model; the panel shows it there marked Steered (event `steered`), and `view()` keeps it in place (`SteerNote`, under the message the answer is to). A steer the answer ends before reading comes back as `unsteered` and goes first in the queue. `plan.steer: false` turns it off for a model; an app without the route (404) shows Queue only. Queued messages each go as their own message when the answer before them ends; each has Steer now, Edit and Remove.
 
 ```ts

package/dist/agent/agent.d.ts

@@ -74,7 +74,7 @@ export interface AgentTurnPlan {
     left?: number;
     /** The person lets the assistant move their screen: the `screen` tool joins when the browser sent its map. */
     screen?: boolean;
-    /** Most model calls in one answer; 8. */
+    /** Most model calls in one answer (`MAX_STEPS`, 8); the last one answers in words, with no tools. */
     maxSteps?: number;
     /**
      * A message sent while the answer runs joins it at its next step (Steer); on unless `false`,
@@ -99,6 +99,10 @@ export interface AgentTurnEnd {
     capped: boolean;
 }
 export declare const CAP_NOTE = "The spend cap for this month is reached: answer now from what you have, in a sentence or two. You can call no more tools.";
+/** The last call of an answer: no tools, so the answer never ends on a tool's result with nothing said. */
+export declare const LAST_STEP_NOTE = "This is the answer's last call and you can call no more tools: answer now from what you found. If the task is not finished, say what is done, what is left and that you can carry on when asked.";
+/** Most model calls in one answer when the plan names none. */
+export declare const MAX_STEPS = 8;
 /** Put before a message the person steered into a running answer, for the model only. */
 export declare const STEER_NOTE = "The person sent this while you were working. Take it into account from here on; it does not cancel what you were asked before unless it says so.";
 /** A card in the chat: a draft the person saves or dismisses, or a write made at once (Undo). */

package/dist/agent/agent.js

@@ -24,6 +24,10 @@ import { chunkEvents, itemsOf } from './stream.js';
 import { openAiTools, } from './tools.js';
 import { UsageMeter } from './usage.js';
 export const CAP_NOTE = 'The spend cap for this month is reached: answer now from what you have, in a sentence or two. You can call no more tools.';
+/** The last call of an answer: no tools, so the answer never ends on a tool's result with nothing said. */
+export const LAST_STEP_NOTE = "This is the answer's last call and you can call no more tools: answer now from what you found. If the task is not finished, say what is done, what is left and that you can carry on when asked.";
+/** Most model calls in one answer when the plan names none. */
+export const MAX_STEPS = 8;
 /** Put before a message the person steered into a running answer, for the model only. */
 export const STEER_NOTE = 'The person sent this while you were working. Take it into account from here on; it does not cancel what you were asked before unless it says so.';
 /** Past this many tokens in the chat Think summarizes its older part. */
@@ -596,7 +600,7 @@ export class LumenAgent extends Think {
             tools,
             ...(lazy ? { activeTools: lazy.active() } : {}),
             ...(plan.first ? { toolChoice: { type: 'tool', toolName: plan.first } } : {}),
-            maxSteps: plan.maxSteps ?? 8,
+            maxSteps: plan.maxSteps ?? MAX_STEPS,
             ...turnSettings(plan.model, {
                 effort: plan.effort ?? 'default',
                 ...(plan.out ? { out: plan.out } : {}),
@@ -620,6 +624,17 @@ export class LumenAgent extends Think {
             };
             return capped;
         }
+        // The last call answers in words: past it the loop stops, so a tool call there would end the
+        // answer on a tool's result with nothing said.
+        if (ctx.stepNumber >= (t.plan.maxSteps ?? MAX_STEPS) - 1) {
+            const last = {
+                ...messages,
+                activeTools: [],
+                toolChoice: 'none',
+                instructions: `${t.system}\n\n${LAST_STEP_NOTE}`,
+            };
+            return last;
+        }
         // A forced first call is for the first step only; tools a search found join each step.
         const auto = t.plan.first && ctx.stepNumber > 0;
         if (!auto && !t.lazy && !steered)

package/dist/agent/stream.js

@@ -1,4 +1,5 @@
 import { rowCount } from '../assistant/tools.js';
+import { webStep } from '../assistant/web.js';
 /** A tool's name as a line's title: `notes__find_notes` → `find notes`. */
 export function stepTitle(name) {
     const own = name.includes('__') ? name.slice(name.indexOf('__') + 2) : name;
@@ -51,6 +52,7 @@ function settledCall(id, c, out, ms, cards, screens = new Map()) {
     }
     const error = out.error ?? (o && typeof o.error === 'string' ? o.error : undefined);
     const rows = rowCount(o && 'result' in o ? o.result : out.output);
+    const web = webStep(c.name, c.input, out.output, !!error || obj(out.output)?.ok === false);
     const step = {
         kind: 'step',
         id,
@@ -63,6 +65,7 @@ function settledCall(id, c, out, ms, cards, screens = new Map()) {
         ...(card?.kind === 'write' || (o && typeof o.change === 'string')
             ? { write: { undo: true, status: card?.status === 'undone' ? 'undone' : 'done' } }
             : {}),
+        ...(web ? { web } : {}),
     };
     return [{ event: 'step', data: step }];
 }

package/dist/assistant/engine.js

@@ -775,7 +775,7 @@ export class Assistant {
         if (failed)
             lines.push(`${failed} tool source${failed === 1 ? '' : 's'} could not be reached this turn.`);
         if (fns.web)
-            lines.push('Web: use web__fetch first for APIs and plain pages; the browser tools only for pages that need JavaScript or after a fetch fails.');
+            lines.push('Web: use web__fetch first for APIs and plain pages; the browser tools only for pages that need JavaScript or after a fetch fails. To find pages, search, then read the one to three that answer the question. The person sees each page you read numbered under your answer, in the order you read them (1 for the first page read in this answer): mark a fact you took from a page with its number, like [1], and never cite a page you did not read.');
         if (main) {
             const reach = this.above(main.tier);
             if (reach.length) {

package/dist/assistant/web.d.ts

@@ -4,6 +4,7 @@
  * for pages that need a real browser or after a fetch fails. ScrapeQL sits behind Cloudflare Access,
  * so the app reaches it with one Access service token an admin pastes in Settings.
  */
+import type { StepWeb } from '@columenlabs/lumen-ui-types/assistant';
 import { type ToolSource } from './tools.js';
 export interface WebFetchOptions {
     /** Hosts never fetched (the app's own, behind sign-in), each with its subdomains. */
@@ -46,3 +47,10 @@ export interface AccessToken {
  * binding (`f` is then the binding's fetch). Every one of them reads.
  */
 export declare function scrapeqlTools(url: string, token: AccessToken | null, f?: typeof fetch): ToolSource;
+/**
+ * What a finished web tool call shows the person, or undefined for any other tool: a search's
+ * words and the pages it listed, or the page a read opened (`web__fetch` after redirects; ScrapeQL's
+ * `scrape`, `extract`, `ask` and `screenshot` by the address they were given). A failed read names
+ * the page it tried, never a page read.
+ */
+export declare function webStep(name: string, input: unknown, output: unknown, failed: boolean): StepWeb | undefined;

package/dist/assistant/web.js

@@ -1,9 +1,3 @@
-/**
- * The web: `fetch`, a plain GET from the Worker (no browser, 10 s, 2 MB), tried first for APIs, JSON
- * and plain pages; and ScrapeQL's own MCP tools (`search`, `scrape`, `extract`, `ask`) over its API
- * for pages that need a real browser or after a fetch fails. ScrapeQL sits behind Cloudflare Access,
- * so the app reaches it with one Access service token an admin pastes in Settings.
- */
 import { plainTools, remoteMcpTools } from './tools.js';
 /** Fetches one answer may have in flight at once; more wait their turn. */
 const PARALLEL = 20;
@@ -177,3 +171,68 @@ export function scrapeqlTools(url, token, f) {
         return { ...open, tools: open.tools.map((t) => ({ ...t, read: true })) };
     };
 }
+const LINK = /\[([^\]\n]{0,300})\]\((https?:\/\/[^)\s]{1,2000})\)/g;
+/** Results a search step lists; more are cut. */
+const RESULTS_MAX = 10;
+/** Text a tool's output carries: a plain string, or the `result` string it was wrapped in. */
+function textOf(v) {
+    if (typeof v === 'string')
+        return v;
+    const r = v && typeof v === 'object' ? v.result : undefined;
+    return typeof r === 'string' ? r : '';
+}
+/** A page's title from the HTML a fetch read, when it has one. */
+function htmlTitle(text) {
+    const m = /<title[^>]*>([^<]{1,300})<\/title>/i.exec(text.slice(0, 50_000));
+    const t = m?.[1]?.replace(/\s+/g, ' ').trim();
+    return t || undefined;
+}
+/**
+ * What a finished web tool call shows the person, or undefined for any other tool: a search's
+ * words and the pages it listed, or the page a read opened (`web__fetch` after redirects; ScrapeQL's
+ * `scrape`, `extract`, `ask` and `screenshot` by the address they were given). A failed read names
+ * the page it tried, never a page read.
+ */
+export function webStep(name, input, output, failed) {
+    if (!name.startsWith('web__'))
+        return undefined;
+    const tool = name.slice(5);
+    const args = (input && typeof input === 'object' ? input : {});
+    if (tool === 'search') {
+        const query = typeof args.query === 'string' ? args.query.slice(0, 300) : undefined;
+        const results = [];
+        for (const m of textOf(output).matchAll(LINK)) {
+            if (results.length >= RESULTS_MAX)
+                break;
+            const url = m[2];
+            if (results.some((r) => r.url === url))
+                continue;
+            const title = m[1].trim();
+            results.push(title ? { url, title } : { url });
+        }
+        return { ...(query ? { query } : {}), results };
+    }
+    if (failed) {
+        const url = typeof args.url === 'string' ? args.url : '';
+        return /^https?:\/\//.test(url) ? { tried: { url } } : undefined;
+    }
+    if (tool === 'fetch') {
+        // A long answer reaches the model cut, as text: read its fields from that text.
+        const cut = typeof output === 'string' ? output : '';
+        const o = (output && typeof output === 'object' ? output : {});
+        if (o.ok === false || cut.startsWith('{"ok":false'))
+            return undefined;
+        const said = cut ? /"url":"(https?:\/\/[^"]{1,2000})"/.exec(cut)?.[1] : undefined;
+        const url = typeof o.url === 'string' ? o.url : (said ?? (typeof args.url === 'string' ? args.url : ''));
+        if (!/^https?:\/\//.test(url))
+            return undefined;
+        const body = typeof o.text === 'string' ? o.text : cut;
+        const title = body ? htmlTitle(body) : undefined;
+        return { read: title ? { url, title } : { url } };
+    }
+    if (['scrape', 'extract', 'ask', 'screenshot'].includes(tool)) {
+        const url = typeof args.url === 'string' ? args.url : '';
+        return /^https?:\/\//.test(url) ? { read: { url } } : undefined;
+    }
+    return undefined;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@columenlabs/lumen-ai",
-  "version": "0.1.216",
+  "version": "0.1.218",
   "description": "Lumen's assistant and agent parts: a chat assistant's server (Claude tiers, tools over MCP, the web, hand off, spend caps), serving an MCP server from a Worker, read-only tool sets and metered tool calls.",
   "license": "MIT",
   "type": "module",
@@ -52,8 +52,8 @@
     "@ai-sdk/openai": "4.0.108",
     "@anthropic-ai/sdk": "0.133.0",
     "@cloudflare/think": "0.23.0",
-    "@columenlabs/lumen-ops": "0.1.216",
-    "@columenlabs/lumen-ui-types": "0.1.216",
+    "@columenlabs/lumen-ops": "0.1.218",
+    "@columenlabs/lumen-ui-types": "0.1.218",
     "agents": "0.32.0",
     "ai": "7.0.162",
     "just-bash": "3.6.0",