pi-roundtable 0.9.10 → 0.9.12

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,22 @@ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
5
5
 
6
6
  ## [Unreleased]
7
7
 
8
+ ## [0.9.12] - 2026-10-11
9
+
10
+ ### Added
11
+
12
+ - `generate_image({ prompt, reference_paths? })`: a core session tool, offered to agent and owner sessions only when a plugin fills the `images` provider, that draws a new picture from a description, saves it under `<scratch dir or workspace>/drawings/`, attaches it to the reply, and names the saved path. Up to 4 reference images (PNG, JPEG, WebP or GIF by content, 20 MiB each, 40 MiB together) resolve inside the workspace or the scratch dir like `attach_file` paths. A provider error is a tool error with its message. Owner-only by default; the agents' prompt names it only when offered.
13
+ - `ImageDrawer` takes an optional third argument `{ signal? }`, the asking turn's abort signal; an existing two-argument drawer still fits.
14
+
15
+ ## [0.9.11] - 2026-10-11
16
+
17
+ ### Added
18
+
19
+ - `ToolTurn.client`, `TurnRequest.client`, `ConversationTurnInput.client` and `SessionContext.client()`: what the client that started a turn can do for its tools, as a `ClientPort` (`names`, and `call(name, args, options?)` with `ClientCallOptions`: `{ timeoutMs?, maxValueBytes?, signal? }`). A transport that carries a client able to run functions for a tool, such as the browser extension of a web chat, provides one with `context.turns.run({ ..., client })`; the runtime keeps it with the turn and clears it when the turn ends. `call` never rejects for a transport problem: it resolves `ClientCallResult`, `{ ok: true, value, parts }` with `ClientPart`s (`{ mimeType, data }`, such as a screenshot) or `{ ok: false, error, message }` where `error` is a `ClientCallError`: `no_client`, `refused`, `timeout`, `cancelled`, `disconnected`, `too_large`, `busy` or `failed`. One call is open on a client at a time. The core adds the turn's signal to every call, so stopping the turn cancels a call in flight. A turn whose transport gives none (Discord, a schedule, an agent's turn, the CLI) leaves it undefined, and `SessionContext.client` is optional on the type so a context built before this still type-checks.
20
+ - `ToolOutput` (`string | { text, images? }`): a tool's `run` may return text with images, which reach the model as image parts of the tool result. Each image goes through the same checks as a user's attachment (png, jpeg, gif or webp, size and pixel limits, at most 4, a large one downscaled); one that fails is the tool's error result, not a provider rejection.
21
+ - `ToolTurn.attachments?()` (optional, so a `ToolTurn` built by hand still type-checks; the turns the core builds always have it; `ToolAttachmentInfo`: `{ file, name, contentType, size, addedAt }`): the files someone attached to the conversation, the newest first, only the names `ToolTurn.attachment()` opens; a session without an attachment store lists none.
22
+ - `testPlugin().runTool` takes `client` in its options (a `ClientPort` the test fakes), and `testPlugin().images` records the images a successful `runTool` returned.
23
+
8
24
  ## [0.9.10] - 2026-10-10
9
25
 
10
26
  ### Added
package/docs/plugins.md CHANGED
@@ -586,17 +586,29 @@ The host logs counts only (`files`, `bytes`, `unattributedBytes`), never a file
586
586
  ### `tools`: what agents can call
587
587
 
588
588
  `defineTool` takes a name (lowercase words joined by underscores), a description the model reads to decide when to call it, a Typebox parameter schema, the lowest tier that may call it, and the function.
589
- `run` receives the arguments, already typed, and the turn: the speaker, the channel, the agent, an abort signal, `workspace`, `attachFile`, and `attachment`.
589
+ `run` receives the arguments, already typed, and the turn: the speaker, the channel, the agent, an abort signal, `workspace`, `attachFile`, `attachment`, `attachments`, `origin`, and `client`.
590
590
  `turn.workspace` is `{ workspace, scratchDir? }` in a session with a shared workspace (an agent's), the same roots as `SessionContext.workspace`, and undefined elsewhere: a tool that makes a file can save it there, under the scratch dir when there is one, and name its path, so the model sends it on with `attach_file` or a `discord_send_message` file `path` instead of carrying its bytes.
591
- It returns the text the model reads.
591
+ It returns the text the model reads, or `{ text, images? }` (`ToolOutput`) to show the model images beside it: each image is `{ mimeType, data: Uint8Array }` and reaches the model as an image part of the tool result.
592
592
  `turn.origin` is where the turn's request came from, as the transport that carried it reported it, and undefined when the channel gives none: the web chat sets it to the browser origin its connection opened with (`https://chat.example.com`, `chrome-extension://<id>`), a Discord message, a schedule or an agent's turn leave it undefined.
593
593
  Only a channel's transport sets it, never the message text or the model, so a tool may write it on a record it creates, such as an issue's source line; a tool that needs it fails closed when it is undefined.
594
594
  A claim passes it to `context.turns.run` as `origin`, and the hold of the same call sees it too, since `hold` receives the same `ToolTurn`.
595
595
 
596
596
  `turn.attachment(file)` opens a file someone attached to the conversation, by the name the turn's `## Attachments` block lists, and returns `{ file, name, contentType, size, bytes() }`: a tool passes a user's file on without the model carrying its bytes.
597
597
  Only the conversation's own files open; a path, a hidden name, or a missing file throws a `ToolRefusal` the model reads, and so does a file the host's [retention period](#retention-removing-used-files-after-a-period) removed, which says so.
598
+ `turn.attachments?.()` lists those files, the newest first, as `{ file, name, contentType, size, addedAt }` (`ToolAttachmentInfo`): the names `attachment` opens and nothing else; a session that keeps no attachments lists none. The turns the core builds always have it; it is optional on the type for code that builds a turn by hand.
598
599
  Throw `ToolRefusal` for a call the model should correct; any other error fails the call.
599
600
 
601
+ ##### `turn.client`: asking the party that started the turn
602
+
603
+ Some transports carry a client that can do something for a tool: the web chat's browser extension can read the page the person has open.
604
+ `turn.client` is that client's `ClientPort` when the turn's transport provides one and the client declared it, and undefined otherwise (a Discord message, a schedule, an agent's turn, and a web chat connection that declared nothing have none), so a tool that needs it fails closed with a `ToolRefusal` when it is undefined.
605
+ `client.names` lists the functions the client declared, and `client.call(name, args, { timeoutMs?, maxValueBytes? })` asks it to run one.
606
+ The call never rejects for a transport problem: it resolves `{ ok: true, value, parts }`, with the value the client answered with and its binary `parts` (`{ mimeType, data }`, such as a screenshot), or `{ ok: false, error, message }` where `error` is `no_client`, `refused`, `timeout`, `cancelled`, `disconnected`, `too_large`, `busy`, or `failed`.
607
+ One call is open on a client at a time: a second one made meanwhile resolves `busy`, so make calls one after another.
608
+ The core gives every call the turn's `signal`, so stopping the turn cancels the call in flight (`cancelled`) and the tool need not handle it.
609
+ The value and the parts come from the client, outside the host, so a tool treats them as untrusted data: it checks their shape, and passes images on as `ToolOutput` images rather than trusting their names or types.
610
+ The transport sets `client` and bounds each call (its default timeout, its ceiling and its size limits); a claim passes it to `context.turns.run` as `client`.
611
+
600
612
  <!-- example: examples/tools.ts -->
601
613
  ```ts
602
614
  import { definePlugin, defineTool, ToolRefusal } from "pi-roundtable";
@@ -681,6 +693,8 @@ A Pi package should use the host's peer dependency on `pi-roundtable`, not bundl
681
693
  Transient `SessionContext.runTask` tasks return only text and cannot attach to their parent's reply.
682
694
 
683
695
  The model attaches a file it already has on disk with the core's `attach_file({ path, filename? })`. A session with a workspace (`SessionContext.workspace`, an agent's) registers it; the path must resolve, symlinks followed, inside the workspace or the scratch dir, a relative path resolves in the workspace, and `filename` defaults to the path's base name. It queues the file with `attachReplyFile`, so `REPLY_FILE_LIMITS` apply and a surface without reply files refuses it as a tool error; its result tells the model the file will appear with its reply. It holds nothing, since it only posts in the channel the turn answers in, and its tier defaults to the owner, like any tool the core's table does not name.
696
+
697
+ With an `images` provider, agent and owner sessions also offer the core's `generate_image({ prompt, reference_paths? })`, which draws a new picture from a description (`prompt`, at most 4,000 characters). `reference_paths` lists up to 4 local images, checked like `attach_file` paths (inside the workspace or the scratch dir, symlinks followed, regular files only), each a PNG, JPEG, WebP or GIF by content, at most 20 MiB each and 40 MiB together; a refused reference is a tool error before the provider is called. It passes them to the provider as `ReferenceImage`s with the turn's signal, saves the picture under `<scratch dir or workspace>/drawings/generated-<ISO time>.<ext>`, and attaches it with `attachReplyFile`; when it cannot be attached the file is kept and the result says so. The result names the saved path, never the bytes, so the model can send it on or edit it. A provider error, such as a refusal, is a tool error carrying the provider's message. It holds nothing, and its tier defaults to the owner; `toolTiers` lowers it. The agents' prompt mentions it only when it is offered.
684
698
  The Discord tools that upload, `discord_send_message` and `discord_edit_message`, take each file as exactly one of `path` (a local file, read when the call runs, with the same roots; `filename` defaults to its base name) or `dataBase64` with a `filename`, at most 8 MiB in all. A session without a workspace takes only `dataBase64`, and so does remote MCP, whose schemas stay inline only. A path outside the roots, a missing file, or a directory is a tool error naming the path. A held call's card (`approvalCard(call, workspace?, limits?)`) lists each file it sends by path with the file's size now, never its bytes, and shows an input within `limits.totalChars` (1,500) whole; a longer input has its string values cut, longest first and never below `limits.valueChars` (200), to their start plus `… [N chars]` (N in code points) until it fits, and if it still does not, the rest is dropped and the keys no longer shown are listed. `approvalDetails(call, workspace?)` gives the same call as data (`{ action, tool, input, files? }`, the whole input, files as `{ path, bytes? }`); the gate passes it to `Prompts.confirm` as `details`, for a surface that is not text.
685
699
  Calling the helper outside `context.turns.run` or an agent-team turn, or after that turn finishes, throws `ReplyFileError`; direct standalone runtime calls have no reply collector.
686
700
  Await work that produces files before returning from the tool.
@@ -1267,10 +1281,11 @@ An unknown or misspelled slot stops startup with an error listing the valid name
1267
1281
  | Slot | The core's default | Your replacement |
1268
1282
  |---|---|---|
1269
1283
  | `judge` | Asks the configured model small questions: does this reply approve the held actions, how hard is this turn, which agents does this group message concern | An object with `askYesNo`, `askChoice`, and `askScore` |
1270
- | `images` | No drawing; each agent gets an avatar generated from its display name | `async (prompt, references) => bytes` returning PNG bytes |
1284
+ | `images` | No drawing; each agent gets an avatar generated from its display name, and no session offers `generate_image` | `async (prompt, references, options?) => bytes` returning PNG, JPEG, WebP or GIF bytes; `options.signal` aborts with the turn that asked |
1271
1285
  | `runtime` | None: the runtime plugin builds the Pi runtime itself | `(deps) => runtime`, an [`AgentRuntime`](#the-runtime-slot-replace-pi) that runs every conversation |
1272
1286
 
1273
- When no `images` provider is configured, `agent_create` has no `avatar_prompt` parameter and `agent_avatar` is unavailable.
1287
+ A filled `images` slot draws agents' avatars and also powers the core's [`generate_image`](#attach-files-to-the-agents-reply) tool.
1288
+ When no `images` provider is configured, `agent_create` has no `avatar_prompt` parameter, `agent_avatar` is unavailable, and no session offers `generate_image`.
1274
1289
  The owner's profile panel reports the missing provider and offers no redraw option.
1275
1290
  Each agent gets a picture generated from its display name and the assistant's icon, so you can tell agents apart.
1276
1291
  `bunx roundtable doctor` reports whether the slot is filled.
@@ -1865,6 +1880,7 @@ The factory gets the session's `SessionContext`, fixed when the session is made:
1865
1880
  - `addressee` is whom tool descriptions should name: the primary owner exactly as configured in their own private conversation, another person by their name and pronouns (their name alone when they gave none), and `THE_SPEAKER` in a shared conversation, or a private one of someone the host knows by no name yet.
1866
1881
  - `memory` is `"none"` for a persona that declares it, and `"speaker"` otherwise; a memory tool of your own loads nothing for `"none"`. A task the session runs with `runTask` keeps its conversation's `memory`.
1867
1882
  - `speaker()` is the running turn's speaker, undefined between turns; a tool reads it when it runs, never when the session is made, since a session outlives the turn it is built in.
1883
+ - `client()` is what the client that started the running turn can do for its tools, as `TurnRequest.client` says, and undefined between turns and for a turn whose transport gives none; it is optional on the type so a context built before 0.9.11 still type-checks.
1868
1884
  - `origin()` is where the running turn's request came from, as `TurnRequest.origin` says, and undefined between turns and for a turn whose transport gives none; read it when the tool runs, like `speaker()`. It is optional on the type so a context built before 0.9.10 still type-checks.
1869
1885
 
1870
1886
  The core's Jev compactor is one such extension:
@@ -2017,7 +2033,7 @@ Call `context.turns.run(input)` inside your claim's queue task to run a turn in
2017
2033
  It shows typing and the stop control on the channel's surface, runs the turn, and emits `turnStarted` and `turnEnded` with the turn's `kind`.
2018
2034
  It catches runtime errors as failed results and posts the answer, failure notice, or stopped notice through the surface.
2019
2035
  Supply `reply(result)` to handle the reply yourself.
2020
- `input` has `channel`, `kind`, `text`, `speaker`, and optionally `origin` (where the request came from, as your claim's transport knows it, such as a connection's browser origin; tools read it as `turn.origin`, so never fill it from message text), `attachments`, `selection` (by default the plugins' `agentSelection`), `steerable`, `interactive`, `confirmed`, `reply`, and `conversation`.
2036
+ `input` has `channel`, `kind`, `text`, `speaker`, and optionally `origin` (where the request came from, as your claim's transport knows it, such as a connection's browser origin; tools read it as `turn.origin`, so never fill it from message text), `client` (a `ClientPort` for what the client that started the turn can do, which tools read as `turn.client`; only your transport provides one), `attachments`, `selection` (by default the plugins' `agentSelection`), `steerable`, `interactive`, `confirmed`, `reply`, and `conversation`.
2021
2037
  It rejects with `NotLinkedError` during `setup`, and with a `PluginError` on a host whose runtime plugin has not provided a runtime.
2022
2038
 
2023
2039
  Before each turn runs, `context.turns.run` records its conversation in the host's registry, `CONVERSATIONS`: at the first turn its key, surface, kind, visibility, owner, and title, and at every later turn only that it was active.
@@ -2960,7 +2976,8 @@ It returns:
2960
2976
  | `contribution` | What the plugin added, as the host would collect it: `tools`, `prompt`, `seeds`, `events`, `services`, `http`, and the rest |
2961
2977
  | `tools`, `tiers` | The tool names, and the table that says what tier each needs |
2962
2978
  | `holds` | The plugin's `holdRules` chained as the host links them (`holdChain`): `holds(tool, input, { workspace?, scratchDir? })` returns the description of a call that must be approved first, or `undefined`; `holds.inTurn(tool, input, context, turn)` asks them as a turn does, and returns a promise when a hold looks things up |
2963
- | `runTool(name, args, { speaker, origin, channel, workspace }?)` | Runs a tool the way an agent's turn would, in the channel (default `test:1`) for the speaker, and returns the text the model reads; `origin` is the turn's `turn.origin`, and `workspace` (`{ workspace, scratchDir? }`) is the session's `turn.workspace` |
2979
+ | `runTool(name, args, { speaker, origin, client, channel, workspace }?)` | Runs a tool the way an agent's turn would, in the channel (default `test:1`) for the speaker, and returns the text the model reads; `origin` is the turn's `turn.origin`, `client` its `turn.client` (a `ClientPort` the test fakes), and `workspace` (`{ workspace, scratchDir? }`) is the session's `turn.workspace` |
2980
+ | `images` | The images successful `runTool` calls returned as `ToolOutput` images, recorded as `{ channel, image: { mimeType, data } }` |
2964
2981
  | `files` | Accepted files from `runTool`, recorded as `{ channel, file: ReplyFile }`; inject a surface with `supportsFiles: true` and pass its channel to test attachment tools |
2965
2982
  | `events` | The events the plugin itself reported through `context.events`, and those of `context.turns` |
2966
2983
  | `conversations`, `turns`, `surfaces` | What the plugin sees as `context.conversations`, `context.turns`, and `context.surfaces`, for a test to drive its claims |
@@ -3365,6 +3382,11 @@ Import from the entries listed below; source area files are internal.
3365
3382
  | `ChannelClaim` | `pi-roundtable` | type |
3366
3383
  | `ChannelKey` | `pi-roundtable` | type |
3367
3384
  | `ChatSurface` | `pi-roundtable` | type |
3385
+ | `ClientCallError` | `pi-roundtable` | type |
3386
+ | `ClientCallOptions` | `pi-roundtable` | type |
3387
+ | `ClientCallResult` | `pi-roundtable` | type |
3388
+ | `ClientPart` | `pi-roundtable` | type |
3389
+ | `ClientPort` | `pi-roundtable` | type |
3368
3390
  | `ConfigError` | `pi-roundtable` | value |
3369
3391
  | `ContextUse` | `pi-roundtable` | type |
3370
3392
  | `Contribution` | `pi-roundtable` | type |
@@ -3527,7 +3549,9 @@ Import from the entries listed below; source area files are internal.
3527
3549
  | `Tier` | `pi-roundtable` | type |
3528
3550
  | `TierConfig` | `pi-roundtable` | type |
3529
3551
  | `ToolAttachment` | `pi-roundtable` | type |
3552
+ | `ToolAttachmentInfo` | `pi-roundtable` | type |
3530
3553
  | `ToolContribution` | `pi-roundtable` | type |
3554
+ | `ToolOutput` | `pi-roundtable` | type |
3531
3555
  | `ToolRefusal` | `pi-roundtable` | value |
3532
3556
  | `ToolSelection` | `pi-roundtable` | type |
3533
3557
  | `ToolSpec` | `pi-roundtable` | type |
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pi-roundtable",
3
- "version": "0.9.10",
3
+ "version": "0.9.12",
4
4
  "description": "A plugin-driven Pi agent server for one owner, reached through Discord or the web",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -25,6 +25,8 @@ export function agentSystemPrompt(input: {
25
25
  speaker?: Speaker;
26
26
  /** Whether agents can redraw avatars; false without an image provider. Default true. */
27
27
  avatars?: boolean;
28
+ /** Whether the session offers generate_image; true only with an image provider. Default false. */
29
+ images?: boolean;
28
30
  }): string {
29
31
  const { agent, coordinator, shared, workDir, owner, shellUser, group } =
30
32
  input;
@@ -37,7 +39,7 @@ export function agentSystemPrompt(input: {
37
39
  `You are "${agent.displayName}" (agent name \`${agent.name}\`), one of ${o.name}'s agents in ${o.his} Discord agent server. Each agent owns one channel and one conversation; you all share ${o.his} tools and ${o.his} memory.`,
38
40
  roleText(agent, coordinator, who),
39
41
  `The team: agent_list shows every agent and group; agent_get and agent_update read and improve any agent's prompt, display name, model, and thinking level, yours included; agent_create adds an agent with its own channel; ${input.avatars === false ? "" : "agent_avatar redraws an avatar; "}schedule_list with agent reads another agent's schedules. A message marked as coming from another agent is that agent speaking, not ${w.name}; only ${w.name} approves held actions.`,
40
- `Your shell and file tools run on ${o.his} VPS as the user ${shellUser}, in the shared workspace ${workDir}.${input.scratchDir ? ` Write temporary files under the scratch dir ${input.scratchDir} ($TMPDIR), not elsewhere in /tmp.` : ""} Commands that are destructive or change the system, and writes outside the workspace${input.scratchDir ? " and the scratch dir" : ""}, are held for ${w.his} confirmation: tell ${w.him} exactly what will run and ask ${w.him} to confirm. To show a file you have, attach it to your reply with attach_file or give its path to discord_send_message; never read an image or base64 a file to send it.`,
42
+ `Your shell and file tools run on ${o.his} VPS as the user ${shellUser}, in the shared workspace ${workDir}.${input.scratchDir ? ` Write temporary files under the scratch dir ${input.scratchDir} ($TMPDIR), not elsewhere in /tmp.` : ""} Commands that are destructive or change the system, and writes outside the workspace${input.scratchDir ? " and the scratch dir" : ""}, are held for ${w.his} confirmation: tell ${w.him} exactly what will run and ask ${w.him} to confirm. To show a file you have, attach it to your reply with attach_file or give its path to discord_send_message; never read an image or base64 a file to send it.${input.images ? " To make a new picture, describe it to generate_image, which saves it and attaches it to your reply." : ""}`,
41
43
  ];
42
44
  if (group) {
43
45
  const others = group.members
@@ -149,6 +149,7 @@ export class DiscordAgentTeam implements AgentOps, AgentTeam {
149
149
  shellUser,
150
150
  ...(scratchDir ? { scratchDir } : {}),
151
151
  avatars: this.#avatars(),
152
+ images: this.#avatars(),
152
153
  ...(group
153
154
  ? {
154
155
  group: {
@@ -1,5 +1,6 @@
1
+ import { stat } from "node:fs/promises";
1
2
  import { join } from "node:path";
2
- import { RECORDS } from "./attachment-dir.ts";
3
+ import { listDir, RECORDS } from "./attachment-dir.ts";
3
4
  import { expiredMessage, wasExpired } from "./attachment-expiry.ts";
4
5
  import { isAttachmentName, plainText } from "./attachment-name.ts";
5
6
 
@@ -15,6 +16,18 @@ export interface ToolAttachment {
15
16
  bytes(): Promise<Uint8Array>;
16
17
  }
17
18
 
19
+ /** A file someone attached to the conversation, as a tool lists it. */
20
+ export interface ToolAttachmentInfo {
21
+ /** The file name inside the conversation's attachment store; what `## Attachments` lists. */
22
+ file: string;
23
+ /** The name the person gave it, when the core recorded one; else the file name. */
24
+ name: string;
25
+ contentType: string;
26
+ size: number;
27
+ /** When the file was saved, as an ISO time. */
28
+ addedAt: string;
29
+ }
30
+
18
31
  /** A name that is no attachment of the conversation, or a file that is not there. */
19
32
  export class AttachmentLookupError extends Error {
20
33
  override name = "AttachmentLookupError";
@@ -72,3 +85,32 @@ export async function openAttachment(
72
85
  },
73
86
  };
74
87
  }
88
+
89
+ /**
90
+ * The files of a conversation's attachment store that a tool may open, the newest first. Only
91
+ * names an attachment can have are listed: no hidden entry, no directory. A directory that is not
92
+ * there lists nothing.
93
+ */
94
+ export async function listAttachments(
95
+ dir: string,
96
+ ): Promise<ToolAttachmentInfo[]> {
97
+ const listed: ToolAttachmentInfo[] = [];
98
+ for (const file of await listDir(dir)) {
99
+ if (!isAttachmentName(file)) continue;
100
+ const info = await stat(join(dir, file)).catch(() => undefined);
101
+ if (!info?.isFile()) continue;
102
+ const opened = await openAttachment(dir, file).catch(() => undefined);
103
+ if (!opened) continue;
104
+ listed.push({
105
+ file,
106
+ name: opened.name,
107
+ contentType: opened.contentType,
108
+ size: opened.size,
109
+ addedAt: info.mtime.toISOString(),
110
+ });
111
+ }
112
+ return listed.sort(
113
+ (a, b) =>
114
+ b.addedAt.localeCompare(a.addedAt) || a.file.localeCompare(b.file),
115
+ );
116
+ }
@@ -56,10 +56,14 @@ export interface ReferenceImage {
56
56
  mimeType: string;
57
57
  }
58
58
 
59
- /** Draws one image from a prompt and reference images; returns PNG or other decodable bytes. */
59
+ /**
60
+ * Draws one image from a prompt and reference images; returns PNG or other decodable bytes.
61
+ * `options.signal`, when given, aborts with the turn that asked; a drawer may ignore it.
62
+ */
60
63
  export type ImageDrawer = (
61
64
  prompt: string,
62
65
  references: readonly ReferenceImage[],
66
+ options?: { signal?: AbortSignal },
63
67
  ) => Promise<Uint8Array>;
64
68
 
65
69
  /** The replaceable parts the core runs on; a plugin provides at most the ones it replaces. */
@@ -1,12 +1,20 @@
1
1
  import type { Static, TObject } from "typebox";
2
- import type { ToolAttachment } from "./attachments/tool-attachment.ts";
2
+ import type {
3
+ ToolAttachment,
4
+ ToolAttachmentInfo,
5
+ } from "./attachments/tool-attachment.ts";
6
+ import type { ClientPort } from "./domain/client.ts";
3
7
  import type { ReplyFile } from "./domain/conversation.ts";
4
8
  import { PluginError } from "./errors.ts";
5
9
  import { type HoldRule, isPromise } from "./holds.ts";
6
10
  import { messages } from "./i18n/index.ts";
7
11
  import { type RoundtablePlugin, refuseRemovedFields } from "./plugin.ts";
8
12
  import type { AgentTurnScope, ChannelKey, SessionTool } from "./sessions.ts";
9
- import { toolError, toolText } from "./shared/tool-result.ts";
13
+ import {
14
+ type ToolOutput,
15
+ toolError,
16
+ toolOutput,
17
+ } from "./shared/tool-result.ts";
10
18
  import { type Speaker, TIERS, type Tier } from "./speakers.ts";
11
19
  import { toolTurn } from "./tool-turn.ts";
12
20
 
@@ -34,6 +42,14 @@ export interface ToolTurn {
34
42
  * when it is undefined.
35
43
  */
36
44
  origin?: string;
45
+ /**
46
+ * What the client that started the turn can do for this tool, such as run a function in the
47
+ * browser the person chats from; undefined for a turn whose transport gives none, such as a
48
+ * Discord message or a schedule, or whose client declared nothing. Every call carries this turn's
49
+ * `signal`, so stopping the turn cancels a call in flight. A tool that needs one fails closed
50
+ * when it is undefined.
51
+ */
52
+ client?: ClientPort;
37
53
  /**
38
54
  * The session's shared workspace and scratch dir, where a tool may save a file for the model to
39
55
  * pass on by path; undefined for a session without a workspace.
@@ -48,6 +64,12 @@ export interface ToolTurn {
48
64
  * refuses every call.
49
65
  */
50
66
  attachment(file: string): Promise<ToolAttachment>;
67
+ /**
68
+ * The files someone attached to this conversation, the newest first; only names that
69
+ * `attachment` opens. A session without an attachment store lists none. The turns the core builds
70
+ * always have it; code that builds a turn by hand may leave it out.
71
+ */
72
+ attachments?(): Promise<readonly ToolAttachmentInfo[]>;
51
73
  }
52
74
 
53
75
  /**
@@ -85,9 +107,11 @@ export interface ToolSpec<Schema extends TObject> {
85
107
  args: Static<Schema>,
86
108
  turn: ToolTurn,
87
109
  ) => string | undefined | Promise<string | undefined>;
88
- run(args: Static<Schema>, turn: ToolTurn): Promise<string> | string;
110
+ run(args: Static<Schema>, turn: ToolTurn): Promise<ToolOutput> | ToolOutput;
89
111
  }
90
112
 
113
+ export type { ToolAttachmentInfo, ToolOutput };
114
+
91
115
  /** A tool a plugin adds, as `defineTool` builds it. */
92
116
  export interface ToolContribution {
93
117
  readonly name: string;
@@ -142,7 +166,7 @@ export function defineTool<Schema extends TObject>(
142
166
  parameters: spec.parameters,
143
167
  execute: async (_toolCallId, params, signal) => {
144
168
  try {
145
- return toolText(
169
+ return toolOutput(
146
170
  await spec.run(
147
171
  params as Static<Schema>,
148
172
  toolTurn(context, signal),
@@ -0,0 +1,61 @@
1
+ /** A binary piece of a client call's result, such as a screenshot. */
2
+ export interface ClientPart {
3
+ mimeType: string;
4
+ data: Uint8Array;
5
+ }
6
+
7
+ /** Why a client call did not return a value. */
8
+ export type ClientCallError =
9
+ /** The turn has no client, or its client is gone. */
10
+ | "no_client"
11
+ /** The client declined the call. */
12
+ | "refused"
13
+ /** The client did not answer in time. */
14
+ | "timeout"
15
+ /** The turn was stopped, or the caller's signal aborted, before the client answered. */
16
+ | "cancelled"
17
+ /** The client's connection closed before it answered. */
18
+ | "disconnected"
19
+ /** The client's answer passed the size the transport accepts. */
20
+ | "too_large"
21
+ /** A call is already open on this client; make calls one at a time. */
22
+ | "busy"
23
+ /** The client tried and failed, or answered with something that is not a result. */
24
+ | "failed";
25
+
26
+ /** What a client call comes to: the value the client answered with and its binary parts, or why not. */
27
+ export type ClientCallResult =
28
+ | { ok: true; value: unknown; parts: readonly ClientPart[] }
29
+ | { ok: false; error: ClientCallError; message: string };
30
+
31
+ /** How one client call is bounded. */
32
+ export interface ClientCallOptions {
33
+ /** How long to wait for the client's answer; the transport has a default and its own ceiling. */
34
+ timeoutMs?: number;
35
+ /** The most bytes of JSON the answer's value may take; the transport has a default and its own ceiling. */
36
+ maxValueBytes?: number;
37
+ /**
38
+ * Cancels the call when it aborts: the client is told, and the call resolves `cancelled`. A tool
39
+ * need not pass the turn's signal; the core adds it to every call.
40
+ */
41
+ signal?: AbortSignal;
42
+ }
43
+
44
+ /**
45
+ * What the client that started a turn can do for its tools, such as a browser extension running a
46
+ * function on the page the person has open. Only the transport that carries the turn provides one.
47
+ */
48
+ export interface ClientPort {
49
+ /** The names of the functions the client declared it can run. */
50
+ readonly names: readonly string[];
51
+ /**
52
+ * Asks the client to run `name` with `args`. It never rejects for a transport problem: a failure
53
+ * resolves `{ ok: false }`. One call is open at a time; another call meanwhile resolves `busy`.
54
+ * The value and the parts come from the client, so a tool treats them as untrusted data.
55
+ */
56
+ call(
57
+ name: string,
58
+ args: unknown,
59
+ options?: ClientCallOptions,
60
+ ): Promise<ClientCallResult>;
61
+ }
@@ -1,6 +1,7 @@
1
1
  import type { AgentTurnScope, TurnSelection } from "../sessions.ts";
2
2
  import type { Speaker } from "../speakers.ts";
3
3
  import type { TurnAttachments } from "./attachment.ts";
4
+ import type { ClientPort } from "./client.ts";
4
5
  import type { ChannelKey } from "./conversation.ts";
5
6
  import type { InterimPosts } from "./interim.ts";
6
7
  import type { TurnProgress } from "./progress.ts";
@@ -37,6 +38,12 @@ export interface TurnRequest {
37
38
  * as `ToolTurn.origin`.
38
39
  */
39
40
  origin?: string;
41
+ /**
42
+ * What the client that started the turn can do for its tools, such as run a function in a
43
+ * browser; a tool reads it as `ToolTurn.client`. A channel whose transport has no such client
44
+ * leaves it out. Only a channel's transport provides one, never the message text or the model.
45
+ */
46
+ client?: ClientPort;
40
47
  /**
41
48
  * Who the conversation belongs to, as the host records it: `private` to one principal, or
42
49
  * `shared` by whoever its claim admits, which it is taken to be when absent. A private
@@ -5,6 +5,7 @@ import type {
5
5
  ConversationVisibility,
6
6
  } from "../conversations/conversation-registry.ts";
7
7
  import type { TurnAttachments } from "../domain/attachment.ts";
8
+ import type { ClientPort } from "../domain/client.ts";
8
9
  import type { TurnResult } from "../domain/conversation.ts";
9
10
  import type { TurnConversation } from "../domain/ports.ts";
10
11
  import type { TurnProgress } from "../domain/progress.ts";
@@ -35,6 +36,11 @@ export interface ConversationTurnInput {
35
36
  * transport gives none, and never fill it from message text.
36
37
  */
37
38
  origin?: string;
39
+ /**
40
+ * What the client that started the turn can do for its tools, as the claim's transport knows it;
41
+ * tools read it as `ToolTurn.client`. Leave it out for a turn the transport gives none.
42
+ */
43
+ client?: ClientPort;
38
44
  /** Files and images the message carries. */
39
45
  attachments?: TurnAttachments;
40
46
  /** The turn's tools; by default the plugins' `agentSelection`, read at the turn. */
@@ -191,6 +197,7 @@ export function conversationTurns(
191
197
  text: input.text,
192
198
  speaker,
193
199
  ...(input.origin === undefined ? {} : { origin: input.origin }),
200
+ ...(input.client === undefined ? {} : { client: input.client }),
194
201
  ...(conversation ? { conversation } : {}),
195
202
  ...(input.attachments
196
203
  ? { attachments: input.attachments }
@@ -24,7 +24,6 @@ import { promptScopeOf } from "../interactions/prompts.ts";
24
24
  import { MEMORY_TOOLS } from "../modules/memory/owner-memory.ts";
25
25
  import { withoutReplyFiles } from "../reply-files.ts";
26
26
  import type { TransientTask } from "../sessions.ts";
27
- import type { Speaker, Tier } from "../speakers.ts";
28
27
  import { type ToolTiers, toolsForTier, toolTiers } from "../tool-tiers.ts";
29
28
  import { AbortGuard } from "./abort-guard.ts";
30
29
  import { bridgeTurnRefusal, chosenAgentModel } from "./bridge-guard.ts";
@@ -36,6 +35,7 @@ import { interimPoster } from "./interim-text.ts";
36
35
  import { preflightTools } from "./preflight-tools.ts";
37
36
  import { PromptSlot, workTimeout } from "./prompt-slot.ts";
38
37
  import { carriesMemory, recordMemoryTurn } from "./reader-history.ts";
38
+ import type { RunningTurn } from "./running-turn.ts";
39
39
  import type { PiAgentRuntimeOptions, TurnMessage } from "./runtime-types.ts";
40
40
  import { archiveSessions } from "./session-archive.ts";
41
41
  import {
@@ -71,11 +71,8 @@ export class PiAgentRuntime implements AgentRuntime {
71
71
  readonly #tiers: ToolTiers;
72
72
  readonly #factory: SessionFactory;
73
73
  readonly #sessions: ConversationSessions;
74
- /** The tier, speaker and origin of each conversation's running turn. */
75
- readonly #turns = new Map<
76
- ChannelKey,
77
- { tier: Tier; speaker: Speaker; origin?: string; withheld: boolean }
78
- >();
74
+ /** The tier, speaker, origin and client of each conversation's running turn. */
75
+ readonly #turns = new Map<ChannelKey, RunningTurn>();
79
76
  /** Each conversation's turn in progress, which the owner may steer or stop. */
80
77
  readonly #running = new Map<ChannelKey, SteerableRun>();
81
78
  readonly #memoryTurns = new Map<ChannelKey, () => void>();
@@ -88,6 +85,7 @@ export class PiAgentRuntime implements AgentRuntime {
88
85
  this.#factory = new SessionFactory(options, {
89
86
  speaker: (channel) => this.#turns.get(channel)?.speaker,
90
87
  origin: (channel) => this.#turns.get(channel)?.origin,
88
+ client: (channel) => this.#turns.get(channel)?.client,
91
89
  withholdsMemory: (channel) => this.#turns.get(channel)?.withheld === true,
92
90
  runTask: (scope, task) => this.#runTask(scope, task),
93
91
  });
@@ -197,6 +195,7 @@ export class PiAgentRuntime implements AgentRuntime {
197
195
  tier,
198
196
  speaker: request.speaker,
199
197
  ...(request.origin === undefined ? {} : { origin: request.origin }),
198
+ ...(request.client === undefined ? {} : { client: request.client }),
200
199
  withheld,
201
200
  });
202
201
  const missing = wanted.filter((name) => !registered.has(name));
@@ -0,0 +1,14 @@
1
+ import type { ClientPort } from "../domain/client.ts";
2
+ import type { Speaker, Tier } from "../speakers.ts";
3
+
4
+ /** What the runtime keeps about a conversation's running turn, for its tools to read. */
5
+ export interface RunningTurn {
6
+ tier: Tier;
7
+ speaker: Speaker;
8
+ /** Where the turn's request came from, as `TurnRequest.origin` says. */
9
+ origin?: string;
10
+ /** What the client that started the turn can do, as `TurnRequest.client` says. */
11
+ client?: ClientPort;
12
+ /** Whether the turn loads no memory. */
13
+ withheld: boolean;
14
+ }
@@ -168,6 +168,9 @@ export function runtimePlugin(
168
168
  }),
169
169
  confirmations: heldActions,
170
170
  toolTiers: context.toolTiers,
171
+ ...(providers.filled.has("images")
172
+ ? { images: providers.images }
173
+ : {}),
171
174
  stopping: () => context.queue.closed,
172
175
  get agents() {
173
176
  return agents();
@@ -4,6 +4,7 @@ import type {
4
4
  ExtensionFactory,
5
5
  ModelRuntime,
6
6
  } from "@earendil-works/pi-coding-agent";
7
+ import type { ImageDrawer } from "../contract/providers.ts";
7
8
  import type { AgentSessions, LoadedSkill } from "../contract/runtime.ts";
8
9
  import type { ChannelKey } from "../domain/conversation.ts";
9
10
  import type { InterimTextMode } from "../domain/interim.ts";
@@ -97,6 +98,8 @@ export interface PiAgentRuntimeOptions {
97
98
  interimPrimaryChars?: number;
98
99
  /** How long a new session waits for its MCP tools to register. */
99
100
  mcpConnectTimeoutMs?: number;
101
+ /** The host's image provider, when one fills the slot; sessions then offer `generate_image`. */
102
+ images?: ImageDrawer;
100
103
  /** Defaults to true: bridge shared turns refuse raw private exchanges inaccessible to the reader. */
101
104
  memory?: boolean;
102
105
  }
@@ -113,6 +116,8 @@ export interface CoreExtensions {
113
116
  readAttachment: ExtensionFactory;
114
117
  /** `attach_file`, for a session with a workspace. */
115
118
  attachFile?: ExtensionFactory;
119
+ /** `generate_image`, for a host with an image provider. */
120
+ generateImage?: ExtensionFactory;
116
121
  confirmationGate: ExtensionFactory;
117
122
  askUser: ExtensionFactory;
118
123
  selfCompactGuard: ExtensionFactory;
@@ -144,6 +149,9 @@ export function sessionExtensions(
144
149
  ...(core.attachFile
145
150
  ? [{ name: "attach-file", factory: core.attachFile }]
146
151
  : []),
152
+ ...(core.generateImage
153
+ ? [{ name: "generate-image", factory: core.generateImage }]
154
+ : []),
147
155
  { name: "confirmation-gate", factory: core.confirmationGate },
148
156
  { name: "ask-user", factory: core.askUser },
149
157
  { name: "self-compact-guard", factory: core.selfCompactGuard },
@@ -10,6 +10,8 @@ import {
10
10
  type ToolDefinition,
11
11
  } from "@earendil-works/pi-coding-agent";
12
12
  import { channelSegment } from "../attachments/attachment-dir.ts";
13
+ import type { ImageDrawer } from "../contract/providers.ts";
14
+ import type { ClientPort } from "../domain/client.ts";
13
15
  import type { ChannelKey } from "../domain/conversation.ts";
14
16
  import { ConfigError } from "../domain/errors.ts";
15
17
  import type { AgentTurnScope } from "../domain/ports.ts";
@@ -27,8 +29,13 @@ import {
27
29
  ATTACH_FILE_TOOL,
28
30
  attachFileExtension,
29
31
  } from "../shared/attach-file-tool.ts";
32
+ import {
33
+ GENERATE_IMAGE_TOOL,
34
+ generateImageExtension,
35
+ } from "../shared/generate-image-tool.ts";
30
36
  import { packageDir } from "../shared/package-dir.ts";
31
37
  import { readAttachmentExtension } from "../shared/read-attachment-tool.ts";
38
+ import type { WorkspaceRoots } from "../shared/workspace-files.ts";
32
39
  import { type Speaker, THE_SPEAKER } from "../speakers.ts";
33
40
  import { currentToolNames } from "../tool-tiers.ts";
34
41
  import { toolTurn } from "../tool-turn.ts";
@@ -55,6 +62,7 @@ import type { PromptSlot } from "./prompt-slot.ts";
55
62
  import type {
56
63
  ChannelSession,
57
64
  CompactionEnd,
65
+ CoreExtensions,
58
66
  PiAgentRuntimeOptions,
59
67
  } from "./runtime-types.ts";
60
68
  import {
@@ -75,6 +83,8 @@ export interface SessionFactoryDeps {
75
83
  speaker(channel: ChannelKey): Speaker | undefined;
76
84
  /** Where the conversation's running turn came from, as its transport reported it. */
77
85
  origin(channel: ChannelKey): string | undefined;
86
+ /** What the client that started the conversation's running turn can do for its tools. */
87
+ client(channel: ChannelKey): ClientPort | undefined;
78
88
  /** Whether the conversation's running turn loads no memory, as the agent server's option keeps a speaker out of it. */
79
89
  withholdsMemory(channel: ChannelKey): boolean;
80
90
  /**
@@ -139,6 +149,8 @@ export class SessionFactory {
139
149
  ASK_USER_TOOL,
140
150
  // Registered only by a session with a workspace.
141
151
  ATTACH_FILE_TOOL,
152
+ // Registered only with an image provider.
153
+ GENERATE_IMAGE_TOOL,
142
154
  ...groups
143
155
  .filter((group) => selection.groups.includes(group.name))
144
156
  .flatMap((group) => group.tools),
@@ -358,6 +370,7 @@ export class SessionFactory {
358
370
  },
359
371
  speaker: () => this.#deps.speaker(turnKey),
360
372
  origin: () => this.#deps.origin(turnKey),
373
+ client: () => this.#deps.client(turnKey),
361
374
  runTask: (task) =>
362
375
  this.#deps.runTask({ turn: turnKey, home: channel }, task),
363
376
  };
@@ -377,6 +390,10 @@ export class SessionFactory {
377
390
  ...(context.workspace
378
391
  ? { attachFile: attachFileExtension(context.workspace) }
379
392
  : {}),
393
+ ...imageTools(
394
+ this.#options.images,
395
+ context.workspace ?? { workspace: cwd },
396
+ ),
380
397
  confirmationGate: confirmationGateExtension(gate, slot, (signal) =>
381
398
  toolTurn(context, signal),
382
399
  ),
@@ -438,6 +455,14 @@ export class SessionFactory {
438
455
  }
439
456
  }
440
457
 
458
+ /** `generate_image` for a session of a host with an image provider; nothing without one. */
459
+ export function imageTools(
460
+ images: ImageDrawer | undefined,
461
+ roots: WorkspaceRoots,
462
+ ): Pick<CoreExtensions, "generateImage"> {
463
+ return images ? { generateImage: generateImageExtension(images, roots) } : {};
464
+ }
465
+
441
466
  /** Pi's bash tool with TMPDIR set to `scratchDir`, so `mktemp` and tools write there. */
442
467
  export function scratchBash(cwd: string, scratchDir: string): ToolDefinition {
443
468
  // SAFETY: Pi's bash definition is a ToolDefinition typed by its own parameter schema, which the session takes as any custom tool.
@@ -1,4 +1,5 @@
1
1
  import type { ExtensionFactory } from "@earendil-works/pi-coding-agent";
2
+ import type { ClientPort } from "./domain/client.ts";
2
3
  import { PluginError } from "./errors.ts";
3
4
  import type { Principal } from "./identity/principal-store.ts";
4
5
  import type { OwnerIdentity } from "./identity.ts";
@@ -98,6 +99,12 @@ export interface SessionContext {
98
99
  * method, which reads as undefined.
99
100
  */
100
101
  origin?(): string | undefined;
102
+ /**
103
+ * What the client that started the running turn can do for its tools, as `TurnRequest.client`
104
+ * says; undefined between turns and for a turn whose transport gives none. A context built
105
+ * before 0.9.11 has no such method, which reads as undefined.
106
+ */
107
+ client?(): ClientPort | undefined;
101
108
  /** Runs a task beside this session, under its confirmation gate. */
102
109
  runTask(task: TransientTask): Promise<string>;
103
110
  }
@@ -0,0 +1,173 @@
1
+ import { mkdir, writeFile } from "node:fs/promises";
2
+ import { join } from "node:path";
3
+ import type { ExtensionFactory } from "@earendil-works/pi-coding-agent";
4
+ import { Type } from "typebox";
5
+ import type { ImageDrawer, ReferenceImage } from "../contract/providers.ts";
6
+ import {
7
+ attachReplyFile,
8
+ REPLY_FILE_LIMITS,
9
+ ReplyFileError,
10
+ } from "../reply-files.ts";
11
+ import { toolError, toolText } from "./tool-result.ts";
12
+ import {
13
+ findWorkspaceFile,
14
+ readWorkspaceFile,
15
+ WorkspaceFileError,
16
+ type WorkspaceRoots,
17
+ } from "./workspace-files.ts";
18
+
19
+ export const GENERATE_IMAGE_TOOL = "generate_image";
20
+
21
+ /** The limits on the reference images one call passes to the drawer. */
22
+ export const REFERENCE_IMAGE_LIMITS = Object.freeze({
23
+ maxImages: 4,
24
+ maxImageBytes: 20 * 1024 * 1024,
25
+ maxTotalBytes: 40 * 1024 * 1024,
26
+ });
27
+
28
+ const MAX_PROMPT_CHARS = 4000;
29
+
30
+ /** The image type of `data` by its content, or undefined for anything else. */
31
+ export function imageTypeOf(
32
+ data: Uint8Array,
33
+ ): { mimeType: string; ext: string } | undefined {
34
+ const starts = (bytes: readonly number[], at = 0) =>
35
+ bytes.every((byte, i) => data[at + i] === byte);
36
+ if (starts([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]))
37
+ return { mimeType: "image/png", ext: "png" };
38
+ if (starts([0xff, 0xd8, 0xff])) return { mimeType: "image/jpeg", ext: "jpg" };
39
+ if (starts([0x47, 0x49, 0x46, 0x38]))
40
+ return { mimeType: "image/gif", ext: "gif" };
41
+ if (starts([0x52, 0x49, 0x46, 0x46]) && starts([0x57, 0x45, 0x42, 0x50], 8))
42
+ return { mimeType: "image/webp", ext: "webp" };
43
+ return undefined;
44
+ }
45
+
46
+ async function readReferences(
47
+ paths: readonly string[],
48
+ roots: WorkspaceRoots,
49
+ ): Promise<ReferenceImage[]> {
50
+ const { maxImages, maxImageBytes, maxTotalBytes } = REFERENCE_IMAGE_LIMITS;
51
+ if (paths.length > maxImages)
52
+ throw new WorkspaceFileError(
53
+ `${paths.length} reference images given; at most ${maxImages} are allowed.`,
54
+ );
55
+ const files = await Promise.all(
56
+ paths.map((path) => findWorkspaceFile(path, roots)),
57
+ );
58
+ let total = 0;
59
+ for (const file of files) {
60
+ if (file.size > maxImageBytes)
61
+ throw new WorkspaceFileError(
62
+ `${file.path} is ${file.size} bytes; a reference image may contain at most ${maxImageBytes} bytes.`,
63
+ );
64
+ total += file.size;
65
+ }
66
+ if (total > maxTotalBytes)
67
+ throw new WorkspaceFileError(
68
+ `the reference images total ${total} bytes; at most ${maxTotalBytes} are allowed.`,
69
+ );
70
+ const references: ReferenceImage[] = [];
71
+ for (const file of files) {
72
+ const data = await readWorkspaceFile(file, maxImageBytes);
73
+ const type = imageTypeOf(data);
74
+ if (!type)
75
+ throw new WorkspaceFileError(
76
+ `${file.path} is not a PNG, JPEG, WebP or GIF image.`,
77
+ );
78
+ references.push({
79
+ data: Buffer.from(data).toString("base64"),
80
+ mimeType: type.mimeType,
81
+ });
82
+ }
83
+ return references;
84
+ }
85
+
86
+ /**
87
+ * Registers `generate_image`, which draws a new picture with the host's image provider, saves it
88
+ * under `<scratch dir or workspace>/drawings`, and attaches it to the turn's reply.
89
+ */
90
+ export function generateImageExtension(
91
+ draw: ImageDrawer,
92
+ roots: WorkspaceRoots,
93
+ ): ExtensionFactory {
94
+ return (pi) => {
95
+ pi.registerTool({
96
+ name: GENERATE_IMAGE_TOOL,
97
+ label: "Generate image",
98
+ description:
99
+ 'Generate a new picture from a description. Describe the subject, style and composition; say "no text" when you will add text to it later. The picture is saved to a file whose path the result names, and attached to your reply.',
100
+ parameters: Type.Object(
101
+ {
102
+ prompt: Type.String({
103
+ minLength: 1,
104
+ maxLength: MAX_PROMPT_CHARS,
105
+ description:
106
+ "What to draw: subject, style, composition, and anything to leave out.",
107
+ }),
108
+ reference_paths: Type.Optional(
109
+ Type.Array(Type.String({ minLength: 1, maxLength: 4096 }), {
110
+ maxItems: REFERENCE_IMAGE_LIMITS.maxImages,
111
+ description:
112
+ "Paths of PNG, JPEG, WebP or GIF images inside the workspace or the scratch dir to draw from.",
113
+ }),
114
+ ),
115
+ },
116
+ { additionalProperties: false },
117
+ ),
118
+ execute: async (_toolCallId, params, signal) => {
119
+ let references: ReferenceImage[];
120
+ try {
121
+ references = await readReferences(
122
+ params.reference_paths ?? [],
123
+ roots,
124
+ );
125
+ } catch (error) {
126
+ if (error instanceof WorkspaceFileError)
127
+ return toolError(`Not drawn: ${error.message}`);
128
+ throw error;
129
+ }
130
+ signal?.throwIfAborted();
131
+ let data: Uint8Array;
132
+ try {
133
+ data = await draw(
134
+ params.prompt,
135
+ references,
136
+ signal ? { signal } : {},
137
+ );
138
+ } catch (error) {
139
+ signal?.throwIfAborted();
140
+ return toolError(
141
+ `The image provider did not draw it: ${error instanceof Error ? error.message : String(error)}`,
142
+ );
143
+ }
144
+ signal?.throwIfAborted();
145
+ const type = imageTypeOf(data);
146
+ if (!type)
147
+ return toolError(
148
+ "The image provider returned something that is not a PNG, JPEG, WebP or GIF image.",
149
+ );
150
+ const dir = join(roots.scratchDir ?? roots.workspace, "drawings");
151
+ await mkdir(dir, { recursive: true });
152
+ const name = `generated-${new Date().toISOString().replace(/[:.]/g, "-")}.${type.ext}`;
153
+ const path = join(dir, name);
154
+ await writeFile(path, data);
155
+ try {
156
+ if (data.byteLength > REPLY_FILE_LIMITS.maxFileBytes)
157
+ throw new ReplyFileError(
158
+ `it is ${data.byteLength} bytes; a reply file may contain at most ${REPLY_FILE_LIMITS.maxFileBytes} bytes.`,
159
+ );
160
+ attachReplyFile({ name, data });
161
+ } catch (error) {
162
+ if (!(error instanceof ReplyFileError)) throw error;
163
+ return toolText(
164
+ `Drew the picture and saved it to ${path} (${data.byteLength} bytes), but it is not attached: ${error.message}`,
165
+ );
166
+ }
167
+ return toolText(
168
+ `Drew the picture, saved it to ${path} (${data.byteLength} bytes); it will appear with your reply.`,
169
+ );
170
+ },
171
+ });
172
+ };
173
+ }
@@ -1,8 +1,56 @@
1
+ import {
2
+ isModelImage,
3
+ MAX_IMAGES_PER_TURN,
4
+ prepareImageBytes,
5
+ } from "../attachments/image-prep.ts";
6
+ import { ImagePreparationError } from "../attachments/image-preparation-error.ts";
7
+
1
8
  /** The text a tool returns, in the shape Pi expects. */
2
9
  export function toolText(text: string) {
3
10
  return { content: [{ type: "text" as const, text }], details: {} };
4
11
  }
5
12
 
13
+ /** A tool's output: text, or text with images the model reads beside it. */
14
+ export type ToolOutput =
15
+ | string
16
+ | {
17
+ text: string;
18
+ images?: readonly { mimeType: string; data: Uint8Array }[];
19
+ };
20
+
21
+ /** `output` in the shape Pi expects: a text part, then one image part for each image. */
22
+ export async function toolOutput(output: ToolOutput) {
23
+ if (typeof output === "string") return toolText(output);
24
+ const images = output.images ?? [];
25
+ if (images.length > MAX_IMAGES_PER_TURN)
26
+ return toolError(
27
+ `the tool returned ${images.length} images; at most ${MAX_IMAGES_PER_TURN} fit in one result.`,
28
+ );
29
+ const parts: { type: "image"; data: string; mimeType: string }[] = [];
30
+ for (const [index, image] of images.entries()) {
31
+ const label = `image ${index + 1} could not be prepared`;
32
+ const mimeType = image.mimeType.split(";")[0]?.trim().toLowerCase() ?? "";
33
+ if (!isModelImage({ contentType: mimeType }))
34
+ return toolError(
35
+ `${label}: ${image.mimeType} is not an image type the model reads (png, jpeg, gif or webp).`,
36
+ );
37
+ try {
38
+ const ready = await prepareImageBytes(image.data, mimeType);
39
+ parts.push({ type: "image", ...ready });
40
+ } catch (error) {
41
+ if (error instanceof ImagePreparationError)
42
+ return toolError(`${label}: ${error.message}`);
43
+ return toolError(
44
+ `${label}: the bytes are not a readable ${mimeType} image.`,
45
+ );
46
+ }
47
+ }
48
+ return {
49
+ content: [{ type: "text" as const, text: output.text }, ...parts],
50
+ details: {},
51
+ };
52
+ }
53
+
6
54
  /** A refused call's reason, which the model reads as the tool's error result. */
7
55
  export function toolError(text: string) {
8
56
  return { ...toolText(text), isError: true };
@@ -1,12 +1,34 @@
1
1
  import {
2
2
  AttachmentLookupError,
3
+ listAttachments,
3
4
  openAttachment,
4
5
  } from "./attachments/tool-attachment.ts";
5
6
  import type { ToolTurn } from "./define.ts";
6
7
  import { ToolRefusal } from "./define.ts";
8
+ import type { ClientPort } from "./domain/client.ts";
7
9
  import { attachReplyFile } from "./reply-files.ts";
8
10
  import type { SessionContext } from "./sessions.ts";
9
11
 
12
+ /**
13
+ * `port` as a tool sees it in a turn: every call carries the turn's `signal` beside the caller's,
14
+ * so stopping the turn cancels the call in flight.
15
+ */
16
+ function clientIn(port: ClientPort, turn: AbortSignal | undefined): ClientPort {
17
+ return {
18
+ names: port.names,
19
+ call: (name, args, options) => {
20
+ const signals = [options?.signal, turn].filter(
21
+ (signal): signal is AbortSignal => signal !== undefined,
22
+ );
23
+ const signal = signals.length > 1 ? AbortSignal.any(signals) : signals[0];
24
+ return port.call(name, args, {
25
+ ...options,
26
+ ...(signal ? { signal } : {}),
27
+ });
28
+ },
29
+ };
30
+ }
31
+
10
32
  /**
11
33
  * The turn a tool's `run` and `hold` receive, built from the session that makes the call and the
12
34
  * signal that stops it.
@@ -16,14 +38,20 @@ export function toolTurn(
16
38
  signal: AbortSignal | undefined,
17
39
  ): ToolTurn {
18
40
  const origin = context.origin?.();
41
+ const client = context.client?.();
19
42
  return {
20
43
  speaker: context.speaker(),
21
44
  channel: context.turnChannel,
22
45
  agent: context.agent,
23
46
  signal,
24
47
  ...(origin === undefined ? {} : { origin }),
48
+ ...(client === undefined ? {} : { client: clientIn(client, signal) }),
25
49
  ...(context.workspace ? { workspace: context.workspace } : {}),
26
50
  attachFile: attachReplyFile,
51
+ attachments: async () =>
52
+ context.attachmentDir === undefined
53
+ ? []
54
+ : listAttachments(context.attachmentDir),
27
55
  attachment: async (file) => {
28
56
  if (context.attachmentDir === undefined)
29
57
  throw new ToolRefusal("this session keeps no attachments");
package/src/index.ts CHANGED
@@ -17,7 +17,10 @@ export type {
17
17
  TeamStatus,
18
18
  } from "./core/agents/team-status.ts";
19
19
  export type { AttachmentRetention } from "./core/attachments/attachment-retention.ts";
20
- export type { ToolAttachment } from "./core/attachments/tool-attachment.ts";
20
+ export type {
21
+ ToolAttachment,
22
+ ToolAttachmentInfo,
23
+ } from "./core/attachments/tool-attachment.ts";
21
24
  export {
22
25
  AGENT_SERVER_PLUGIN,
23
26
  AGENT_TEAM_SERVICE,
@@ -86,7 +89,12 @@ export type {
86
89
  } from "./core/conversations/conversation-registry.ts";
87
90
  export type { Migration, MigrationReport } from "./core/db/migrations.ts";
88
91
  export { migrateDatabase } from "./core/db/migrations.ts";
89
- export type { ToolContribution, ToolSpec, ToolTurn } from "./core/define.ts";
92
+ export type {
93
+ ToolContribution,
94
+ ToolOutput,
95
+ ToolSpec,
96
+ ToolTurn,
97
+ } from "./core/define.ts";
90
98
  export { definePlugin, defineTool, ToolRefusal } from "./core/define.ts";
91
99
  export type {
92
100
  DefinedRoundtable,
@@ -100,6 +108,13 @@ export type {
100
108
  TurnAttachments,
101
109
  } from "./core/domain/attachment.ts";
102
110
  export { NO_ATTACHMENTS } from "./core/domain/attachment.ts";
111
+ export type {
112
+ ClientCallError,
113
+ ClientCallOptions,
114
+ ClientCallResult,
115
+ ClientPart,
116
+ ClientPort,
117
+ } from "./core/domain/client.ts";
103
118
  export type {
104
119
  HeldCall,
105
120
  OutboundReply,
package/src/testing.ts CHANGED
@@ -30,6 +30,7 @@ import {
30
30
  commandGuard,
31
31
  ownerRootCommand,
32
32
  } from "./core/discord/owner-command.ts";
33
+ import type { ClientPart, ClientPort } from "./core/domain/client.ts";
33
34
  import type {
34
35
  PendingConfirmation,
35
36
  ReplyFile,
@@ -264,6 +265,8 @@ export interface TestPluginResult {
264
265
  events: RecordedEvent[];
265
266
  /** Files accepted by successful runTool calls, copied and recorded with their channel. */
266
267
  files: { channel: ChannelKey; file: ReplyFile }[];
268
+ /** Images returned by successful runTool calls, with their channel. */
269
+ images: { channel: ChannelKey; image: ClientPart }[];
267
270
  runTool(
268
271
  name: string,
269
272
  args: Record<string, unknown>,
@@ -272,6 +275,8 @@ export interface TestPluginResult {
272
275
  speaker?: Speaker;
273
276
  /** Where the turn's request came from, as `ToolTurn.origin` shows it to the tool. */
274
277
  origin?: string;
278
+ /** What the party that started the turn can do for the tool, as `ToolTurn.client` shows it. */
279
+ client?: ClientPort;
275
280
  channel?: ChannelKey;
276
281
  workspace?: SessionContext["workspace"];
277
282
  },
@@ -606,6 +611,7 @@ export async function testPlugin(
606
611
  await runtime?.preflight?.();
607
612
  let stopped = false;
608
613
  const files: { channel: ChannelKey; file: ReplyFile }[] = [];
614
+ const images: { channel: ChannelKey; image: ClientPart }[] = [];
609
615
  return {
610
616
  contribution,
611
617
  holds: linked.holds,
@@ -617,6 +623,7 @@ export async function testPlugin(
617
623
  tiers,
618
624
  events,
619
625
  files,
626
+ images,
620
627
  async runTool(name, args, runOptions) {
621
628
  const tool = registry.tools.find((item) => item.name === name);
622
629
  if (!tool) throw new PluginError(`tool ${name} is not registered`);
@@ -626,7 +633,12 @@ export async function testPlugin(
626
633
  params: Record<string, unknown>,
627
634
  signal?: AbortSignal,
628
635
  ): Promise<{
629
- content: readonly { type: string; text?: string }[];
636
+ content: readonly {
637
+ type: string;
638
+ text?: string;
639
+ data?: string;
640
+ mimeType?: string;
641
+ }[];
630
642
  isError?: boolean;
631
643
  }>;
632
644
  }[] = [];
@@ -643,6 +655,7 @@ export async function testPlugin(
643
655
  attachmentDir: ownerAttachmentDir(attachmentDir, channel),
644
656
  speaker: () => runOptions?.speaker,
645
657
  origin: () => runOptions?.origin,
658
+ client: () => runOptions?.client,
646
659
  runTask: async () => {
647
660
  throw new Error("test session cannot run tasks");
648
661
  },
@@ -661,9 +674,21 @@ export async function testPlugin(
661
674
  surfaces.of(channel)?.supportsFiles === true,
662
675
  async () => {
663
676
  const output = await execute("test-call", args);
677
+ for (const item of output.content)
678
+ if (item.type === "image" && item.data && item.mimeType)
679
+ images.push({
680
+ channel,
681
+ image: {
682
+ mimeType: item.mimeType,
683
+ data: new Uint8Array(Buffer.from(item.data, "base64")),
684
+ },
685
+ });
664
686
  return {
665
687
  ok: true,
666
- text: output.content.map((item) => item.text ?? "").join("\n"),
688
+ text: output.content
689
+ .filter((item) => item.type === "text")
690
+ .map((item) => item.text ?? "")
691
+ .join("\n"),
667
692
  };
668
693
  },
669
694
  );