@noetaris/harness-types 0.3.0 → 0.5.0

package/README.md

@@ -6,19 +6,51 @@ This package defines the common TypeScript interfaces that all LLM adapter packa
 
 ## Overview
 
-`@noetaris/harness-types` is a types-only package with zero runtime code and zero dependencies. It provides the canonical definitions for messages, tools, and the `LLM` interface — the surface that all adapters (Anthropic, OpenAI, Gemini, etc.) must satisfy.
+`@noetaris/harness-types` is a types package with zero dependencies and one small runtime helper, `toBlocks()`. It provides the canonical definitions for messages, tools, and the `LLM` interface — the surface that all adapters (Anthropic, OpenAI, Gemini, etc.) must satisfy.
 
 ## Types
 
 | Type | Description |
 |------|-------------|
-| `Message` | Discriminated union: `{ role: 'user'; content: string }` \| `{ role: 'assistant'; content?: string; toolCalls?: ToolCall[] }` \| `{ role: 'tool'; toolCallId: string; content: string }` |
+| `Message` | Discriminated union: `{ role: 'user'; content: string \| ContentBlock[] }` \| `{ role: 'assistant'; content?: string; toolCalls?: ToolCall[] }` \| `{ role: 'tool'; toolCallId: string; content: string }` |
+| `TextBlock` | `{ type: 'text'; text: string }` — a run of text inside a user message |
+| `ImageBlock` | `{ type: 'image'; data: string; mediaType: string }` — an image inside a user message; `data` is raw base64 (see [Image input](#image-input)) |
+| `ContentBlock` | `TextBlock \| ImageBlock` |
 | `Tool` | `{ name: string; description: string; inputSchema: Record<string, unknown> }` |
 | `ToolCall` | `{ id: string; name: string; input: unknown }` |
-| `LLMResponse` | `{ text: string; toolCalls: ToolCall[]; stopReason: 'end' \| 'tool_use' \| 'max_tokens' }` |
-| `LLMUsageEvent` | `{ tokens: { input: number; output: number }; modelId: string; stopReason: LLMResponse['stopReason'] }` |
+| `LLMResponse` | `{ text: string; toolCalls: ToolCall[]; stopReason: 'end' \| 'tool_use' \| 'max_tokens'; usage: { inputTokens: number; outputTokens: number; contextWindowSize?: number } }` — `usage` is always present; all adapters must populate it |
+| `LLMRequestEvent` | `"llm.request"` payload: `{ modelId: string; providerName: string; messages?: unknown; tools?: unknown }` (`messages`/`tools` opt-in) |
+| `LLMUsageEvent` | `"llm.response"` payload: `{ tokens: { input: number; output: number }; modelId: string; stopReason: LLMResponse['stopReason']; providerName: string; output?: unknown; contextWindowSize?: number }` |
+| `ToolCallEvent` | `"tool.call"` payload: `{ toolName: string; toolCallId: string; input?: unknown }` (`input` opt-in) |
+| `ToolResultEvent` | `"tool.result"` payload: `{ toolName: string; toolCallId: string; durationMs: number; result?: unknown; error?: unknown }` (`result`/`error` opt-in) |
 | `LLM` | `{ invoke(messages: Message[], options?: { tools?: Tool[] }): Promise<LLMResponse> }` |
 
+The one runtime export is `toBlocks(content: string | ContentBlock[]): ContentBlock[]`. It turns a string into a single text block and returns an array as it is.
+
+## Image input
+
+A `user` message can carry images next to text. Pass an array of blocks instead of a string:
+
+```ts
+import type { Message } from '@noetaris/harness-types'
+
+const message: Message = {
+  role: 'user',
+  content: [
+    { type: 'text', text: 'What is in this picture?' },
+    { type: 'image', data: pngBytes.toString('base64'), mediaType: 'image/png' },
+  ],
+}
+```
+
+- `data` is raw base64 with no `data:` prefix. Adapters do not strip or check it.
+- Only base64 is supported. URL sources are not supported yet.
+- Only the `user` arm can carry images. `assistant` and `tool` messages stay text.
+- `content: []` is allowed by the type, but providers may reject it. Don't send an empty array.
+- `mediaType` is a plain string. Anthropic, OpenAI and Ollama check it and throw on types they do not accept. Gemini passes any value through and the provider decides. The accepted types differ per adapter. See the README of your adapter: [Anthropic](https://github.com/noetaris-lab/harness-anthropic), [OpenAI](https://github.com/noetaris-lab/harness-openai), [Gemini](https://github.com/noetaris-lab/harness-google), [Ollama](https://github.com/noetaris-lab/harness-ollama).
+- The model must be able to read images. The adapters do not check this.
+- Large images in session state are expensive to store. See [Large media in session state](https://github.com/noetaris-lab/harness/blob/main/LIMITATIONS.md).
+
 ## Installation
 
 ```sh
@@ -39,6 +71,22 @@ class MyAdapter implements LLM {
 }
 ```
 
+An adapter that handles image input can use `toBlocks()` to treat a string and an array the same way:
+
+```ts
+import { toBlocks } from '@noetaris/harness-types'
+
+for (const block of toBlocks(message.content)) {
+  if (block.type === 'text') {
+    // block.text
+  } else {
+    // block.data (base64) and block.mediaType
+  }
+}
+```
+
+`toBlocks` is a value import, not a type import. An adapter that calls it needs `@noetaris/harness-types` installed at runtime, not only at build time.
+
 ## Related Packages
 
 - [`@noetaris/harness`](https://github.com/noetaris-lab/harness) — core execution engine

package/dist/index.d.ts

@@ -3,16 +3,35 @@
  *
  * @packageDocumentation
  */
+/** A run of plain text inside a user message. */
+type TextBlock = {
+    type: 'text';
+    text: string;
+};
+/**
+ * An image inside a user message.
+ *
+ * `data` is raw base64 with no `data:` prefix. `mediaType` is the MIME type
+ * (e.g. `'image/png'`); Anthropic, OpenAI and Ollama throw on types they do not accept; Gemini passes any value through.
+ */
+type ImageBlock = {
+    type: 'image';
+    data: string;
+    mediaType: string;
+};
+/** One piece of user-message content. */
+type ContentBlock = TextBlock | ImageBlock;
 /**
  * A single turn in a conversation.
  *
- * - `user` — text input from the human side.
+ * - `user` — input from the human side: a string, or blocks mixing text and images.
+ *   An empty block array is allowed by the type but may be rejected by providers; don't send one.
  * - `assistant` — model reply, optionally carrying tool-call requests.
  * - `tool` — result returned to the model after a tool call.
  */
 type Message = {
     role: 'user';
-    content: string;
+    content: string | ContentBlock[];
 } | {
     role: 'assistant';
     content?: string;
@@ -22,6 +41,13 @@ type Message = {
     toolCallId: string;
     content: string;
 };
+/**
+ * Normalize user-message content to blocks.
+ *
+ * A string becomes one text block (an empty string stays one empty text block).
+ * An array is returned as-is, same reference, with no validation.
+ */
+declare function toBlocks(content: string | ContentBlock[]): ContentBlock[];
 /**
  * A tool the LLM may invoke during a conversation turn.
  *
@@ -49,11 +75,24 @@ interface ToolCall {
  *
  * `text` may be empty when `stopReason` is `'tool_use'`.
  * `toolCalls` is empty when `stopReason` is `'end'` or `'max_tokens'`.
+ * `usage` is always present — all adapters must populate it.
  */
 interface LLMResponse {
     text: string;
     toolCalls: ToolCall[];
     stopReason: 'end' | 'tool_use' | 'max_tokens';
+    usage: {
+        /** Number of input tokens consumed by this call. */
+        inputTokens: number;
+        /** Number of output tokens generated by this call. */
+        outputTokens: number;
+        /**
+         * The model's total context window capacity in tokens.
+         * Present only when the adapter can determine it.
+         * Absent when the adapter cannot determine capacity.
+         */
+        contextWindowSize?: number;
+    };
 }
 /**
  * Payload emitted by LLM adapters as an `"llm.request"` observer event,
@@ -70,11 +109,13 @@ interface LLMRequestEvent {
     tools?: unknown;
 }
 /**
- * Token-usage payload emitted by LLM adapters as an `llm.response` observer event.
+ * Token-usage payload emitted by LLM adapters as an `"llm.response"` observer event.
  *
  * Adapters that cannot report real token counts (e.g. mock adapters) emit `{ input: 0, output: 0 }`.
  *
  * `output` is opt-in — it may contain sensitive data; adapters default to omitting it.
+ * `contextWindowSize` mirrors `LLMResponse.usage.contextWindowSize` — present when the
+ * adapter can determine it; absent otherwise.
  */
 interface LLMUsageEvent {
     tokens: {
@@ -85,6 +126,7 @@ interface LLMUsageEvent {
     stopReason: LLMResponse['stopReason'];
     providerName: string;
     output?: unknown;
+    contextWindowSize?: number;
 }
 /**
  * Payload emitted on `"tool.call"` before a tool is dispatched.
@@ -129,4 +171,4 @@ interface LLM {
     }): Promise<LLMResponse>;
 }
 
-export type { LLM, LLMRequestEvent, LLMResponse, LLMUsageEvent, Message, Tool, ToolCall, ToolCallEvent, ToolResultEvent };
+export { type ContentBlock, type ImageBlock, type LLM, type LLMRequestEvent, type LLMResponse, type LLMUsageEvent, type Message, type TextBlock, type Tool, type ToolCall, type ToolCallEvent, type ToolResultEvent, toBlocks };

package/dist/index.js

@@ -1 +1,8 @@
+// src/index.ts
+function toBlocks(content) {
+  return typeof content === "string" ? [{ type: "text", text: content }] : content;
+}
+export {
+  toBlocks
+};
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}
+{"version":3,"sources":["../src/index.ts"],"sourcesContent":["/**\n * Shared LLM protocol types used by all harness LLM adapters.\n *\n * @packageDocumentation\n */\n\n/** A run of plain text inside a user message. */\nexport type TextBlock = { type: 'text'; text: string }\n\n/**\n * An image inside a user message.\n *\n * `data` is raw base64 with no `data:` prefix. `mediaType` is the MIME type\n * (e.g. `'image/png'`); Anthropic, OpenAI and Ollama throw on types they do not accept; Gemini passes any value through.\n */\nexport type ImageBlock = { type: 'image'; data: string; mediaType: string }\n\n/** One piece of user-message content. */\nexport type ContentBlock = TextBlock | ImageBlock\n\n/**\n * A single turn in a conversation.\n *\n * - `user` — input from the human side: a string, or blocks mixing text and images.\n *   An empty block array is allowed by the type but may be rejected by providers; don't send one.\n * - `assistant` — model reply, optionally carrying tool-call requests.\n * - `tool` — result returned to the model after a tool call.\n */\nexport type Message =\n  | { role: 'user'; content: string | ContentBlock[] }\n  | { role: 'assistant'; content?: string; toolCalls?: ToolCall[] }\n  | { role: 'tool'; toolCallId: string; content: string }\n\n/**\n * Normalize user-message content to blocks.\n *\n * A string becomes one text block (an empty string stays one empty text block).\n * An array is returned as-is, same reference, with no validation.\n */\nexport function toBlocks(content: string | ContentBlock[]): ContentBlock[] {\n  return typeof content === 'string' ? [{ type: 'text', text: content }] : content\n}\n\n/**\n * A tool the LLM may invoke during a conversation turn.\n *\n * `inputSchema` must be a valid JSON Schema object describing the tool's\n * input parameters (LLM adapters pass it to the provider as-is).\n */\nexport interface Tool {\n  name: string\n  description: string\n  inputSchema: Record<string, unknown>\n}\n\n/**\n * A single tool-call request emitted by the LLM in an assistant turn.\n *\n * `id` is provider-assigned and must be echoed back in the corresponding\n * `role: 'tool'` message so the model can correlate results.\n */\nexport interface ToolCall {\n  id: string\n  name: string\n  input: unknown\n}\n\n/**\n * Normalized response returned by every {@link LLM} adapter.\n *\n * `text` may be empty when `stopReason` is `'tool_use'`.\n * `toolCalls` is empty when `stopReason` is `'end'` or `'max_tokens'`.\n * `usage` is always present — all adapters must populate it.\n */\nexport interface LLMResponse {\n  text: string\n  toolCalls: ToolCall[]\n  stopReason: 'end' | 'tool_use' | 'max_tokens'\n  usage: {\n    /** Number of input tokens consumed by this call. */\n    inputTokens: number\n    /** Number of output tokens generated by this call. */\n    outputTokens: number\n    /**\n     * The model's total context window capacity in tokens.\n     * Present only when the adapter can determine it.\n     * Absent when the adapter cannot determine capacity.\n     */\n    contextWindowSize?: number\n  }\n}\n\n/**\n * Payload emitted by LLM adapters as an `\"llm.request\"` observer event,\n * immediately before the provider call is made.\n *\n * `messages` and `tools` are opt-in — they may contain sensitive data and are\n * typed as `unknown` because adapters pass their internal translated\n * representations; the framework never inspects this content.\n */\nexport interface LLMRequestEvent {\n  modelId:      string\n  providerName: string\n  messages?:    unknown   // opt-in; adapter's internal translated representation\n  tools?:       unknown   // opt-in; adapter's internal translated representation\n}\n\n/**\n * Token-usage payload emitted by LLM adapters as an `\"llm.response\"` observer event.\n *\n * Adapters that cannot report real token counts (e.g. mock adapters) emit `{ input: 0, output: 0 }`.\n *\n * `output` is opt-in — it may contain sensitive data; adapters default to omitting it.\n * `contextWindowSize` mirrors `LLMResponse.usage.contextWindowSize` — present when the\n * adapter can determine it; absent otherwise.\n */\nexport interface LLMUsageEvent {\n  tokens:            { input: number; output: number }\n  modelId:           string\n  stopReason:        LLMResponse['stopReason']\n  providerName:      string\n  output?:           unknown   // opt-in; normalized LLMResponse or raw provider output; may be sensitive\n  contextWindowSize?: number\n}\n\n/**\n * Payload emitted on `\"tool.call\"` before a tool is dispatched.\n *\n * `input` is opt-in — it may contain sensitive data; omit by default.\n */\nexport interface ToolCallEvent {\n  toolName:   string\n  toolCallId: string\n  input?:     unknown\n}\n\n/**\n * Payload emitted on `\"tool.result\"` after a tool completes or throws.\n *\n * `result` and `error` are opt-in — they may contain sensitive data; omit by default.\n * Set `error` when the tool threw; leave it absent on success.\n */\nexport interface ToolResultEvent {\n  toolName:   string\n  toolCallId: string\n  durationMs: number\n  result?:    unknown\n  error?:     unknown\n}\n\n/**\n * Core interface every LLM adapter must implement.\n *\n * @example\n * ```ts\n * const response = await llm.invoke(messages, { tools: [myTool] })\n * ```\n */\nexport interface LLM {\n  /**\n   * Send a message list to the LLM and return a normalized response.\n   *\n   * @param messages - Conversation history, oldest first.\n   * @param options.tools - Optional tools the model may invoke.\n   */\n  invoke(messages: Message[], options?: { tools?: Tool[] }): Promise<LLMResponse>\n}\n"],"mappings":";AAuCO,SAAS,SAAS,SAAkD;AACzE,SAAO,OAAO,YAAY,WAAW,CAAC,EAAE,MAAM,QAAQ,MAAM,QAAQ,CAAC,IAAI;AAC3E;","names":[]}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@noetaris/harness-types",
-  "version": "0.3.0",
+  "version": "0.5.0",
   "description": "Shared LLM type contract for the @noetaris/harness ecosystem",
   "type": "module",
   "repository": {