@arnilo/prism 0.0.3 → 0.0.4

package/docs/mcp-tools.md

@@ -0,0 +1,139 @@
+# MCP client bridge
+
+## What it does
+
+`@arnilo/prism-mcp` connects Prism hosts to remote [Model Context Protocol](https://modelcontextprotocol.io) servers and maps discovered tools to ordinary `ToolDefinition`s. Transports are stdio subprocesses and Streamable HTTP. The package wraps the official MCP TypeScript SDK (`@modelcontextprotocol/sdk` v1.29+) and does **not** add MCP-specific branches to core Prism.
+
+Primary API:
+
+```ts
+import { connectMcpTools } from "@arnilo/prism-mcp";
+
+const bridge = await connectMcpTools({
+  serverId: "fs",
+  transport: { type: "stdio", command: "node", args: ["server.js"] },
+});
+
+// bridge.tools are ToolDefinition[] — register with createToolRegistry / createAgent
+await bridge.refresh(); // re-list after notifications or TTL expiry
+await bridge.close();   // close client + transport
+```
+
+Advanced hosts that manage their own `Client` + `Transport` can call `attachMcpToolBridge(client, transport, options)` after `client.connect(transport)`.
+
+## When to use it
+
+- **Integrate external MCP tool servers** (filesystem, databases, SaaS adapters) without reimplementing JSON-RPC transports in your app.
+- **Keep core dispatch gates** — register returned tools and let `dispatchToolCall` enforce permission, JSON Schema validation (`ToolValidator`), middleware, abort, and parallel execution (Plan 055 Tasks 1–2).
+- **Explicit lifecycle** — connect, refresh on `notifications/tools/list_changed`, and `close()` when the session ends.
+
+Do **not** use this package as a sandbox, permission engine, or auto-discovery loader. Hosts must trust configured commands/URLs and gate registration.
+
+## Inputs / request
+
+`connectMcpTools()` requires a stable `serverId` plus an explicit stdio or Streamable HTTP transport. Optional bounds control list caching, call timeout, result bytes, name prefix, and abort behavior; defaults are listed below.
+
+## Outputs / response / events
+
+The resolved `McpToolBridge` exposes `tools`, `refresh()`, and `close()`. Each discovered MCP tool becomes a normal Prism `ToolDefinition`; calls return `ToolResult`, with remote `isError` mapped to `ToolResult.error`. List-change notifications invalidate the cache but register nothing automatically.
+
+## Request/response example
+
+```json
+{
+  "request": { "serverId": "docs", "transport": { "type": "stdio", "command": "node", "args": ["server.js"] } },
+  "mappedTool": { "name": "mcp:docs:search", "parameters": { "type": "object" } }
+}
+```
+
+## Implementation example
+
+```ts
+import { createToolRegistry } from "@arnilo/prism";
+import { connectMcpTools } from "@arnilo/prism-mcp";
+
+const bridge = await connectMcpTools({
+  serverId: "docs",
+  transport: { type: "stdio", command: "node", args: ["server.js"] },
+  callTimeoutMs: 30_000,
+});
+const registry = createToolRegistry({ duplicate: "error" });
+for (const tool of bridge.tools) registry.register(tool);
+```
+
+## Tool naming and mapping
+
+| MCP | Prism |
+| --- | --- |
+| `tools/list` `inputSchema` | `ToolDefinition.parameters` |
+| `tools/call` arguments | Parsed `ToolCallContent.arguments` |
+| `tools/call` content blocks | `ToolResult.content` (`text`, `image`; resource/audio/link → descriptive `text`) |
+| Tool `name` | Prefixed `mcp:<serverId>:<name>` (override with `namePrefix`) |
+| `isError` results | `ToolResult.error` with summarized text |
+| `structuredContent` | `ToolResult.value` / metadata |
+
+Duplicate prefixed names throw `McpToolNameCollisionError` at refresh time.
+
+## Extension and configuration notes
+
+| Option | Default | Purpose |
+| --- | --- | --- |
+| `serverId` | required | Stable identifier used in default name prefix |
+| `transport` | required | `stdio` or `streamable-http` config |
+| `namePrefix` | `mcp:<serverId>:` | Registry namespace for remote tools |
+| `listCacheTtlMs` | `30000` | Skip re-listing until TTL expires (invalidated on list-changed) |
+| `callTimeoutMs` | `60000` | Per-call MCP request timeout |
+| `maxResultBytes` | `10000000` | Bound mapped result content |
+| `signal` | none | Abort connect and trigger close on abort |
+
+### Stdio transport
+
+```ts
+{
+  type: "stdio",
+  command: "node",
+  args: ["path/to/server.js"],
+  env?: Record<string, string>,
+  cwd?: string,
+  stderr?: "inherit" | "pipe" | "ignore" | "overlapped",
+}
+```
+
+The host explicitly chooses the executable, arguments, environment, and working directory. Prism does not search `PATH` for unknown servers or inject credentials.
+
+### Streamable HTTP transport
+
+```ts
+{
+  type: "streamable-http",
+  url: "https://mcp.example.com/mcp",
+  requestInit?: RequestInit,
+  sessionId?: string,
+}
+```
+
+Only `http:` and `https:` URLs are accepted. Authentication (Bearer tokens, cookies) is supplied through `requestInit.headers` by the host.
+
+## Security and performance notes
+
+| Risk | Mitigation |
+| --- | --- |
+| Untrusted subprocess (stdio) | Explicit `command` / `args` / `env` / `cwd`; review before deploy |
+| SSRF / open redirects (HTTP) | Host URL allow-lists, network policy, no implicit discovery |
+| Tool-name shadowing | Prefixed names + `createToolRegistry({ duplicate: "error" })` |
+| Oversized server output | `maxResultBytes` on content mapping |
+| Unvalidated arguments | Register tools with `createJsonSchemaToolArgumentValidator()` at dispatch |
+| Missing permission gate | `PermissionPolicy` on `tool:mcp:<serverId>:<name>:execute` (or broader deny rules) |
+
+MCP output is untrusted. Apply `SecretRedactor` and host logging policy to `ToolResult` before persisting or displaying.
+
+## Related APIs
+
+- [Tools](tools.md): registry, dispatch, validation
+- [Tool execution primitives](tool-execution-primitives.md): Plan 055 design and conformance matrix
+- [Host security guide](host-security.md): permission, trust, validation checklist
+- Package README: [`@arnilo/prism-mcp`](../packages/mcp/README.md)
+
+## Testing
+
+Package tests use in-memory MCP transports (no network). Hosts should integration-test their configured stdio commands and HTTP endpoints in staging before production registration.

package/docs/middleware-hooks.md

@@ -101,6 +101,7 @@ export const extension: Extension = {
 - `retry` middleware may stop retrying or adjust delay, but runtime still owns retry event emission, abort-aware waiting, and provider-turn boundaries.
 - The registry does not discover packages, read manifests, load config, call providers, execute tools, read resources, or start sessions.
 - Hosts may pass a middleware registry into `createExtensionKernel({ middleware })` to share it with direct host code.
+- For OpenTelemetry export, prefer `session.subscribe()` + `@arnilo/prism-observability-opentelemetry` (see [Observability](observability.md)) rather than adding a parallel event bus. Middleware hooks remain for transforming payloads at named boundaries.
 
 ## Security and performance notes
 
@@ -118,6 +119,7 @@ export const extension: Extension = {
 - [Input and prompt assembly](input-and-prompt-assembly.md): `input_assembly` and `prompt_build` helper call sites.
 - [Compaction and retry policies](compaction-and-retry.md): compaction/retry middleware payloads and runtime timing.
 - [Context and skills](context-and-skills.md): `context` helper call site.
+- [Observability](observability.md): optional OpenTelemetry adapter over `AgentEvent` streams.
 - [Public contracts](public-contracts.md): provider, tool, context, session, and extension contracts that runtimes can pass through hooks.
 
 Permission checks for tools, extensions, and resources are hard guards; middleware can transform payloads but cannot bypass a denied `PermissionPolicy`.

package/docs/migration.md

@@ -2,12 +2,12 @@
 
 ## What it does
 
-This page is the single navigation entry for the two cross-cutting migrations external apps hit when moving from Prism's development defaults to its production persistence and explicit-capability surfaces:
+Prism 0.0.4 is source-compatible with documented 0.0.3 agent construction; no mandatory code migration is required. New capabilities are additive and inactive until configured. This page covers two optional adoption paths:
 
-1. **In-memory / JSONL → database-backed persistence** — swap the single-process development `SessionStore` for a host-implemented `ProductionPersistenceStore` / `SessionStore` adapter, and optionally attach a durable `RunLedger`.
-2. **Permissive capability defaults → explicit capability activation** — move from "omitted tool/skill lists activate everything in scope" (pre-Phase 38 behavior) to named, fail-closed tool/skill activation.
+1. **In-memory / JSONL → database-backed persistence** — replace the single-process development `SessionStore` with `@arnilo/prism-session-store-sqlite`, `@arnilo/prism-session-store-postgres`, or a host implementation, and optionally attach its durable `RunLedger`.
+2. **Legacy permissive capability configuration → explicit activation** — name tools/skills and keep omitted capabilities fail-closed.
 
-It is a thin, link-first guide: it states before/after shapes and points at the detailed pages for schema, indexes, redaction, branch handles, capability semantics, and security.
+It states before/after shapes and links detailed schema, redaction, branch, capability, and security guidance.
 
 ## When to use it
 
@@ -15,7 +15,7 @@ Read this page when:
 
 - you are taking an app from the `createMemorySessionStore()` / `createJsonlSessionStore()` path to a multi-process, multi-tenant, or durable database backend;
 - you are hardening an agent that previously relied on "every scoped tool/skill is active" and need to name capabilities explicitly;
-- you are adopting the Phase 34–40 production surfaces (atomic append, branch handles, run/event/tool/usage ledger, security boundary hardening) for the first time.
+- you are adopting 0.0.4 persistence, checkpoints/leases, workflows, structured output, multimodality, or explicit tool safety for the first time.
 
 If you are new to Prism, start at [Session stores](session-stores.md) and [Agent/session runtime](agent-session-runtime.md) instead.
 
@@ -26,6 +26,8 @@ There is no runtime import for this page. The migrations below use these surface
 | Surface | Where | Migration role |
 | --- | --- | --- |
 | `SessionStore` | `@arnilo/prism` | Runtime seam swapped from memory/JSONL to DB. |
+| `createSqlitePersistence` | `@arnilo/prism-session-store-sqlite` | Local durable session, ledger, query, checkpoint, and lease adapter. |
+| `createPostgresPersistence` | `@arnilo/prism-session-store-postgres` | Multi-process pooled persistence with advisory-lock migrations. |
 | `ProductionPersistenceStore` | `@arnilo/prism` | Adapter-facing contract for paginated, multi-tenant reads (`query*`, optional `readBranchPath`). |
 | `RunLedger` / `RunLedgerRecord` | `@arnilo/prism` | Durable run/event/tool-call/usage ledger attached via `AgentConfig.runLedger` / `RunOptions.runLedger`. |
 | `SessionAppendOptions` / `SessionAppendConflictError` / `SessionBranchHandle` | `@arnilo/prism` | Atomic append, retry dedup, durable branch handles. |
@@ -77,34 +79,23 @@ Capability migration (before/after):
 
 ### Migration 1 — in-memory / JSONL → database-backed persistence
 
-A complete, network-free reference adapter that implements these contracts against in-memory tables (and wires a `RunLedger`, branch-handle checkout, fork, and prior-run timeline resume) lives at [`examples/external-app-db-backed.ts`](../examples/external-app-db-backed.ts). The steps below mirror its structure.
+Runnable references: [`examples/workflow-sqlite-resume.ts`](../examples/workflow-sqlite-resume.ts), credential-gated [`examples/workflow-postgres-resume.ts`](../examples/workflow-postgres-resume.ts), and the network-free custom-adapter example [`examples/external-app-db-backed.ts`](../examples/external-app-db-backed.ts).
 
-Step 1: implement a `SessionStore` (or the richer `ProductionPersistenceStore`) against your database. The runtime only requires `append(entry, options?)`, `list(sessionId)`, and optional `get(id)` / `readBranchPath(query)`.
+Step 1: replace the development store with a first-party adapter. Use PostgreSQL instead when multiple processes or sustained concurrent writers matter.
 
 ```ts
 // Before: development store, single process.
 import { createJsonlSessionStore } from "@arnilo/prism/node/session-store-jsonl";
-const store = createJsonlSessionStore("./sessions.jsonl");
-
-// After: host-implemented database adapter implementing the documented contract, no real DB needed to satisfy the contract.
-import type { SessionStore, SessionEntry, SessionAppendOptions, PersistencePage, SessionBranchRead } from "@arnilo/prism";
-
-const store: SessionStore = {
-  async append(entry: SessionEntry, options?: SessionAppendOptions) {
-    // 1. idempotency dedup: insert (session_id, expected_parent_id, idempotency_key, entry_id)
-    //    into prism_session_append_idempotency; unique hit => SessionAppendConflictError { idempotencyDuplicate: true }
-    // 2. expectedParentId existence check => SessionAppendConflictError { expectedParentId } if missing
-    // 3. insert prism_session_entries row; duplicate id fails the transaction
-    // 4. optionally compare-and-swap prism_branches.leaf_entry_id
-  },
-  async list(sessionId: string) { /* O(n) development fallback only */ return []; },
-  async readBranchPath(query: SessionBranchRead): Promise<PersistencePage<SessionEntry>> {
-    // one recursive CTE / ancestor query — do NOT list(sessionId)+in-memory walk for long sessions
-    return { items: [] };
-  },
-};
+const oldStore = createJsonlSessionStore("./sessions.jsonl");
+
+// After: local durable adapter. The same object implements SessionStore,
+// RunLedger, ProductionPersistenceStore, checkpoints, and leases.
+import { createSqlitePersistence } from "@arnilo/prism-session-store-sqlite";
+const store = createSqlitePersistence({ filename: "./prism.db" });
 ```
 
+Custom adapters remain supported through `SessionStore` / `ProductionPersistenceStore`; implement indexed `readBranchPath()` rather than full-session scans.
+
 Step 2: optionally attach a durable run/event/tool/usage ledger and ownership scope so a process exit leaves enough to resume and bill:
 
 ```ts
@@ -174,7 +165,7 @@ Runtime skill activation remains explicit: `RunOptions.activeSkills` narrows per
 
 ## Extension and configuration notes
 
-- **Persistence is host-owned.** Prism ships no database adapter, no DDL, no migration runner. Hosts own connection pools, transactions, cursor encoding, retention jobs, and tenant isolation. The runtime only talks to `SessionStore` (+ optional `readBranchPath`) and `RunLedger`.
+- **Persistence remains host-configured.** Optional SQLite/PostgreSQL packages ship adapters and versioned setup, but hosts choose connection paths/pools, TLS, credentials, retention, tenant policy, and lifecycle. Core only consumes `SessionStore`, `RunLedger`, checkpoint, and lease contracts.
 - **`RunLedger` is not a `SessionStore` replacement.** Messages, branches, and session entries still flow through `SessionStore.append()`; the ledger records run/event/tool/usage facts. See [Runs and usage ledger](runs-and-usage.md).
 - **Capability activation is config over code.** Every seam lives on `AgentDefinition` / `AgentDefinitionResolutionContext` / `RunOptions`; no auto-activation, no privilege grant. A declaration cannot grant permissions or bypass `toolNames`.
 - **Migration order is decoupled.** You can adopt database persistence without changing capability activation, and vice versa. Both migrations are independent config swaps.
@@ -190,7 +181,9 @@ Runtime skill activation remains explicit: `RunOptions.activeSkills` narrows per
 
 ## Related APIs
 
-- [Database persistence](database-persistence.md): production persistence contracts, reference schema, indexes, conditional append, retention, migrations, NoSQL mapping.
+- [Database persistence](database-persistence.md): production contracts, reference schema, indexes, conditional append, retention, migrations, and custom adapters.
+- [SQLite persistence](sqlite-persistence.md): local durable first-party adapter and writer ceiling.
+- [PostgreSQL persistence](postgres-persistence.md): pooled multi-process adapter, TLS/pool ownership, and live gate.
 - [Session stores](session-stores.md): `SessionStore` contract, `SessionAppendOptions`, `SessionAppendConflictError`, branch handles, `readBranchPath`.
 - [Session stores and branching](session-stores-and-branching.md): detailed branch semantics and helper reference.
 - [Runs and usage ledger](runs-and-usage.md): `RunLedger` record shapes, redaction, and event/usage ordering.

package/docs/model-registry.md

@@ -36,7 +36,7 @@ import { createModelRegistry, type ModelConfig } from "@arnilo/prism";
 | --- | --- |
 | `provider` / `model` | Required registry key. |
 | `displayName` | Human-readable label. |
-| `capabilities` | Input/output modes plus reasoning/tools/streaming booleans. |
+| `capabilities` | Input/output modes (`text`, `image`, `audio`, `file`, `document`) plus reasoning/tools/streaming booleans and optional `structuredOutput` (`true` or `"json_schema"`) for native JSON-schema requests. |
 | `limits` | Context and output-token limits. |
 | `cost` | Input/output/cache read/cache write pricing. |
 | `cache` | Generic `ModelCacheCapabilities`. |
@@ -74,7 +74,7 @@ The registry emits no events and performs no I/O.
   "model": {
     "provider": "demo",
     "model": "demo-large",
-    "capabilities": { "input": ["text"], "tools": true, "streaming": true },
+    "capabilities": { "input": ["text", "image", "audio", "file", "document"], "tools": true, "streaming": true },
     "limits": { "contextWindow": 128000, "maxOutputTokens": 8192 },
     "cost": { "input": 10, "output": 30, "cacheRead": 2, "currency": "USD", "unit": "1M tokens" },
     "cache": { "kind": "cache_control", "maxBreakpoints": 4, "longRetention": true }
@@ -91,7 +91,7 @@ const model: ModelConfig = {
   provider: "demo",
   model: "demo-large",
   displayName: "Demo Large",
-  capabilities: { input: ["text"], output: ["text"], tools: true, streaming: true },
+  capabilities: { input: ["text", "document"], output: ["text"], tools: true, streaming: true },
   limits: { contextWindow: 128_000, maxOutputTokens: 8_192 },
   cost: { input: 10, output: 30, cacheRead: 2, cacheWrite: 12, currency: "USD", unit: "1M tokens" },
   cache: { kind: "cache_control", maxBreakpoints: 4, minCacheableTokens: 1024, longRetention: true },
@@ -110,12 +110,14 @@ Provider packages register models through `ProviderPackageAPI.registerModel(mode
 ## Security and performance notes
 
 - Model metadata must not contain credentials or secrets.
+- Declare truthful `capabilities.input` tags. Prism core rejects undeclared modalities in `assembleProviderInput()` when the list is present.
 - Registration is in-memory and O(1) by provider/model key.
 - `ModelConfig.cache` is declarative capability info only; it does not grant permissions, select tools, or bypass auth.
 - Provider-specific behavior belongs in provider packages, not Prism core.
 
 ## Related APIs
 
+- [Multimodal content](multimodal-content.md): `audio`/`file`/`document` blocks and `MODEL_INPUT_CAPABILITIES`.
 - [Provider layer](provider-layer.md): provider/model registry overview.
 - [Provider caching](provider-caching.md): `ModelCacheCapabilities` and cache helpers.
 - [Provider packages](provider-packages.md): package registration of model metadata.

package/docs/multimodal-content.md

@@ -0,0 +1,148 @@
+# Multimodal content
+
+## What it does
+
+Prism core ships generic `audio`, `file`, and `document` `ContentBlock` types plus bounded media resolution helpers. Blocks carry MIME type, optional name, and exactly one source: inline base64 `data`, remote `url`, or host `resourceUri`. Optional `transcript` metadata can accompany audio/document blocks.
+
+`assembleProviderInput()` calls `assertMessagesSupportModelCapabilities()` so declared `ModelCapabilities.input` tags are enforced before provider calls. First-party provider packages map supported blocks locally; unsupported combinations fail closed with `UnsupportedModalityError` or an explicit provider error.
+
+## When to use it
+
+- **Host apps** attaching PDFs, audio clips, or generic files to user messages before a provider turn.
+- **Resource loaders** returning binary payloads for `resourceUri` references under trust/permission policy.
+- **Provider authors** reading truthful `ModelCapabilities.input` tags (`text`, `image`, `audio`, `file`, `document`) before mapping wire formats.
+
+Do not embed provider upload IDs, tenant-scoped remote file IDs, or API-specific handles in core content blocks.
+
+## Inputs / request
+
+```ts
+import {
+  resolveMediaContentBlock,
+  assertSsrfAllowedUrl,
+  type AudioContent,
+  type FileContent,
+  type DocumentContent,
+} from "@arnilo/prism";
+
+const file: FileContent = {
+  type: "file",
+  mediaType: "application/pdf",
+  name: "report.pdf",
+  data: base64Pdf,
+};
+
+const audio: AudioContent = {
+  type: "audio",
+  mediaType: "audio/wav",
+  resourceUri: "package://demo/sample.wav",
+  durationMs: 12_000,
+};
+
+await resolveMediaContentBlock(file, { bounds: { maxItemBytes: 10_000_000 } });
+```
+
+Known `ModelCapabilities.input` tags are exported as `MODEL_INPUT_CAPABILITIES`:
+
+| Tag | Block type | First-party mapping (declared capability required) |
+| --- | --- | --- |
+| `text` | `text` (default) | All providers |
+| `image` | `image` | OpenAI Responses, OpenRouter, OpenCode Go Anthropic route, Kimi, NeuralWatt |
+| `audio` | `audio` | OpenAI Responses (`input_audio`) |
+| `file` | `file` | OpenAI Responses (`input_file`); Anthropic routes map PDF only |
+| `document` | `document` | OpenAI Responses (`input_file`); OpenCode Go Anthropic route; Kimi |
+
+## Outputs / response / events
+
+- `resolveMediaContentBlock()` returns `{ mediaType, bytes, name?, durationMs?, transcript?, metadata? }`.
+- `assertModelSupportsContentBlocks()` / `assertMessagesSupportModelCapabilities()` throw `UnsupportedModalityError` when a declared capability list omits the block modality.
+- `assertMediaBlocksWithinBounds()` enforces per-item bytes, total request bytes, item count, and audio duration ceilings.
+- `assertSsrfAllowedUrl()` rejects private/link-local/metadata hosts unless explicitly allow-listed.
+- `sniffMediaMimeType()` / `assertDeclaredMediaTypeMatches()` compare declared MIME types to magic bytes.
+- No events are emitted and no provider calls occur in these helpers.
+
+Default ceilings:
+
+| Constant | Value |
+| --- | --- |
+| `DEFAULT_MAX_MEDIA_ITEM_BYTES` | 10 MB |
+| `DEFAULT_MAX_MEDIA_REQUEST_BYTES` | 32 MiB |
+| `DEFAULT_MAX_AUDIO_DURATION_MS` | 5 minutes |
+| `DEFAULT_MEDIA_FETCH_TIMEOUT_MS` | 30 seconds |
+| `DEFAULT_MAX_MEDIA_ITEMS_PER_REQUEST` | 32 items |
+
+## Request/response example
+
+```json
+{
+  "block": {
+    "type": "file",
+    "mediaType": "application/pdf",
+    "name": "report.pdf",
+    "resourceUri": "package://demo/report.pdf"
+  },
+  "resolved": {
+    "mediaType": "application/pdf",
+    "bytes": "<Uint8Array>",
+    "name": "report.pdf"
+  }
+}
+```
+
+## Implementation example
+
+```ts
+import {
+  assembleProviderInput,
+  loadBinaryResource,
+  resolveMediaContentBlock,
+  UnsupportedModalityError,
+} from "@arnilo/prism";
+
+const loader = {
+  async load(uri) {
+    return { uri, mediaType: "application/pdf", data: pdfBytes };
+  },
+};
+
+const bytes = await loadBinaryResource(loader, "package://demo/report.pdf");
+const resolved = await resolveMediaContentBlock(
+  { type: "document", mediaType: "application/pdf", resourceUri: "package://demo/report.pdf" },
+  { loader },
+);
+
+try {
+  await assembleProviderInput({
+    model: { provider: "demo", model: "text-only", capabilities: { input: ["text"] } },
+    input: [{ role: "user", content: [{ type: "file", mediaType: "application/pdf", data: "..." }] }],
+  });
+} catch (error) {
+  if (error instanceof UnsupportedModalityError) {
+    // Host-visible reject before provider HTTP.
+  }
+}
+```
+
+## Extension and configuration notes
+
+- URL fetches use injectable `fetch` for tests and custom transports; production hosts should supply TLS, auth, and logging policy outside Prism core.
+- `resourceUri` resolution requires a caller-provided `ResourceLoader` and optional `ResourceLoadContext.permission` check.
+- Local filesystem paths should use trust policies such as `createPathTrustPolicy()` before exposing URIs to loaders.
+- Provider upload/create/delete lifecycles are provider-package-local. `@arnilo/prism-provider-openai` inlines files under 4 MiB as `data:<mediaType>;base64,...` `file_data`, otherwise uses a bounded per-run upload cache and best-effort `DELETE /v1/files` cleanup after each stream.
+- Shared wire helpers live in `@arnilo/prism/providers/media` (`serializeOpenAIResponsesInputFile`, `serializePdfDocumentWireBlock`, `createBoundedUploadCache`).
+
+## Security and performance notes
+
+- SSRF deny-by-default blocks loopback, RFC1918, link-local, and cloud metadata hostnames unless `SsrfPolicy.allowedHostnames` is set.
+- MIME validation rejects common magic-byte spoofing; extensions alone are never trusted.
+- Byte budgets use base64 size estimates before decode and re-check decoded `buffer.length` after read/fetch.
+- Media errors omit raw bytes/base64 payloads from messages.
+- Fetch readers are cancelled promptly after bound violations or abort signals.
+
+## Related APIs
+
+- [Input and prompt assembly](input-and-prompt-assembly.md): attachments and `assembleProviderInput()` capability checks.
+- [Resource loading](resource-loading.md): `loadBinaryResource()` and text/JSON helpers.
+- [Model registry](model-registry.md): `ModelCapabilities.input` metadata.
+- [Provider conformance](provider-conformance.md): serialized request coverage for content blocks.
+- [Persistence, credentials, and multimodality primitives](persistence-credentials-multimodality-primitives.md): Plan 056 inventory and threat model.

package/docs/observability.md

@@ -0,0 +1,163 @@
+# Observability
+
+## What it does
+
+Prism exposes provider and tool timing through stable, metadata-only `AgentEvent` variants. Hosts subscribe via `session.subscribe()` or persist events through `RunLedger`. Core helpers build `ProviderTurnMetadata` and classify HTTP failures without echoing prompts, tool arguments, or credentials.
+
+Optional package `@arnilo/prism-observability-opentelemetry` maps those events to OpenTelemetry spans and low-cardinality metrics. OpenTelemetry is **not** a dependency of `@arnilo/prism`.
+
+APIs:
+
+- `ProviderTurnMetadata`, `ToolExecutionMetadata` on `AgentEvent`
+- `createProviderTurnMetadata()`, `readProviderHttpStatus()` in `@arnilo/prism`
+- `createOpenTelemetryInstrumentation()`, `wrapOpenTelemetryApi()`, `createInMemoryTelemetry()` in `@arnilo/prism-observability-opentelemetry`
+
+## When to use it
+
+Use agent events when you need run-scoped latency, retry attempt numbers, token/cache usage, tool duration, or error classification in-process or through your own exporter.
+
+Use the OpenTelemetry adapter when you already run the OpenTelemetry SDK and want spans/metrics without forking the runtime.
+
+Do not parse raw provider SSE for timing — provider packages normalize stream events; the session emits `provider_turn_*` once per `generate()` attempt.
+
+## Inputs / request
+
+Core metadata helpers:
+
+```ts
+import { createProviderTurnMetadata, readProviderHttpStatus } from "@arnilo/prism";
+
+const metadata = createProviderTurnMetadata(request, providerId, { attempt: 2, latencyMs: 120 });
+const httpStatus = readProviderHttpStatus(errorInfo);
+```
+
+New agent event variants (metadata only):
+
+| Variant | When | Key fields |
+| --- | --- | --- |
+| `provider_turn_started` | Before each provider `generate()` attempt | `turn`, `metadata: ProviderTurnMetadata` |
+| `provider_turn_finished` | After success or failure of that attempt | `metadata` (includes `latencyMs`, optional `httpStatus`), `usage?`, `error?` |
+
+`ToolExecutionMetadata` on terminal tool events:
+
+| Field | Meaning |
+| --- | --- |
+| `durationMs` | Wall time from dispatch start to finish/block/error |
+| `status` | `finished` \| `error` \| `blocked` |
+
+OpenTelemetry adapter:
+
+```ts
+import { trace, metrics } from "@opentelemetry/api";
+import { createOpenTelemetryInstrumentation, wrapOpenTelemetryApi } from "@arnilo/prism-observability-opentelemetry";
+
+const { tracer, meter } = wrapOpenTelemetryApi(trace.getTracer("app"), metrics.getMeter("app"));
+const telemetry = createOpenTelemetryInstrumentation({ tracer, meter, onExporterError: console.error });
+
+const detach = telemetry.attachSession(session);
+// or: for await (const event of session.subscribe()) telemetry.handleAgentEvent(event);
+```
+
+Set `enabled: false` or omit `tracer`/`meter` for a no-op adapter.
+
+## Outputs / response / events
+
+Provider turn metadata fields:
+
+| Field | Source |
+| --- | --- |
+| `providerId` | Active provider id |
+| `model` | `ProviderRequest.model` |
+| `requestId` | `request.metadata.requestId` or `request.options.sessionId` |
+| `attempt` | Retry attempt (1-based) |
+| `latencyMs` | Set on `provider_turn_finished` |
+| `httpStatus` | Numeric `ErrorInfo.code` when present |
+| `rateLimitRemaining` / `rateLimitResetMs` | Reserved for provider adapters (optional) |
+
+OpenTelemetry mapping (when enabled):
+
+| Agent event | Span | Metric labels |
+| --- | --- | --- |
+| `agent_started` / `agent_finished` | `prism.agent.run` | — |
+| `provider_turn_*` | `prism.provider.turn` | `provider_id`, `outcome` on duration histogram |
+| `tool_execution_*` (terminal) | `prism.tool.execute` when started | `status` on duration histogram |
+| `provider_turn_finished` / `agent_finished` usage | span attributes | `prism.provider.tokens` counter (`kind`: input/output/cache_*) |
+
+High-cardinality identifiers (`sessionId`, `runId`, `requestId`, `toolCallId`) are **span attributes only**, never metric labels.
+
+## Request/response example
+
+```json
+{
+  "type": "provider_turn_finished",
+  "sessionId": "sess_01J...",
+  "runId": "run_01J...",
+  "turn": 1,
+  "metadata": {
+    "providerId": "openai",
+    "model": { "provider": "openai", "model": "gpt-4.1" },
+    "requestId": "sess_01J...",
+    "attempt": 2,
+    "latencyMs": 842,
+    "httpStatus": 503
+  },
+  "error": { "message": "upstream unavailable", "code": 503 }
+}
+```
+
+```json
+{
+  "type": "tool_execution_finished",
+  "sessionId": "sess_01J...",
+  "runId": "run_01J...",
+  "result": { "toolCallId": "call_1", "name": "echo" },
+  "metadata": { "durationMs": 12, "status": "finished" }
+}
+```
+
+## Implementation example
+
+```ts
+import { createAgent, createMockProvider, providerDone, providerTextDelta } from "@arnilo/prism";
+import { createInMemoryTelemetry, createOpenTelemetryInstrumentation } from "@arnilo/prism-observability-opentelemetry";
+
+const memory = createInMemoryTelemetry();
+const telemetry = createOpenTelemetryInstrumentation({ tracer: memory.tracer, meter: memory.meter });
+
+const session = createAgent({
+  model: { provider: "mock", model: "demo" },
+  provider: createMockProvider([providerTextDelta("hi"), providerDone()]),
+}).createSession();
+
+const detach = telemetry.attachSession(session);
+await session.run("hello");
+detach();
+
+console.log(memory.spans.map((span) => span.name));
+```
+
+## Extension and configuration notes
+
+- Events flow through `redactAgentEvent` before subscribers and ledger writes — configure `createSecretRedactor` on the agent/run.
+- `retry_scheduled` still signals backoff; each retry attempt emits its own `provider_turn_*` pair with `metadata.attempt`.
+- NeuralWatt `neuralwatt:telemetry` provider events remain package-local; hosts may forward numeric cost/energy into custom metrics.
+- `@arnilo/prism-observability-opentelemetry` is optional and included through `@arnilo/prism-sdk` and `@arnilo/prism-all`; instrumentation remains disabled until a host configures it.
+- Exporter failures are isolated: instrumentation catches tracer/meter errors and invokes `onExporterError` without affecting the run.
+- Disabled instrumentation performs no per-delta span work (`enabled: false` or missing tracer/meter).
+
+## Security and performance notes
+
+- Default events are metadata-only — no prompts, streamed deltas, tool arguments, or credentials.
+- Opt-in content in other event types (`message_delta`, tool `result`) is still subject to `redactAgentEvent`.
+- Metric labels stay low-cardinality (`provider_id`, `outcome`, `status`, token `kind`); never use `sessionId`/`runId` as labels.
+- Target overhead when enabled is under 5% excluding exporter I/O; disabled hooks allocate no spans.
+- Provider transport limits and redaction order are documented in [Provider primitives](provider-primitives.md).
+
+## Related APIs
+
+- [Agent events](agent-events.md): full `AgentEvent` union and subscriber semantics.
+- [Runs and usage ledger](runs-and-usage.md): durable `AgentEventRecord` persistence.
+- [Middleware hooks](middleware-hooks.md): transform boundaries alongside event subscribers.
+- [Provider primitives](provider-primitives.md): frozen observability contract for Plan 054.
+- [Credentials and redaction](credentials-and-redaction.md): secret redaction before events and ledger rows.
+- [Workflows](workflows.md): package-local `WorkflowEvent` stream that can wrap redacted `AgentEvent`s from agent nodes.