@arnilo/prism 0.3.1 → 0.4.0

package/docs/sheets.md

@@ -0,0 +1,229 @@
+# Spreadsheets, CSV parsing, and typed schema inference (`@arnilo/prism-office/sheets`)
+
+## What it does
+
+The `@arnilo/prism-office/sheets` package provides fail-closed, high-fidelity spreadsheet (XLSX) and delimiter-separated (CSV/TSV/PSV) data ingestion with automatic dialect sniffing, typed column schema inference, and **strict financial decimal safety**.
+
+### Headline Guarantee: Strict Financial Decimal Safety
+
+> [!IMPORTANT]
+> **Zero Float Coercion on Decimal Paths**:
+> In financial and enterprise data processing, floating-point rounding errors (IEEE-754 `double`) silently distort monetary totals, balance ledgers, and transaction reconciliations.
+>
+> In `@arnilo/prism-office/sheets`:
+> - Money-like and decimal values are **never converted to JavaScript numbers (`Number()`, `parseFloat()`, or unary `+`)**.
+> - All decimal and currency values are parsed, normalized, and emitted as exact canonical decimal strings: `{ type: "decimal", value: "1234.56" }`.
+> - Currency markers (`$`, `€`, `£`, `¥`, `₹`, `CHF`, `USD`, `EUR`, etc.) and accounting parentheses `($1,234.56)` are normalized safely into canonical strings (`"-1234.56"`).
+> - Ambiguous numbers (e.g. scientific notation `1.23e5` or inconsistent locale separators) are preserved as raw strings with `flags: ["numeric-ambiguous"]` rather than guessed.
+> - An automated source-scan test in the regression suite enforces that no floating-point conversions exist on decimal paths across the codebase.
+
+### Core Capabilities
+
+- **Pure In-Memory Operation**: Accepts `Uint8Array` binary archives or `string | Uint8Array` CSV text. Zero filesystem access, zero network I/O, zero `process.env` lookups, and zero background worker threads.
+- **Fail-Closed Container Gating**: XLSX archives must begin with the standard ZIP container magic signature (`PK\x03\x04`). CSV inputs support UTF-8 (with automatic BOM stripping); UTF-16 encoded buffers are refused fail-closed with `ERR_PRISM_SHEETS_UNSUPPORTED_FORMAT`.
+- **Automatic CSV Dialect Sniffing**: Analyzes the first 4 KiB / 50 lines to detect candidate delimiters (`,`, `;`, `\t`, `|`) using variance-based scoring outside quoted regions, correctly distinguishing delimiters from commas within quoted addresses or company names.
+- **RFC 4180 State Machine**: Handles embedded newlines in quoted fields, doubled-quote escaping (`""`), and CRLF / LF line endings in a single, non-backtracking linear pass (<60 ms for 1 MB CSVs).
+- **Read-Only Formula Preservation**: OpenXML formula cells are extracted as `{ type: "formula", formula: "=SUM(A1:A10)", cachedValue: "100" }` and are **never evaluated or executed**.
+- **Bounded Sampling Window + Full Validation**: Infers column types across an initial window (default 500 rows) and validates all subsequent records against the inferred schema, reporting structured `type-mismatch` warnings without silent coercion.
+- **Privacy-Guaranteed Telemetry Seam**: Optional, dependency-free `SheetsTelemetry` hook emits `sheets.parse` spans carrying byte, row, column, and duration metrics with zero cell text or confidential payload leakage.
+
+## When to use it
+
+Use `@arnilo/prism-office/sheets` when autonomous agents, data pipelines, or enterprise workflows need to:
+1. Ingest untrusted customer XLSX or CSV files with strict, unbypassable byte, row, column, and sheet caps.
+2. Parse tabular financial records, invoices, ledgers, or pricing sheets with mathematical decimal precision guarantees.
+3. Automatically determine CSV delimiters, quotes, and headers without manual dialect configuration.
+4. Extract structural column schemas (`name`, `type`, `nullRate`, `sample`, `flags`) for automated data cataloging, SQL generation, or tool routing.
+
+Do **not** use this package for interactive spreadsheet calculation/formula engines, macro execution, chart generation, or real-time collaborative editing.
+
+## Inputs / request
+
+### Primary Parsing Functions
+
+| Function | Signature | Description |
+| --- | --- | --- |
+| `parseWorkbook` | `(bytes: Uint8Array, options?: ParseWorkbookOptions) => Promise<WorkbookParse>` | Parses an XLSX workbook binary buffer, enforces caps and ZIP signature gating, extracts raw cells, resolves shared strings and styles, and infers schemas per worksheet. |
+| `parseCsv` | `(input: string \| Uint8Array, options?: ParseCsvOptions) => Promise<CsvParse>` | Parses CSV/TSV/PSV input with automatic dialect sniffing, BOM stripping, RFC 4180 state machine processing, and schema inference. |
+| `inferAndTransformRows` | `(rows: readonly (readonly CellValue[])[], caps: ResolvedSheetsCaps) => InferAndTransformResult` | Pure function that infers column types over a sampling window and validates full rows against the inferred schema. |
+
+### Decimal & Normalization Utilities
+
+| Function | Signature | Description |
+| --- | --- | --- |
+| `normalizeDecimal` | `(input: string) => NormalizedDecimalResult \| null` | Normalizes currency strings, accounting negatives, and thousands separators into canonical decimal strings (`^-?\d+(\.\d+)?$`). Returns `null` if ambiguous or invalid. |
+| `isCanonicalDecimal` | `(str: string) => boolean` | Checks if a string is already in canonical decimal format (e.g. `"1234.56"`, `"-0.05"`, `"42"`). |
+| `isCurrencyString` | `(str: string) => boolean` | Checks if a string contains explicit currency symbols or currency codes (`$`, `€`, `USD`, etc.). |
+| `isScientificNotation` | `(str: string) => boolean` | Checks if a string represents scientific notation (e.g. `"1.23e5"`, `"4.56E-3"`). |
+
+### Cap Validation Utilities
+
+| Function | Signature | Description |
+| --- | --- | --- |
+| `resolveSheetsCaps` | `(caps?: SheetsCaps) => ResolvedSheetsCaps` | Resolves user-configured limits against safe defaults and hard ceilings. |
+| `validateByteCap` | `(byteLength: number, caps: ResolvedSheetsCaps) => void` | Validates input size against `caps.maxBytes`, throwing `SheetsCapError` if exceeded. |
+| `validateZipSignature` | `(bytes: Uint8Array) => void` | Validates standard PKZIP magic bytes (`0x50, 0x4b, 0x03, 0x04`), throwing `SheetsFormatError` if missing. |
+
+### Capacity Limits and Defaults
+
+Limits are verified upfront and enforced progressively to prevent memory exhaustion and denial-of-service from adversarial inputs:
+
+| Cap | Default | Hard Ceiling | Description |
+| --- | --- | --- | --- |
+| `maxBytes` | 32 MiB (`33,554,432`) | 512 MiB (`536,870,912`) | Maximum input buffer or string byte length. |
+| `maxSheets` | 100 | 1,000 | Maximum worksheets in a workbook archive. |
+| `maxRows` | 100,000 | 1,000,000 | Maximum rows per worksheet or CSV document. |
+| `maxColumns` | 1,000 | 16,384 | Maximum columns per worksheet or CSV document. |
+| `inferenceWindowRows` | 500 | 5,000 | Number of rows sampled for column schema inference. |
+| `maxWarnings` | 100 | 1,000 | Maximum validation and dialect warnings recorded. |
+
+## Outputs / response / events
+
+### Error Hierarchy
+
+All error classes inherit from `SheetsError` and carry structured error codes:
+
+| Error Class | Code | Cause / Trigger |
+| --- | --- | --- |
+| `SheetsCapError` | `ERR_PRISM_SHEETS_CAP` | Input size, sheet count, row count, or column count exceeds configured caps. |
+| `SheetsValidationError` | `ERR_PRISM_SHEETS_VALIDATION` | Invalid cap configuration (e.g. non-integer or exceeding hard ceiling). |
+| `SheetsFormatError` | `ERR_PRISM_SHEETS_UNSUPPORTED_FORMAT` | Missing ZIP container signature on XLSX input or unsupported encoding (UTF-16) on CSV input. |
+| `SheetsParseError` | `ERR_PRISM_SHEETS_PARSE_FAILED` | Corrupt archive structure or malformed XML parts. |
+
+### Schema & Warnings Output Shape
+
+```ts
+export interface ColumnSchema {
+  readonly name: string;
+  readonly type: "string" | "integer" | "number" | "decimal" | "date" | "datetime" | "boolean";
+  readonly nullRate: number;
+  readonly sample?: string;
+  readonly flags?: readonly ("numeric-ambiguous" | "mixed-types")[];
+}
+
+export interface InferenceWarning {
+  readonly row: number;
+  readonly column: number;
+  readonly columnName?: string;
+  readonly expectedType: string;
+  readonly actualValue: string;
+  readonly message: string;
+}
+```
+
+## Request/response example
+
+### Input CSV
+
+```csv
+transaction_id,item_description,total_amount,unit_price,refund_fee
+TXN-1001,"Software Consulting","$1,234.56",1234.56,"($ 50.00)"
+TXN-1002,"Cloud Server Hosting","€ 450.00",450.00,"-$ 10.00"
+TXN-1003,"Hardware Device","£ 2,500.00",2500.00,"$ 0.00"
+```
+
+### Parsed Output (`CsvParse`)
+
+```json
+{
+  "dialect": {
+    "delimiter": ",",
+    "quote": "\"",
+    "hasHeader": true
+  },
+  "schema": [
+    { "name": "transaction_id", "type": "string", "nullRate": 0, "sample": "TXN-1001", "flags": [] },
+    { "name": "item_description", "type": "string", "nullRate": 0, "sample": "Software Consulting", "flags": [] },
+    { "name": "total_amount", "type": "decimal", "nullRate": 0, "sample": "$1,234.56", "flags": [] },
+    { "name": "unit_price", "type": "decimal", "nullRate": 0, "sample": "1234.56", "flags": [] },
+    { "name": "refund_fee", "type": "decimal", "nullRate": 0, "sample": "($ 50.00)", "flags": [] }
+  ],
+  "rows": [
+    ["transaction_id", "item_description", "total_amount", "unit_price", "refund_fee"],
+    ["TXN-1001", "Software Consulting", { "type": "decimal", "value": "1234.56" }, { "type": "decimal", "value": "1234.56" }, { "type": "decimal", "value": "-50.00" }],
+    ["TXN-1002", "Cloud Server Hosting", { "type": "decimal", "value": "450.00" }, { "type": "decimal", "value": "450.00" }, { "type": "decimal", "value": "-10.00" }],
+    ["TXN-1003", "Hardware Device", { "type": "decimal", "value": "2500.00" }, { "type": "decimal", "value": "2500.00" }, { "type": "decimal", "value": "0.00" }]
+  ],
+  "warnings": []
+}
+```
+
+## Implementation example
+
+```ts
+import { parseWorkbook, parseCsv, type SheetsTelemetry } from "@arnilo/prism-office/sheets";
+
+// 1. Parse XLSX workbook with custom caps
+const xlsxBytes = new Uint8Array([...]); // Untrusted file bytes
+const workbook = await parseWorkbook(xlsxBytes, {
+  caps: {
+    maxRows: 50_000,
+    maxColumns: 500,
+    maxBytes: 16 * 1024 * 1024,
+  },
+});
+
+for (const sheet of workbook.sheets) {
+  console.log(`Sheet: ${sheet.name} (${sheet.rows.length} rows)`);
+  for (const col of sheet.schema) {
+    console.log(`  Column [${col.name}] inferred as ${col.type} (nullRate: ${col.nullRate})`);
+  }
+}
+
+// 2. Parse CSV with automatic dialect sniffing and telemetry
+const csvText = `id;name;revenue\n1;"Acme, Corp";1000000.00\n2;"Global, Inc";500000.50\n`;
+
+const telemetry: SheetsTelemetry = {
+  startSpan(name, attributes) {
+    console.log(`Span started: ${name}`, attributes);
+    return {
+      setAttribute() {},
+      addEvent() {},
+      recordError() {},
+      end() {},
+    };
+  },
+};
+
+const csvResult = await parseCsv(csvText, { telemetry });
+console.log(`Detected delimiter: "${csvResult.dialect.delimiter}"`);
+console.log(`Revenue value:`, csvResult.rows[1][2]);
+// Output: { type: "decimal", value: "1000000.00" }
+```
+
+## Extension and configuration notes
+
+### Sub-package Pinning
+To avoid pulling in CLI frameworks or extraneous dependencies, `@arnilo/prism-office/sheets` directly pins the exact underlying modular packages:
+- `@office-open/xlsx@0.12.3`
+- `@office-open/xml@0.12.3`
+
+### Custom Telemetry Hook
+The `SheetsTelemetry` seam allows optional OpenTelemetry instrumentation without adding runtime telemetry dependencies:
+```ts
+const telemetry: SheetsTelemetry = {
+  startSpan(name, attributes) {
+    // Maps to tracer.startSpan with allow-listed metadata (bytes, rows, columns, sheetCount)
+    // Cell contents and user data are NEVER passed to telemetry spans.
+    return activeSpan;
+  },
+};
+```
+
+### Self-Hosting & Operational Notes
+- **Zero Network & Storage Dependencies**: `@arnilo/prism-office/sheets` does not write files or contact network services. Host engines own persistence, storage buckets, and lake datasets.
+- **Fail-Closed Container Gating**: Malicious or non-standard files are rejected before allocation or XML decompression occurs.
+
+## Security and performance notes
+
+- **Pure In-Memory Operation**: No temporary files, no shell execution, no binary spawning, and zero network sockets.
+- **ZIP Signature Gating**: Buffers must begin with standard PKZIP container signatures (`0x50, 0x4B, 0x03, 0x04`). Extension-based type inference is strictly prohibited.
+- **Fail-Closed Caps**: Input size and element count caps are evaluated before entering XML translation passes, preventing zip-bomb and decompression amplification attacks.
+- **Strict Anti-Corruption Invariant**: Zero floating-point conversions on decimal paths guarantee exact financial calculations and ledger balances.
+- **Performance Budget**: Warm parsing of 1 MB CSV files completes in under 60 ms; workbook parsing completes in under 100 ms.
+
+## Related APIs
+
+- [`@arnilo/prism-office/documents`](./documents.md): Specification-compliant OpenXML document generation, parsing, patching, and preview rendering for DOCX, XLSX, and PPTX.
+- [`@arnilo/prism-document-reader`](./document-reader.md): Bounded literal text extraction from PDF and DOCX documents for coding agent tools.
+- [`@arnilo/prism-observability-opentelemetry`](./observability.md): OpenTelemetry instrumentation and trace adapters.

package/docs/supervisors.md

@@ -17,12 +17,16 @@ Use a supervisor when a host or agent must choose a child dynamically. Use `@arn
 | `delegate({ childId, input, threadId?, limits?, signal? })` | Invokes one allow-listed child. Input is text and byte-bounded. |
 | `hooks.before` | May reject, modify redacted input, or narrow limits/policy. |
 | `hooks.after` | Observes redacted terminal summary; failures cannot alter settled result. |
-| `limits` | Depth 4/16, active children 4/32, input 64 KiB/1 MiB, steps 8/64, tools 32/256, tokens 20k/1m, timeout 60s/30m, event queue 128/4096 default/hard. Over-cap `delegate()` throws `SupervisorLimitError` before incrementing `activeChildren`. Hook rejection and timeout decrement the count exactly once (no leaked timers). |
+| `limits` | Depth 4/16, active children 4/32, input 64 KiB/1 MiB, steps 8/64, tools 32/256, tokens 20k/1m, timeout 60s/30m, event queue 128/4096, child events/delegation 256/4096, child-event bytes 32 KiB/256 KiB default/hard. Over-cap `delegate()` throws `SupervisorLimitError` before incrementing `activeChildren`. Hook rejection and timeout decrement the count exactly once (no leaked timers). |
 
 ## Outputs / response / events
 
 `delegate()` returns the child's `AgentRunResult` or throws its `AgentRunError`/a supervisor denial or limit error. `subscribe()` emits bounded `delegation_started`, `delegation_finished`, `delegation_rejected`, and `delegation_error` metadata events. Graceful close drains already-queued terminal events before the iterator completes (same core multiplexer contract). Hosts may project those events through observability `handleDelegation()` using the parent Prism run ID; no OpenTelemetry dependency enters this package.
 
+### Child event passthrough (opt-in)
+
+`createSupervisor({ childEvents: true })` projects a redacted, size-capped **milestone** subset of child `AgentEvent`s onto the same stream as `delegation_child_event` (tagged `childId`, `delegationId`, `depth`). v1 covers run start/finish/`suspended`/`denied` and tool-execution started/finished/error/blocked — not per-token `message_delta`. Default off: the stream is byte-identical to today (no subscribe, no allocation). Caps: `limits.maxChildEventsPerDelegation` (256/4096) and `limits.maxChildEventBytes` (32 KiB/256 KiB); exceeding either drops further child events and emits one `delegation_child_events_capped` marker (never throws). Events pass through the supervisor `redactor` before emission. Children never receive supervisor internals or store/subscription access. Resume-path rebuilds (`resumeNestedRun`) do not currently project child events — live passthrough is the initial `delegate()` session only.
+
 ## Request/response example
 
 ```json
@@ -50,6 +54,8 @@ const supervisor = createSupervisor({
 const result = await supervisor.delegate({ childId: "research", input: "Check sources" });
 ```
 
+> **Contract — child factories return `Agent`.** `createAgent` must return an `Agent`, not an `AgentSession` (or a plain object). Wrong type throws `SupervisorError: child "<id>" factory must return an Agent, got <type>` on both initial `delegate()` and nested resume. Nested approvals also need a **stable config** plus a **durable (or rebuild-stable) store** — calling `createSession()` inside the factory and returning that session loses the child's checkpointed leaf. Live demo: [`examples/autonomous-coding-loop.ts`](../examples/autonomous-coding-loop.ts) (`childAgent` returns `createAgent(...)`).
+
 ## Durable child approvals
 
 With `checkpoints` + `definitionRevision`, every child run is durable with `interruptBeforeTool: true`. A child that suspends on pending decisions throws `AgentDelegationSuspendedError` out of `delegate()`; when the delegation runs inside a root agent's tool, core converts it into a root suspension whose `interruption.pendingDecisions` carry hashed root-visible approval ids (`sub_<sha256(runId:childApprovalId)>`) and `attribution.path` (redacted child ids, root first, at most 8 deep). Root decisions route back through the same CAS rules: pass `supervisor.resumeNestedRun` as `resumeNestedRun` in the root run's `runState` and in every `resumeAgentRun` options object. The supervisor rebuilds the child from a bounded delegation mapping stored in the same checkpoint store (child id, delegation/thread ids, redacted input, version), re-runs the `before` hook so its narrowing applies to the resumed run (hooks must be idempotent), and re-attributes re-suspensions recursively, so grandchild decisions surface with the full path. A delegating child's own `interruptBeforeTool` also gates its delegate tool, so hosts approve delegation and the child's own side effects as separate stages. Root `*_for_run` stickies record the attribution path and only match the same delegation path; child stickies live on the child run and expire with it. A root approval never widens the child: the child's narrowed permission re-runs at dispatch. Unknown or foreign nested run ids fail closed with one non-enumerating error. Child factories must return stable configs and a durable (or rebuild-stable) session store for resume to work.
@@ -72,9 +78,11 @@ Supervisors propagate parent `identity` and `effectStore` to every child agent/r
 
 ## Related APIs
 
+- [Multi-agent patterns](multi-agent-patterns.md): the decision table comparing this delegation pattern with in-session handoff (swarm) and A2A; handoff keeps one transcript, delegation keeps separate child runs — choose by boundary, budget, and telemetry needs.
 - [Agent identity](agent-identity.md): host-verified identity and narrow delegation.
 - [A2A interoperability](a2a.md): separate remote protocol boundary. `A2ATaskLifecycle` adapts host durable agent/workflow state directly; it does not route A2A execution through local supervisor child planning.
 - [Workflows](workflows.md): preferred deterministic orchestration.
+- Example: [`examples/autonomous-coding-loop.ts`](../examples/autonomous-coding-loop.ts) — per-child models, factory returns `Agent`.
 - [Working and semantic memory](working-and-semantic-memory.md): child scope construction.
 - [Host security](host-security.md): permission and credential boundaries.
 - [Obscura browser engine](obscura.md): optional binary-backed generic tools for child agents.

package/docs/thinking-and-reasoning.md

@@ -59,16 +59,16 @@ Core maps only shapes shared by ≥2 packages (or an explicit no-op). Unique kno
 
 | Package | Recommended family | Notes |
 | --- | --- | --- |
-| `@arnilo/prism-provider-openai` | `openai_reasoning` | First-class body `reasoning` from model + per-turn compat merge; `summary`/`mode`/`context` via compat |
-| `@arnilo/prism-provider-openrouter` | `openai_reasoning` | First-class `resolveOpenRouterReasoning` merge; prefer `reasoning` object over legacy `reasoning_effort` shorthand; `preserveThinking` replays as body `reasoning` |
-| `@arnilo/prism-provider-zai` | `reasoning_effort` (+ optional `thinking_type`) | Official `thinking` / `reasoning_effort` / `tool_stream` / `clear_thinking`; Preserved Thinking via `reasoning_content` |
-| `@arnilo/prism-provider-neuralwatt` | `reasoning_effort` | Budgets / `preserve_thinking` / `clear_thinking` / `chat_template_kwargs` stay package-local on `compat` |
-| `@arnilo/prism-provider-kimi` | K3: `reasoning_effort`; K2.x: `thinking_type` | K2.7-code thinking is always on; do not send conflicting `thinking` + `reasoning_effort` |
-| `@arnilo/prism-provider-opencode-go` | Anthropic route: thinking blocks (`thinking_type` family); OpenAI route: `reasoning_content` preserve + optional `thinking`/`reasoning_effort`/`reasoning` passthrough | Official dual endpoints; MiniMax/Qwen → Anthropic, others → OpenAI |
-| `@arnilo/prism-provider-ai-sdk` | `noop` | Host `LanguageModelV4` owns reasoning settings |
-| `@arnilo/prism-provider-deepseek` | `thinking_type` + `reasoning_effort` | Thinking on by default (`high`). `cacheRetention: "none"` or `thinking: false` disables. Tool turns must replay `reasoning_content` or the API returns 400. |
-| `@arnilo/prism-provider-xai` | replay only | Featured Completions do not send `reasoning_effort`. Reasoning models must replay `reasoning_content` or the prefix cache breaks. Do not flatten thinking into text. |
-| `@arnilo/prism-provider-clinepass` | `reasoning_effort` | Per-model `compat.thinkingLevelMap`. GLM `xhigh` passthrough (never send `max`). K3 `high` → `max`. Unsupported slots omit the field. |
+| `@arnilo/prism-providers/openai` | `openai_reasoning` | First-class body `reasoning` from model + per-turn compat merge; `summary`/`mode`/`context` via compat |
+| `@arnilo/prism-providers/openrouter` | `openai_reasoning` | First-class `resolveOpenRouterReasoning` merge; prefer `reasoning` object over legacy `reasoning_effort` shorthand; `preserveThinking` replays as body `reasoning` |
+| `@arnilo/prism-providers/zai` | `reasoning_effort` (+ optional `thinking_type`) | Official `thinking` / `reasoning_effort` / `tool_stream` / `clear_thinking`; Preserved Thinking via `reasoning_content` |
+| `@arnilo/prism-providers/neuralwatt` | `reasoning_effort` | Budgets / `preserve_thinking` / `clear_thinking` / `chat_template_kwargs` stay package-local on `compat` |
+| `@arnilo/prism-providers/kimi` | K3: `reasoning_effort`; K2.x: `thinking_type` | K2.7-code thinking is always on; do not send conflicting `thinking` + `reasoning_effort` |
+| `@arnilo/prism-providers/opencode-go` | Anthropic route: thinking blocks (`thinking_type` family); OpenAI route: `reasoning_content` preserve + optional `thinking`/`reasoning_effort`/`reasoning` passthrough | Official dual endpoints; MiniMax/Qwen → Anthropic, others → OpenAI |
+| `@arnilo/prism-providers/ai-sdk` | `noop` | Host `LanguageModelV4` owns reasoning settings |
+| `@arnilo/prism-providers/deepseek` | `thinking_type` + `reasoning_effort` | Thinking on by default (`high`). `cacheRetention: "none"` or `thinking: false` disables. Tool turns must replay `reasoning_content` or the API returns 400. |
+| `@arnilo/prism-providers/xai` | replay only | Featured Completions do not send `reasoning_effort`. Reasoning models must replay `reasoning_content` or the prefix cache breaks. Do not flatten thinking into text. |
+| `@arnilo/prism-providers/clinepass` | `reasoning_effort` | Per-model `compat.thinkingLevelMap`. GLM `xhigh` passthrough (never send `max`). K3 `high` → `max`. Unsupported slots omit the field. |
 
 `thinkingFamilyForModel` infers family from existing `compat` shape, then safe provider heuristics (`openai*` → `openai_reasoning`, `neuralwatt` → `reasoning_effort`), then `capabilities.reasoning` → `reasoning_effort`, else `noop`. Docs and packages may map other provider ids explicitly; core avoids provider-specific literals beyond those heuristics.
 

package/docs/tool-conformance.md

@@ -2,14 +2,15 @@
 
 ## What it does
 
-Tool conformance helpers are dependency-free assertions for tool-dispatch configuration tests. They exercise the blocked-reason matrix and the success path of `dispatchToolCall` without network or credentials.
+Tool conformance helpers are dependency-free assertions for tool-dispatch configuration tests. They exercise the blocked-reason matrix and the success path of `dispatchToolCall` without network or credentials, and the tool-disclosure contract (progressive tool loading, plan 041) without any provider call.
 
 Exported from `@arnilo/prism/testing/tool-conformance`:
 
 - `assertToolDispatchConforms(registry, options)`
+- `assertToolDisclosureConforms(options)`
 - `assertToolBlocked(probe, expectedReason)`
 - `dispatchAndCollect(probe)`
-- `ToolConformanceOptions`, `ToolDispatchProbeOptions`
+- `ToolConformanceOptions`, `ToolDispatchProbeOptions`, `ToolDisclosureConformanceOptions`
 
 ## When to use it
 
@@ -45,6 +46,29 @@ await assertToolDispatchConforms(createToolRegistry(), {
 
 `assertToolDispatchConforms` returns `Promise<void>` and throws on the first violation. `dispatchAndCollect` returns `{ result, events }` capturing the emitted `AgentEvent`s for custom assertions.
 
+`assertToolDispatchConforms` returns `Promise<void>` and throws on the first violation. `dispatchAndCollect` returns `{ result, events }` capturing the emitted `AgentEvent`s for custom assertions.
+
+### Disclosure leg (toolsDisclosure "search")
+
+`assertToolDisclosureConforms(options)` asserts the progressive tool-loading contract against the same narrowing the runtime applies (`filterTools` allow/deny bounds, then `selectDisclosedTools`):
+
+- search mode only narrows: the disclosed set is a subset of the allow/deny-filtered input, never wider, never zero, deterministic order for identical turns; a deny-listed tool is never described to the provider
+- the generated `search_tools` tool is always kept in the disclosed set
+- fail closed: an index over the frozen 1024-tool cap discloses the full eligible list
+- `search_tools` output is inert: names plus byte-truncated (512-char) descriptions only — no JSON structure, no tool schemas — oversized hosts descriptions are truncated, never executed, and configured `secrets` never surface even when a description carries one
+- activation is bounded to topK and only ever selects from the eligible set; activated tools stay disclosed on the next turn
+
+```ts
+import { assertToolDisclosureConforms } from "@arnilo/prism/testing/tool-conformance";
+
+assertToolDisclosureConforms({
+  tools: hostTools,                        // incl. schema-bearing and oversized-description tools
+  filter: { deny: ["legacy_tool"] },       // host allow/deny bounds
+  search: { topK: 16 },
+  secrets: [hostSecret],                   // secret-scan of model-facing search output
+});
+```
+
 ## Request/response example
 
 ```ts
@@ -84,5 +108,6 @@ await assertToolDispatchConforms(createToolRegistry(), {
 ## Related APIs
 
 - [Tools](tools.md)
+- [Tool effects](tool-effects.md)
 - [Settings, auth, trust, security](settings-auth-trust-security.md)
 - [Provider conformance](provider-conformance.md)

package/docs/tools.md

@@ -251,13 +251,40 @@ createJsonSchemaToolArgumentValidator({
 });
 ```
 
+## Tool disclosure (progressive tool loading)
+
+`toolsDisclosure` on `AgentConfig` / `RunOptions` (run wins; default `"all"`) controls how active tools reach the provider request. Default `"all"` sends every active tool schema — byte-identical to releases before the option existed. Opt-in `"search"` surfaces a bounded top-k subset per turn (scored lexically against the turn input over name and description) plus the generated `search_tools` tool; the model requests more by calling it.
+
+```ts
+const agent = createAgent({
+  model, provider,
+  tools,                                  // host-active ToolDefinitions (or registry)
+  toolsDisclosure: "search",              // default "all"
+  toolsSearch: { topK: 16 },              // optional; clamped to the hard cap
+});
+```
+
+Limits (mirroring the skill-disclosure DEFAULT/HARD cap pattern):
+
+| Limit | Default | Hard cap |
+| --- | --- | --- |
+| Disclosed tools per turn (`topK`) | 16 | 64 |
+| Indexed tools | — | 1024 (fail closed to full disclosure) |
+| Search query bytes | 4096 | 65536 |
+
+- `search_tools({ query, k? })` returns inert `name: short description [matched: …]` lines — no schemas or tool bodies — and marks returned tools active for the session. Activation is names-only in run persistence (`sessionState.activatedToolNames`, capped at 128 names) and inert for tools absent from the current registry; a host can reset it with `session.clearActivatedTools()`.
+- Fail closed: any index or scoring error discloses the full input list — never zero tools, never wider than the input list. Exhausting the frozen 1024-tool index cap is surfaced the same way.
+- Disclosure never grants access: dispatch re-checks registry membership and allow/deny (`unknown_tool` / `tool_denied`) on every call regardless of what was described. Search results are intersected with the disclosed list structurally — searched tools are only ever selected from that list, never widened.
+- Scoring is BM25-lite lexical (name tokens weigh ×3, IDF from the registry): bounded, dependency-free, deterministic tie-breaks. ponytail ceiling: embedder-backed scoring via `@arnilo/prism-rag` if accuracy fixtures fall short.
+- Cross-link: skills apply the same discipline to prompt text — see [Context and skills](context-and-skills.md).
+
 ## Guardrails
 
 `DispatchToolCallOptions.guardrails` evaluates `tool_input` after `tool_call` middleware normalization and before lookup, permission, validation, execution policy, or side effect. `tool_output` evaluates raw completed results before redaction, event emission, ledger rows, and transcript append. A block returns a blocked result; tripwire fails the enclosing run. See [Guardrails](guardrails.md).
 
 ## Related APIs
 
-- [OpenAPI tools adapter](openapi-tools.md): optional `@arnilo/prism-openapi-tools` `createOpenApiTools` — compile host-selected OpenAPI 3.1 operationIds into bounded `ToolDefinition`s (allow-list only, pinned origin, resolved/bounded schemas, approval + effect-store idempotency on mutations, bounded body/response/retries/pagination, host credential resolver, untrusted output).
+- [OpenAPI tools adapter](openapi-tools.md): optional `@arnilo/prism-coding-tools/openapi` `createOpenApiTools` — compile host-selected OpenAPI 3.1 operationIds into bounded `ToolDefinition`s (allow-list only, pinned origin, resolved/bounded schemas, approval + effect-store idempotency on mutations, bounded body/response/retries/pagination, host credential resolver, untrusted output).
 - [Agent/session runtime](agent-session-runtime.md): dispatches complete provider tool calls through the host-active tool harness and returns tool results on the next provider turn.
 - [Public contracts](public-contracts.md): `ToolDefinition`, `ToolRegistry`, `ToolExecutionContext`, `ToolResult`, and tool `AgentEvent` contracts.
 - [Contribution registries](contribution-registries.md): inert extension/package tool contribution storage.
@@ -270,6 +297,6 @@ createJsonSchemaToolArgumentValidator({
 - [MCP client bridge](mcp-tools.md): optional remote tool mapping plus separate bounded resource/prompt facades; non-tool MCP capabilities never bypass tool dispatch by masquerading as `ToolDefinition`.
 - [Recoverable tool effects](tool-effects.md): optional `tool.effect` + `effectStore` claim/CAS recovery around dispatch.
 - [Recoverable tool effects](tool-effects.md): optional `tool.effect` + `effectStore` claim/CAS recovery around dispatch.
-- [Coding agent tools](coding-agent-tools.md): optional first-party `@arnilo/prism-coding-agent` `shell`/`read`/`write`/`edit` tools a host registers into this harness.
+- [Coding agent tools](coding-agent-tools.md): optional first-party `@arnilo/prism-coding-tools/agent` `shell`/`read`/`write`/`edit` tools a host registers into this harness.
 
 `DispatchToolCallOptions.trust` and `.permission` run before validation or `execute()`; denial emits `tool_execution_blocked`. Middleware cannot bypass either guard. `AgentConfig.validator`/`RunOptions.validate` run after these guards; their output is redacted through the active `SecretRedactor`. `createSecureAgent()` requires all three seams plus non-empty schemas and durable pre-tool approval. Prism does not sandbox tools. See [Security/auth/trust](settings-auth-trust-security.md).

package/docs/web-tools.md

@@ -8,7 +8,7 @@
 
 Use when agent needs explicit public-web discovery or host-approved document retrieval/extraction. Keep search separate from fetch/extract so model cannot select provider, credential, API origin, extraction schema, or cost path.
 
-**Obscura-backed alternative**: the optional [`@arnilo/prism-obscura`](obscura.md) package provides `web_search`/`web_fetch` behavior backed by a host-installed Obscura headless browser through its CLI (one replaceable HTML search profile instead of an API key), plus explicit native `obscura_fetch`/`obscura_scrape` batch tools. It reuses this package's normalized citation/untrusted shapes (`provider: "obscura"`) but does not require credentials; the API-backed Brave/Exa/Firecrawl adapters here remain the preferred path when an API key is available.
+**Obscura-backed alternative**: the optional `@arnilo/prism-web-tools/obscura` subpath ([Obscura](obscura.md)) provides `web_search`/`web_fetch` behavior backed by a host-installed Obscura headless browser through its CLI (one replaceable HTML search profile instead of an API key), plus explicit native `obscura_fetch`/`obscura_scrape` batch tools. It reuses this package's normalized citation/untrusted shapes (`provider: "obscura"`) but does not require credentials; the API-backed Brave/Exa/Firecrawl adapters here remain the preferred path when an API key is available.
 
 ## Inputs / request
 
@@ -69,7 +69,7 @@ Default/hard limits: query 4/16 KiB; results 10/20; URLs 5/20; request 256 KiB/1
 
 Provider credentials never enter tool schemas/results, prompts, telemetry, URLs, or errors. Error text excludes remote bodies. Search snippets, Markdown, and extracted JSON are prompt-injection-capable data: never concatenate them into system instructions or use them to modify tools, permissions, credentials, trust, routing, or schemas. Firecrawl fetches target URLs remotely; Prism cannot claim target DNS pinning after handoff. Use controlled host fetch when that guarantee is required.
 
-Default tests use injected fake fetch and make no public request. Restricted smoke: `PRISM_LIVE_WEB=1 npm run test:live -w @arnilo/prism-web-tools` plus least-privilege provider environment credential. Prefer these tools over `@arnilo/prism-browser` for ordinary public retrieval; use browser automation only for interactive/authenticated/JavaScript-heavy work behind a host egress proxy. Arbitrary HTML execution, model-selected providers, automatic OAuth forwarding, and generic web/MCP passthrough are unsupported.
+Default tests use injected fake fetch and make no public request. Restricted smoke: `PRISM_LIVE_WEB=1 npm run test:live -w @arnilo/prism-web-tools` plus least-privilege provider environment credential. Prefer the [`browser`](browser-automation.md) subpath over ordinary public retrieval; use browser automation only for interactive/authenticated/JavaScript-heavy work behind a host egress proxy. Arbitrary HTML execution, model-selected providers, automatic OAuth forwarding, and generic web/MCP passthrough are unsupported.
 
 ## Related APIs
 

package/docs/wiki.md

@@ -1,8 +1,8 @@
-# LLM Wiki (@arnilo/prism-wiki)
+# LLM Wiki (@arnilo/prism-memory/wiki)
 
 ## What it does
 
-`@arnilo/prism-wiki` implements Andrej Karpathy's **LLM Wiki Pattern** for the Prism agent ecosystem. It acts as a knowledge compiler that transforms raw, immutable sources (source code, AST symbols, notes, markdown clips, transcripts, journal entries) into a persistent, compounding, cross-linked Markdown knowledge base (`.wiki/`).
+The `@arnilo/prism-memory/wiki` subpath implements Andrej Karpathy's **LLM Wiki Pattern** for the Prism agent ecosystem. It acts as a knowledge compiler that transforms raw, immutable sources (source code, AST symbols, notes, markdown clips, transcripts, journal entries) into a persistent, compounding, cross-linked Markdown knowledge base (`.wiki/`).
 
 It integrates Tobias Lütke's [`qmd`](https://github.com/tobi/qmd) on-device hybrid search engine (BM25, vector search, and LLM reranking) and hydrates search results with Context7-inspired hierarchical breadcrumbs (`# Category > ## Topic`) and live clickable source line anchors (`file:///path/to/file#Lxx-Lyy` format), enabling agents and humans to navigate code and notes directly without blind regex loops (`grep`/`rg`).
 
@@ -19,7 +19,7 @@ The Karpathy LLM Wiki pattern is structured into 3 distinct tiers:
 
 1. **Raw Sources (Immutable)**: Source code files, design docs, transcripts, journals, and Markdown notes. Raw sources are strictly read-only and never mutated.
 2. **Compiled Wiki (`.wiki/`)**: Persistent, cross-linked Markdown documents containing synthesized architecture models, entity descriptions, decision records, and line-anchored claims.
-3. **Schema & Protocols (`SCHEMA.md`)**: Operational guidelines governing entity categorization, link formatting (`[[wikilink]]`), citation rules (`file:///path#Lxx-Lyy`), catalog indexing (`index.md`), and chronological change logging (`log.md`).
+3. **Schema & Protocols (`SCHEMA.md`)**: Operational guidelines governing OKF v0.2 emission, entity categorization, citation rules (`file:///path#Lxx-Lyy`), catalog indexing (`index.md`), and chronological change logging (`log.md`).
 
 ## Inputs / request
 
@@ -44,7 +44,7 @@ The Karpathy LLM Wiki pattern is structured into 3 distinct tiers:
 
 - `/wiki-init`: Scaffolds `.wiki/`, instantiates `SCHEMA.md`, `index.md`, and `log.md`, deploys skills, and adds the `qmd` collection.
 - `/wiki-refresh`: Detects modified source files via SHA-256 Merkle diffing, compiles updates to affected entity pages, reconciles contradictions in `log.md`, and runs `qmd update`.
-- `/wiki-lint`: Checks for broken `[[wikilinks]]`, dead line anchors, orphan pages, and unindexed symbols.
+- `/wiki-lint`: Checks OKF frontmatter (`type`, ISO `generated.at`), leftover `[[wikilinks]]`, unresolved relative markdown links, dead line anchors, and orphan pages.
 
 ### Standalone CLI Commands
 
@@ -81,7 +81,7 @@ npx prism-wiki search "How does authentication work?" --mode query
 ### Response Content:
 ```markdown
 ### Match 1: Authentication Architecture > Token Verification
-- **Wiki Page:** [[entities/authentication.md]]
+- **Wiki Page:** `entities/authentication.md`
 - **Category:** Core Module
 - **Freshness:** Current (Source hash matches manifest)
 
@@ -91,14 +91,14 @@ The authentication layer uses asymmetric Ed25519 JWT verification in middleware,
 **Code & Source Anchors (Clickable):**
 - Token verification: `verifyToken()` (`file:///src/auth/jwt.ts#L45-L89`)
 - Revocation check: `assertNotRevoked()` (`file:///src/auth/session-store.ts#L112-L138`)
-- Architecture Decision: [[decisions/ADR-004-ed25519-migration.md]]
+- Architecture Decision: `decisions/ADR-004-ed25519-migration.md`
 ```
 
 ## Implementation example
 
 ```ts
 import { createExtensionKernel } from "@arnilo/prism";
-import { createWikiExtension, initWiki, refreshWiki, lintWiki } from "@arnilo/prism-wiki";
+import { createWikiExtension, initWiki, refreshWiki, lintWiki } from "@arnilo/prism-memory/wiki";
 
 const kernel = createExtensionKernel();
 
@@ -112,16 +112,30 @@ await kernel.load([wiki]);
 
 ## Skills and Auto-Deployment
 
-`@arnilo/prism-wiki` includes two specialized skills formatted according to `.agents/skills/skill-creator`:
+The wiki subpath includes two specialized skills formatted according to `.agents/skills/skill-creator`:
 
 1. **`wiki-maintainer`**: Ingestion, compilation, line-anchor validation, and contradiction reconciliation rules.
 2. **`wiki-searcher`**: Context7 hierarchical breadcrumb query resolution, zero-grep instructions, and compounding insight recording.
 
 When initialized (`wiki-init` or `createWikiExtension`), these skills are automatically deployed to the host workspace's `.agents/skills/` folder so any compatible agent can leverage them immediately.
 
+## OKF v0.2 bundle format
+
+Emitted `.wiki/` trees are [OKF v0.2](https://github.com/GoogleCloudPlatform/open-knowledge-format) bundles. Karpathy compilation (synthesize, don't copy; precise `file:///` anchors; contradiction reconciliation; synchronized catalog/ledger) is unchanged.
+
+| Artifact | OKF rule |
+| --- | --- |
+| Root `index.md` | Only `okf_version: "0.2"` frontmatter; sectioned bullet listings per OKF §8 |
+| `entities/index.md`, `decisions/index.md`, `concepts/index.md` | No frontmatter (progressive disclosure) |
+| Concept pages | `type` required (Module / Concept / Decision Record / Entity / Person / Tool), plus `title`, `description`, `tags`, `sources[].resource`, `generated: { by: prism-wiki/<version>, at: <ISO 8601 UTC> }` |
+| `log.md` | `# Directory Update Log`, `## YYYY-MM-DD` newest first, `* **Verb**: …` |
+| Links | Standard relative markdown. `[[wikilinks]]` are lint errors |
+
+`.manifest.json` remains the compilation ledger (`id`, `category`, `rawSources`, `lastCompiledAt`). Those keys are not copied into page frontmatter. Trust families (`verified`, `status`) are omitted in v1 (unverified). `wiki-refresh` upgrades pages it touches; leftover legacy pages can be re-scaffolded — the format is regenerable from raw sources.
+
 ## Extension and configuration notes
 
-- `@arnilo/prism-wiki` registers tools (`wiki_search`, `wiki_read_page`, `wiki_record_insight`), commands (`wiki-init`, `wiki-refresh`, `wiki-lint`), skills (`wiki-maintainer`, `wiki-searcher`), and instruction injectors (`wiki-guidance`) into Prism registries.
+- The wiki subpath registers tools (`wiki_search`, `wiki_read_page`, `wiki_record_insight`), commands (`wiki-init`, `wiki-refresh`, `wiki-lint`), skills (`wiki-maintainer`, `wiki-searcher`), and instruction injectors (`wiki-guidance`) into Prism registries.
 - It operates with zero core modifications and can be used with any `@arnilo/prism` agent.
 - `qmd` is optional but recommended. When `@tobilu/qmd` is not installed, the search engine falls back to catalog matching against `index.md`.
 
@@ -134,7 +148,7 @@ When initialized (`wiki-init` or `createWikiExtension`), these skills are automa
 
 ## Related APIs
 
-- [`@arnilo/prism-rag`](rag.md): Bounded document chunking and vector context injection.
+- [`@arnilo/prism-memory/rag`](rag.md): Bounded document chunking and vector context injection.
 - [`@arnilo/prism-memory`](working-and-semantic-memory.md): Embedder and VectorStore primitives.
 - [`@arnilo/prism-coding-agent`](coding-agent-tools.md): Code manipulation and reading tools.
 - [`Contribution registries`](contribution-registries.md): Extension contribution model.

package/docs/workflow-orchestration-primitives.md

@@ -12,6 +12,8 @@ Interactive TUI (**C-012**) is **out of scope** for Plan 057 and deferred. Workf
 
 **Phase 11 addendum (2026-07-16):** schedules use a separate generic checkpoint namespace plus per-fire `LeaseStore` claims and deterministic queued-run IDs; SQLite/PostgreSQL need no workflow-specific table or migration. Background execution remains `enqueueWorkflow` + the existing coordinator. Nested workflow nodes call the same runner with inherited policy/ownership/checkpoint/event seams. Shared JSON state is validated, redacted, byte/history bounded, and checkpointed by version. Replay creates a new checkpoint with immutable source lineage and copied terminal evidence; approval-bearing prior paths cannot be copied.
 
+**Plan 045 Task 2 addendum (2026-08-31):** loop nodes keep one acyclic graph node while durable checkpoints append bounded, versioned iteration records. A tool sub-step can suspend before side effects; approved resume re-enters only the incomplete iteration. `node_iteration_started` / `node_iteration_finished` expose stable iteration IDs and bounded/redacted outputs. Replay starts a fresh cursor and emits new iteration events. A host saga treats the loop as one aggregate step and compensates its iteration IDs in reverse order; no workflow-specific SQL or implicit saga coupling is added.
+
 ## When to use it
 
 - **Workflow package authors** should start here, then follow [Agent/session runtime](agent-session-runtime.md), [Agent loops](agent-loops.md), [Runs and usage ledger](runs-and-usage.md), [CLI/RPC](cli-rpc.md), and [Database persistence](database-persistence.md).
@@ -139,6 +141,7 @@ import type { OwnershipScope, SecretRedactor } from "@arnilo/prism";
 
 /** Schema version for checkpoint payload layout (package-owned). */
 export const WORKFLOW_CHECKPOINT_SCHEMA_VERSION = 1 as const;
+export const WORKFLOW_LOOP_ITERATION_SCHEMA_VERSION = 1 as const;
 
 export type WorkflowRunStatus =
   | "queued"
@@ -158,6 +161,16 @@ export interface WorkflowNodeCheckpoint {
   readonly sessionId?: string;
   readonly leafId?: string;
   readonly runId?: string;
+  /** Optional additive loop cursor/ledger; absent on legacy checkpoints. */
+  readonly iteration?: number;
+  readonly lastOutput?: unknown;
+  readonly iterations?: readonly {
+    readonly schemaVersion: typeof WORKFLOW_LOOP_ITERATION_SCHEMA_VERSION;
+    readonly iteration: number;
+    readonly iterationId: string;
+    readonly done: boolean;
+    readonly output?: unknown;
+  }[];
 }
 
 export interface WorkflowCheckpointValue {
@@ -259,6 +272,17 @@ export type WorkflowEvent =
   | { readonly type: "workflow_finished"; readonly workflowId: string; readonly runId: string; readonly status: WorkflowRunStatus; readonly timestamp: string }
   | { readonly type: "node_started"; readonly workflowId: string; readonly runId: string; readonly nodeId: string; readonly timestamp: string }
   | { readonly type: "node_finished"; readonly workflowId: string; readonly runId: string; readonly nodeId: string; readonly timestamp: string }
+  | {
+      readonly type: "node_iteration_started" | "node_iteration_finished";
+      readonly workflowId: string;
+      readonly runId: string;
+      readonly nodeId: string;
+      readonly iteration: number;
+      readonly iterationId: string;
+      readonly done?: boolean;
+      readonly output?: unknown;
+      readonly timestamp: string;
+    }
   | { readonly type: "node_failed"; readonly workflowId: string; readonly runId: string; readonly nodeId: string; readonly error: { readonly message: string; readonly code?: string | number }; readonly timestamp: string }
   | { readonly type: "node_skipped"; readonly workflowId: string; readonly runId: string; readonly nodeId: string; readonly reason?: string; readonly timestamp: string }
   | { readonly type: "checkpoint_saved"; readonly workflowId: string; readonly runId: string; readonly version: number; readonly timestamp: string }