@arnilo/prism 0.11.0 → 0.12.0

package/docs/core.md

@@ -5,20 +5,19 @@ The `@arnilo/prism-core` family package unifies Prism's privileged runtime, sess
 ## Installation
 
 ```bash
-npm install @arnilo/prism @arnilo/prism-core
+bun add @arnilo/prism @arnilo/prism-core
 ```
 
-For database persistence and distributed event streams, install the required optional peer dependencies:
+SQLite persistence uses the Bun runtime's built-in `bun:sqlite`. No package to install.
 
-```bash
-# SQLite sessions & prompt storage
-npm install better-sqlite3
+For PostgreSQL and distributed event streams, install the optional peer:
 
+```bash
 # PostgreSQL sessions, enterprise persistence & prompt storage
-npm install pg
+bun add pg
 
 # NATS JetStream distributed event source
-npm install @nats-io/jetstream @nats-io/transport-node
+bun add @nats-io/jetstream @nats-io/transport-node
 ```
 
 Every peer below is optional and fails closed at first use; the [optional peer dependencies](peer-dependencies.md) matrix lists the exact ranges, pins, and which of them reach the network.
@@ -31,12 +30,12 @@ Every peer below is optional and fails closed at first use; the [optional peer d
 | `@arnilo/prism-core/runtime/supervisor` | Agent-to-Agent (A2A) protocol server, client, event source, and multi-agent supervisor | — |
 | `@arnilo/prism-core/runtime/workflows` | Multi-step DAG workflow coordinator, saga recovery, checkpoints, and loop nodes | — |
 | `@arnilo/prism-core/sessions/codecs` | Checkpoint, cursor, feedback, and search serialization codecs | — |
-| `@arnilo/prism-core/sessions/sqlite` | SQLite session store, leases, lifecycle, and schema migrations | `better-sqlite3` |
+| `@arnilo/prism-core/sessions/sqlite` | SQLite session store, leases, lifecycle, and schema migrations | — (`bun:sqlite`, built in) |
 | `@arnilo/prism-core/sessions/postgres` | PostgreSQL session store, event source, and migrations | `pg` |
 | `@arnilo/prism-core/sessions/nats` | NATS JetStream distributed event source | `@nats-io/jetstream`, `@nats-io/transport-node` |
 | `@arnilo/prism-core/governance/policy` | Capability admission, tool execution approvals, audit log exporter, and OPA evaluator | — |
 | `@arnilo/prism-core/governance/evals` | Offline evaluation runs, scorers, judges, threshold assertions, and trace curation | — |
-| `@arnilo/prism-core/governance/prompts` | Versioned prompt registry, promotion gating, rollback, and storage | `better-sqlite3`, `pg` |
+| `@arnilo/prism-core/governance/prompts` | Versioned prompt registry, promotion gating, rollback, and storage | `pg` (SQLite is `bun:sqlite`) |
 | `@arnilo/prism-core/governance/model-router` | Cost- and latency-aware model routing, token reservations, and failover | — |
 | `@arnilo/prism-core/governance/observability` | OpenTelemetry instrumentation and event tracing | `@opentelemetry/api` |
 | `@arnilo/prism-core/credentials/node` | Keyring-backed encrypted credential store, scrypt envelope encryption, OAuth2 PKCE providers, and OIDC identity verification | `@napi-rs/keyring` (bundled) |
@@ -81,7 +80,7 @@ const validator = createJsonSchemaToolArgumentValidator();
 
 ## Security & Import Isolation
 
-- Subpaths never load database drivers (`pg`, `better-sqlite3`) unless the specific database subpath is imported.
+- Subpaths never load the `pg` driver unless the specific database subpath is imported. SQLite uses the runtime's `bun:sqlite`, not a package.
 - All database and network drivers fail closed with clear actionable error messages when peers are omitted.
 - Root `@arnilo/prism` remains dependency-free contracts and CLI runner.
 - Messaging channels are `@arnilo/prism-channels` (`/telegram`, `/signal`), not a `@arnilo/prism-core` subpath.

package/docs/dev-inspector.md

@@ -17,7 +17,7 @@ Use it when iterating on prompts in a local Prism host and you want a inspectabl
 ### Quickstart — `prism dev` (plan 040 Task 4)
 
 ```bash
-npm install --save-dev @arnilo/prism-coding-tools/dev
+bun add --dev @arnilo/prism-coding-tools/dev
 cd my-agent && npm run dev   # → prism dev → http://127.0.0.1:4311
 ```
 

package/docs/disaster-recovery.md

@@ -29,7 +29,7 @@ This page is the operator runbook for backup, restore, migration rollback, point
 ```sh
 # Protected drill (standard tools only, orchestrated by the script):
 PRISM_PITR_URL=postgresql://user:***@localhost:55436/postgres \
-  node scripts/phase27-dr.test.mjs \
+  bun scripts/phase27-dr.test.mjs \
     --source "$PRISM_TEST_POSTGRES_URL" \
     --target postgresql://user:***@localhost:55432/prism_dr_target \
     --confirm-target prism_dr_restore

package/docs/document-extraction.md

@@ -0,0 +1,140 @@
+# Document extraction (`@arnilo/prism-work/document-extraction`)
+
+> **Optional peer install:** `bun add @arnilo/prism-work @firecrawl/anydoc@0.2.4`. Default hosts skip it. See [Optional peer dependencies](peer-dependencies.md).
+
+## What it does
+
+Converts in-memory Office, OpenDocument, RTF, EPUB, CSV, and text-PDF bytes to Markdown with a host-selected optional peer. Scanned PDFs throw `NeedsOcrError` unless the host injects a Docling runner. This subpath does not call Firecrawl Parse and does not read an API key.
+
+## When to use it
+
+Use when a host needs structured Markdown from a document byte buffer and can install `@firecrawl/anydoc`. Do not use it for the coding read tool (`document-reader`), for editable OOXML (`documents`), or for decimal-safe sheet ingest (`sheets`). Image OCR runs only when the host passes a runner and sets `ocr: true` on that call.
+
+## Inputs / request
+
+`createDocumentExtractor(options?)` loads the peer once. `extract({ bytes, filename?, signal? })` converts one buffer.
+
+| Option | Meaning | Default | Ceiling |
+| --- | --- | --- | --- |
+| `maxBytes` | Input cap. Oversize buffers refuse before the native call | 32 MiB | 512 MiB |
+| `maxTextBytes` | Markdown cap. Oversize results refuse; they are not truncated | 2 MiB | 64 MiB |
+| `maxPages` | OCR page cap. Checked before the runner when anydoc reported a count, and again on the worker result | 32 | 256 |
+| `ocrTimeoutMs` | Wall time the host runner must enforce. This module cannot kill a process | 120 s | 600 s |
+| `maxStderrBytes` | Worker stderr cap. Oversize or truncated stderr fails with no Markdown | 1 MiB | 4 MiB |
+| `ocr` | Host runner. Absent: scanned PDFs throw `NeedsOcrError` | none |  |
+
+Format comes from content. `filename` is used only when content has no signature and its basename ends with `.csv`. A path in `filename` is not opened. `extract({ ocr: true })` selects image OCR only; a text PDF that anydoc converts still skips the runner.
+
+## Outputs / response / events
+
+Success: `{ markdown, format, ocrUsed }`. `pages` is set only when Docling reported it. `ocrUsed` is false for anydoc. Markdown is untrusted data; label it at the wiki or RAG boundary that stores it.
+
+Errors: `DocumentExtractionError` (`code: "ERR_PRISM_DOCUMENT_EXTRACTION"`) with `reason`:
+
+| `reason` | When |
+| --- | --- |
+| `needsOcr` | `NeedsOcrError`. `pages` (1-indexed) and `pageCount`. No Markdown |
+| `unsupported` | No signature, and the name is not CSV |
+| `encrypted` / `malformed` / `resourceLimit` / `missingPart` / `io` | Peer refused. No fallback |
+| `hosted` | Peer reported a hosted-OCR failure. This API never requests hosted OCR |
+| `missingPeer` | `@firecrawl/anydoc` is not installed, at creation |
+| `inputLimit` / `outputLimit` | Cap exceeded. No text returned |
+| `busy` | One conversion is in flight and another is already waiting |
+
+Invalid caps throw `RangeError` at creation. An aborted `signal` throws `AbortError`.
+
+## Request/response example
+
+```json
+{
+  "format": "docx",
+  "ocrUsed": false,
+  "markdown": "Hello Prism DOCX\n"
+}
+```
+
+## Implementation example
+
+```ts
+import { createDocumentExtractor, NeedsOcrError } from "@arnilo/prism-work/document-extraction";
+
+const extractor = await createDocumentExtractor();
+try {
+  const { markdown, format } = await extractor.extract({ bytes, filename, signal });
+  // markdown is untrusted. Do not grant it tool authority.
+} catch (error) {
+  if (error instanceof NeedsOcrError) {
+    // pages and pageCount only. No partial Markdown.
+  }
+  throw error;
+}
+```
+
+Host OCR runner. Prism does not spawn this. Kill the child on abort; do not pass the process env through.
+
+```ts
+import { spawn } from "node:child_process";
+import { createDocumentExtractor, doclingOcrArgs, type OcrWorkerResult } from "@arnilo/prism-work/document-extraction";
+
+const extractor = await createDocumentExtractor({
+  ocr: (request) =>
+    new Promise<OcrWorkerResult>((resolve, reject) => {
+      const child = spawn(python, doclingOcrArgs(request, artifacts), {
+        env: { PATH: process.env.PATH ?? "", HF_HUB_OFFLINE: "1", TRANSFORMERS_OFFLINE: "1", HF_HUB_DISABLE_TELEMETRY: "1" },
+        stdio: ["pipe", "pipe", "pipe"],
+      });
+      const stdout: Buffer[] = [];
+      const stderr: Buffer[] = [];
+      const kill = () => child.kill("SIGKILL");
+      request.signal?.addEventListener("abort", kill, { once: true });
+      child.stdout.on("data", (chunk: Buffer) => stdout.push(chunk));
+      child.stderr.on("data", (chunk: Buffer) => stderr.push(chunk));
+      child.on("error", reject);
+      child.on("close", (exitCode) => {
+        resolve({ exitCode, stdout: Buffer.concat(stdout), stderr: Buffer.concat(stderr), aborted: request.signal?.aborted });
+      });
+      child.stdin.end(request.bytes);
+    }),
+});
+const scanned = await extractor.extract({ bytes: pdfBytes });
+const image = await extractor.extract({ bytes: pngBytes, ocr: true });
+```
+
+Wiki and RAG stay in the memory package. This adapter does not import it. Set `ocrImages` on both sides or images keep the wiki stub.
+
+```ts
+import { createDocumentExtractor, createDocumentIngest } from "@arnilo/prism-work/document-extraction";
+import { createWikiExtension } from "@arnilo/prism-memory/wiki";
+import { replaceDocument } from "@arnilo/prism-memory/rag";
+
+const extractor = await createDocumentExtractor({ ocr });
+const ingest = createDocumentIngest(extractor, { ocrImages: true });
+const wiki = createWikiExtension({ extractDocument: ingest.extractDocument, ocrImages: ingest.ocrImages });
+await replaceDocument({ uri, loader, parser: ingest.parser, store, scope, sourceId });
+// ingest.stats() is counts and milliseconds only. Markdown is untrusted at the wiki and RAG boundary.
+```
+
+## Extension and configuration notes
+
+- Peer version is pinned to `0.2.4`. The published wrapper accepts a third `options` argument that can upload the whole file. This module never passes it.
+- Docling is not an npm dependency. The package ships `docling/ocr.py`. The host pins `docling==2.130.0` and CPU torch (`https://download.pytorch.org/whl/cpu`; default PyPI torch pulls CUDA wheels), prefetches models (~1.4 GiB, about 150 s), and injects a runner. `doclingOcrArgs(request, artifactsPath)` builds the helper argv. Bytes go on stdin. A URL artifact path is rejected.
+- The runner must use a scrubbed env, close stdin after the bytes, kill the worker on abort and on stdout/stderr caps, and not leave the process running. `WorkSandbox.execFile` has no stdin, so it is not this runner. One OCR job in flight per extractor, same slot as anydoc. A third call fails with `busy`.
+- The helper accepts only `ConversionStatus.SUCCESS`. Partial, timeout, missing models, and over-page results are errors. Empty Markdown with a page count is valid. PDF and PNG Markdown are not byte-identical.
+- Native packages: darwin x64/arm64, linux x64/arm64 gnu and musl, win32 x64. No FreeBSD, Android, riscv, or Windows ARM build.
+- Importing `/documents`, `/sheets`, `/connectors`, or `/document-reader` does not load this peer.
+
+## Security and performance notes
+
+- Local conversion only. No `process.env` read, no hosted OCR, no process spawn, no macro execution, no filename filesystem read.
+- Upstream error text is not copied into `Error.message`. A peer message can contain document bytes.
+- One conversion in flight per extractor, plus one waiter. A third call fails with `busy`. Two extractors can run at once; share one if memory matters.
+- `AbortSignal` is checked before the native call and after it returns. It does not preempt the native call. The slot stays held until that call returns, then the result is discarded if the signal aborted.
+- Caps are checked on the input length before native code runs, and on UTF-8 output length after. Defaults match the document-reader defaults. Raising them toward the hard ceiling is host-owned: native work on a large file cannot be cancelled.
+- Small anydoc fixtures (under 10 KiB) on Bun 1.4.2 finished in a few milliseconds with process VmHWM under 64 MiB. The regression gate is 500 ms and 128 MiB for that class of input.
+- After model prefetch, one scanned page was about 5 s cold and 2 s warm, with VmHWM about 1.8–2 GiB. Two converters in one process reached about 3 GiB. Keep one worker. Budget for inputs under 100 KiB: under 30 s and 4 GiB VmHWM. Prefetch itself is host-owned and is not part of that budget.
+
+## Related APIs
+
+- [Document reader](document-reader.md): literal text for the coding read tool, including an opt-in Mistral OCR parser. Separate from this subpath.
+- [Documents, spreadsheets, and presentations](documents.md): editable OOXML model. Not a Markdown converter.
+- [Optional peer dependencies](peer-dependencies.md): install line for `@firecrawl/anydoc`.

package/docs/evaluations.md

@@ -323,7 +323,7 @@ if (records.some((record) => record.status !== "scored" || record.score !== 1))
 
 ```sh
 # Protected run (requires a disposable PostgreSQL instance):
-PRISM_TEST_POSTGRES_URL=postgresql://... node --test scripts/phase27-erp-journey.test.mjs
+PRISM_TEST_POSTGRES_URL=postgresql://... bun test scripts/phase27-erp-journey.test.mjs
 ```
 
 The journey reuses the two-replica failover worker (`scripts/phase27-ha-worker.mjs`) and asserts the comprehensive DR drill evidence (`docs/_evidence/phase27-dr-evidence.json`) is present and not stale. Local substitutes are labelled in the journey evidence and never converted into production claims: an in-memory WORM/SIEM sink (host owns the immutable store in production), in-memory saga checkpoint/lease stores (saga durability is proven in its own suite), and a logical pg-client backup/restore of the ERP tables (comprehensive PITR is in the DR drill evidence). Passing this protected journey **does not** satisfy the 0.3.0 live-service matrix.

package/docs/execution-timeline.md

@@ -211,6 +211,36 @@ const timeline = projectWorkflowTimeline(workflowEvents, {
 
 See runnable host demo in `examples/execution-timeline.ts` for offline workflow timeline projection, cockpit summary, and Mermaid diagram export.
 
+### Host step loops and ledger persistence
+
+When the host owns the step loop (one business step per `session.run()`), correlate its own
+`stepId` / `actionId` / `attemptId` with the Prism `sessionId` / `runId` in the host ledger envelope:
+Prism's events already carry those ids, so the timeline needs no extra correlation field. Project one
+incremental folder per step:
+
+```ts
+const subscription = session.subscribe(); // before run — see ordering note below
+const folder = createTimelineFolder({ content: "metadata" });
+const consume = (async () => {
+  for await (const event of subscription) folder.push(event);
+})();
+const result = await session.run(prompt); // result.sessionId / result.runId
+await consume;
+const timeline = folder.snapshot(); // per-step frozen projection
+```
+
+`subscribe()` registers synchronously, while `session.run()` emits `agent_started` before it returns —
+so start the subscription *before* the run, or the folder misses the run identity and `timeline.runId`
+stays empty. Persist the frozen timeline (default `metadata` policy) into the host ledger as
+newline-delimited JSON. Keep any legacy prose trace as display-only data beside the structural record:
+budgets, turns and stop reasons come from the typed fields, never from parsing prose. External host
+evidence (commits, verifications) is host authority, not a Prism effect — attach it to the ledger
+envelope with an explicit host marker instead of adding a timeline step.
+
+Runnable: [`examples/host-step-loop-timeline.ts`](../examples/host-step-loop-timeline.ts) — eight steps,
+one tool round persisting a `sha256:` argument hash instead of raw args, and external commit evidence
+on the last step.
+
 ### Stop reasons
 
 Run-level `stopReason` mirrors `agent_finished.finishReason` when the loop stopped on a ceiling or a host turn policy (`"host_policy"`); `status` reads `finished:<stopReason>` for those runs and `succeeded` for a natural end. `stopDetail` carries the host's `turnPolicy.stop` reason, bounded to 256 bytes and redacted at the runtime boundary. See [Runs and usage ledger § Clean stops and stop reasons](runs-and-usage.md#clean-stops-and-stop-reasons).

package/docs/guardrails.md

@@ -149,6 +149,59 @@ An explicit citation is `[evidence:<ref>]`, immediately after its claim (within
 
 With `onViolation: "block"`, the standard `GuardrailError` has `reason: "claim_ungrounded"` and bounded metadata `{ claim, contentIndex, start, end }` — never a full response body. `"flag"` returns `action: "allow"` plus that same metadata and `violation: true`, so the response stays visible while the normal `guardrail_decision` event and run ledger preserve the flag. Strict mode treats every standalone number (including dates, percentages, and versions) as a claim; use the option only where that law is wanted.
 
+## Field-evidence provenance
+
+`createFieldEvidenceGuardrail(options)` is a deterministic `tool_input` guardrail for one host-selected
+tool. It verifies that each required field carries a typed source reference and that the claimed
+value's revision and normalized value match the host's evidence set. It performs no retrieval, calls
+no model, and grants no authority: retrieval, grants, digest freeze, and commit-time validation stay
+with the host.
+
+```ts
+import { createFieldEvidenceGuardrail } from "@arnilo/prism";
+
+const provenance = createFieldEvidenceGuardrail({
+  toolName: "synapta:proposal",
+  required: ["amount", "vendor.id"],
+  evidence: ({ toolCallId }) => hostLedger.evidenceFor(toolCallId), // current rows only
+  normalize: (value) => (typeof value === "string" ? Number(value.replace(/[$,]/g, "")) : value),
+});
+
+const agent = createAgent({ model, provider, guardrails: { toolInput: [provenance] } });
+```
+
+A required field must hold a provenance envelope:
+
+```json
+{
+  "amount": { "value": 1250, "source": "tool:invoice_fetch", "path": "invoice.total", "revision": 7 }
+}
+```
+
+Evidence rows are `{ source, path, value, revision? }` — the source's current value. `source` and
+`path` are matched literally; `revision` must match exactly (a claim without a revision cannot match
+an evidence row that has one, and vice versa). After `normalize`, values compare exactly
+(`Object.is`), so a normalizer that returns objects never matches. `required` supports dot paths
+(`vendor.id`), bounded to 256 entries and 256 chars per path; evidence is capped at 4096 rows per call
+and a larger set fails closed instead of truncating.
+
+| Violation | Meaning |
+| --- | --- |
+| `missing_field` | The required path is absent from the call arguments. |
+| `malformed_claim` | The path holds a plain value or an incomplete envelope (missing `value`, `source`, or `path`; invalid `revision`). |
+| `missing_evidence` | The host evidence source returned no valid rows for this call. |
+| `unknown_source` | No evidence row matches the claimed `source` + `path` (invented field or wrong object). |
+| `stale_revision` | Matching rows exist but none carries the claimed revision. |
+| `value_mismatch` | A fresh row exists but its normalized value differs. |
+| `evidence_over_limit` | More than 4096 valid evidence rows; fails closed instead of truncating. |
+
+A violation blocks the call before lookup, permission, validation, or execution: the decision is
+`{ action: "block", reason: "field_evidence", metadata: { field, violation } }`, the blocked
+`ToolResult` stays neutral, and the `guardrail_decision` event carries only the field path and code —
+never the claimed value, evidence value, or any commit/approval semantics. Calls to other tools are
+allowed untouched. The host decides which fields require evidence and supplies it per call; the
+guardrail never widens, rewrites, or commits anything.
+
 ## Extension and configuration notes
 
 Guardrails are callbacks supplied by the host. Prism does not discover, load, retry, or persist callback code. `createSecureAgent()` keeps configured guardrails and only appends run-level checks; it never lets a run remove secure defaults. Custom loops receive guarded `LoopContext.generate()` and `LoopContext.dispatchToolCall()`; host code that directly calls a provider or `ToolDefinition.execute()` is outside the runtime boundary. Guardrail packs follow the same rule: they are host-supplied config, compiled in memory per session, never discovered from disk. Their compiled identity and pack-owned state persist only inside an opt-in durable checkpoint (`persistSessionState`, see above) and nowhere else.