@arnilo/prism 0.3.2 → 0.5.0

package/docs/documents.md

@@ -0,0 +1,214 @@
+# Documents, spreadsheets, and presentations (`@arnilo/prism-office/documents`)
+
+## What it does
+
+The `@arnilo/prism-office/documents` package provides specification-compliant, AI-native OpenXML document generation, parsing, patching, and bounded preview rendering for Microsoft Word (`.docx`), Microsoft Excel (`.xlsx`), and Microsoft PowerPoint (`.pptx`) artifacts.
+
+It operates on a canonical, typed abstract syntax tree (AST) called the **Prism Document Model** (`DocModel`, `SheetModel`, `DeckModel`):
+- **Pure in-memory doctrine**: Functions accept `Uint8Array` container buffers or typed model objects and emit `Uint8Array` buffers or JSON models. Zero filesystem reads, zero network I/O, zero `process.env` lookups, and zero child process spawns.
+- **Draft-07 JSON Schema validation & slicing**: Full runtime structural validation with transitive closure slicing (`getDocumentModelSchema`) allowing LLM tools and agent prompts to extract minimal, self-contained sub-schemas (e.g. `doc.paragraph`, `doc.table`).
+- **Bidirectional round-trip fidelity**: Prism-generated documents parse back into structurally equivalent models verified against a per-kind equality specification.
+- **Typed model patch engine**: Immutably applies `set`, `insert`, `remove`, and `move` operations to document blocks, worksheet cells, and presentation slides with schema re-validation and an interactive `createPatchHistory` undo/redo stack.
+- **Framework-neutral preview blocks & bounded HTML**: Generates structured snapshots (`PreviewBlock[]`) for native desktop/web UI grids and outline trees, as well as safe, sanitize-by-construction HTML fragments (`renderPreviewHtml`) guaranteed to contain no executable scripts, no active pseudo-protocols, and no external hyperlinks.
+- **Boundary text redaction**: Pluggable `SecretRedactor` hook to sanitize extracted text content (paragraphs, cells, notes, tables) at the parse boundary before models are returned.
+- **Dependency-free telemetry seam**: Zero-overhead `DocumentsTelemetry` hook for OpenTelemetry-compatible span tracing without leaking document or chunk text into telemetry collectors.
+
+## When to use it
+
+Use `@arnilo/prism-office/documents` whenever autonomous agents, coding assistants, workflow orchestrators, or enterprise applications need to:
+1. Synthesize professional DOCX reports, financial XLSX spreadsheets, or PPTX presentation decks from structured LLM outputs.
+2. Ingest existing OOXML artifacts into a structured, validated document model for analysis or automated summarization.
+3. Perform atomic, validated updates or localized edits to documents using typed patch operations.
+4. Render safe, bounded HTML previews or framework-neutral outline and grid snapshots in web and desktop hosts.
+
+Do **not** use this package for collaborative real-time editing (OT/CRDT), macro execution, or in-memory spreadsheet formula evaluation (formulas are preserved verbatim as `{ formula, cachedValue? }` read-only pairs).
+
+## Inputs / request
+
+### Generation & Parsing Functions
+
+| Function | Signature | Description |
+| --- | --- | --- |
+| `generateDocument` | `(model: DocumentModel, options: GenerateDocumentOptions) => Promise<GenerateDocumentResult>` | Translates a typed model into spec-compliant OOXML binary bytes (PK zip container) with a SHA-256 content hash. |
+| `parseDocument` | `(bytes: Uint8Array, options: ParseDocumentOptions) => Promise<DocumentModel>` | Verifies PK zip signature, enforces caps, translates OOXML parts, applies optional redaction, and returns a validated model. |
+| `patchDocument` | `(model: DocumentModel, patches: readonly DocumentPatch[], options?: PatchDocumentOptions) => DocumentModel` | Clones the model, applies typed structural patch operations, and validates the resulting model against Draft-07 schemas. |
+| `createPatchHistory` | `(initialModel: DocumentModel) => PatchHistory` | Creates an interactive undo/redo history manager for host editing workflows. |
+| `renderPreviewBlocks` | `(model: DocumentModel, options?: PreviewBlocksOptions) => PreviewBlock[]` | Emits framework-neutral structured blocks (document outlines, bounded sheet grid chunks, slide summaries). |
+| `renderPreviewHtml` | `(model: DocumentModel, options?: PreviewHtmlOptions) => string` | Emits safe, bounded HTML fragments with all entities escaped and external URLs neutralized. |
+| `getDocumentModelSchema`| `(options: GetDocumentModelSchemaOptions) => Record<string, unknown>` | Retrieves full Draft-07 JSON Schema or a self-contained sliced sub-schema with resolved `$defs`. |
+| `validateDocumentModel`| `(model: unknown) => asserts model is DocumentModel` | Validates arbitrary JSON objects against Draft-07 document schemas and structural invariants. |
+
+### Capacity Limits and Defaults
+
+Caps are strictly enforced in memory before compute-intensive translation or parsing:
+
+| Cap | Default | Hard Ceiling | Target |
+| --- | --- | --- | --- |
+| `maxBytes` | 32 MiB (`33,554,432`) | 512 MiB (`536,870,912`) | Binary input buffer & generated OOXML bytes |
+| `maxBlocks` | 10,000 | 50,000 | Total blocks in a `DocModel` |
+| `maxSheets` | 50 | 250 | Worksheets in a `SheetModel` |
+| `maxRows` | 100,000 | 1,000,000 | Rows per worksheet |
+| `maxColumns` | 1,000 | 16,384 | Columns per worksheet |
+| `maxCells` | 1,000,000 | 10,000,000 | Total cells across all worksheets |
+| `maxSlides` | 500 | 2,000 | Total slides in a `DeckModel` |
+| `maxParagraphs` | 10,000 | 50,000 | Total paragraphs in a document |
+| `maxRuns` | 50,000 | 250,000 | Total formatted text runs in a document |
+
+## Outputs / response / events
+
+### Error Hierarchy
+
+All package operations throw typed exceptions derived from `DocumentsError`:
+
+| Error Class | Error Code | Trigger Condition |
+| --- | --- | --- |
+| `DocumentsValidationError` | `ERR_PRISM_DOCUMENTS_INVALID_MODEL` | Input JSON fails Draft-07 schema validation, has dimension mismatches, or invalid decimal string syntax. |
+| `DocumentsCapExceededError` | `ERR_PRISM_DOCUMENTS_CAP_EXCEEDED` | Document exceeds byte size, block count, cell count, or slide count caps. |
+| `DocumentsParseError` | `ERR_PRISM_DOCUMENTS_PARSE_FAILED` | Input buffer lacks PK zip signature, is corrupted, or fails OOXML part parsing. |
+| `DocumentsFormatError` | `ERR_PRISM_DOCUMENTS_UNSUPPORTED_FORMAT` | Format mismatch (e.g. attempting to generate PPTX from a `DocModel`). |
+| `DocumentsPatchError` | `ERR_PRISM_DOCUMENTS_UNSAFE_PATH` | `__proto__`, `prototype`, or `constructor` appears as a `metadata` target or inside a block/slide/sheet `patch` segment (rejected before mutation to prevent `Object.prototype` pollution). |
+| `DocumentsPatchError` | `ERR_PRISM_DOCUMENTS_PATCH_FAILED` | Out-of-bounds index target, unknown patch operation, or invalid patch structure. |
+
+## Request/response example
+
+### JSON Schema Document Models
+
+```json
+{
+  "kind": "sheet",
+  "modelVersion": 1,
+  "title": "Q3 Financial Summary",
+  "sheets": [
+    {
+      "name": "Revenue",
+      "columnWidths": [{ "column": 0, "width": 25 }, { "column": 1, "width": 15 }],
+      "frozenPanes": { "rows": 1, "columns": 0 },
+      "cells": [
+        ["Department", "Operating Budget", "Actual Expense"],
+        ["Engineering", { "type": "decimal", "value": "1500000.00" }, { "type": "decimal", "value": "1420000.50" }],
+        ["Operations", { "type": "decimal", "value": "450000.00" }, { "type": "decimal", "value": "435000.00" }],
+        ["Total", { "formula": "=SUM(B2:B3)", "cachedValue": 1950000 }, { "formula": "=SUM(C2:C3)", "cachedValue": 1855000.5 }]
+      ]
+    }
+  ]
+}
+```
+
+## Implementation example
+
+```ts
+import {
+  generateDocument,
+  parseDocument,
+  patchDocument,
+  createPatchHistory,
+  renderPreviewBlocks,
+  renderPreviewHtml,
+  type DocModel,
+} from "@arnilo/prism-office/documents";
+
+// 1. Define typed document model
+const doc: DocModel = {
+  kind: "doc",
+  modelVersion: 1,
+  title: "Architecture Review",
+  blocks: [
+    { type: "heading", level: 1, text: "System Architecture" },
+    {
+      type: "paragraph",
+      runs: [
+        { text: "Prism operates purely in memory with ", bold: false },
+        { text: "zero-trust security boundaries.", bold: true, italic: true },
+      ],
+    },
+    {
+      type: "table",
+      rows: 2,
+      columns: 2,
+      cells: [
+        ["Module", "Latency"],
+        ["Parser", "12ms"],
+      ],
+    },
+  ],
+};
+
+// 2. Generate in-memory DOCX binary with SHA-256 contentHash
+const { bytes, contentHash } = await generateDocument(doc, { format: "docx" });
+console.log(`Generated DOCX (${bytes.byteLength} bytes, SHA-256: ${contentHash})`);
+
+// 3. Parse OOXML bytes back to a validated model
+const parsed = await parseDocument(bytes, { kind: "doc" });
+
+// 4. Apply typed model patches
+const patched = patchDocument(parsed, [
+  { op: "set", target: { block: 0 }, block: { type: "heading", level: 1, text: "Executive Summary" } },
+  { op: "insert", target: { afterBlock: 0 }, block: { type: "paragraph", text: "New introduction." } },
+]);
+
+// 5. Interactive Undo/Redo editing
+const history = createPatchHistory(patched);
+history.apply([{ op: "set", target: { title: true }, value: "Updated Review" }]);
+console.log(history.canUndo()); // true
+const restored = history.undo(); // restored to "Executive Summary" state
+
+// 6. Generate structured preview blocks & safe HTML
+const blocks = renderPreviewBlocks(restored);
+const htmlSnippet = renderPreviewHtml(restored, { maxHtmlBytes: 256 * 1024 });
+```
+
+## Extension and configuration notes
+
+### Sub-package Pinning
+To avoid pulling in CLI frameworks or extraneous dependencies, `@arnilo/prism-office/documents` directly pins the exact underlying modular packages:
+- `@office-open/docx@0.12.3`
+- `@office-open/xlsx@0.12.3`
+- `@office-open/pptx@0.12.3`
+
+### Draft-07 JSON Schema Slicing
+For AI agent tool generation where token budgets are constrained, `getDocumentModelSchema` provides closure slicing:
+```ts
+// Returns only TableBlock, CellValue, and their transitively required definitions in $defs
+const tableSchema = getDocumentModelSchema({ kind: "doc", slice: "table" });
+```
+
+### Text Redaction Hook (`SecretRedactor`)
+When parsing untrusted or sensitive OOXML containers, hosts can inject a redaction hook to scrub PII or secrets before the document model enters memory:
+```ts
+const sanitizedModel = await parseDocument(rawBytes, {
+  kind: "doc",
+  redactor: {
+    redact: (text: string) => text.replace(/\b\d{3}-\d{2}-\d{4}\b/g, "[REDACTED-SSN]"),
+  },
+});
+```
+
+### Telemetry Seam
+The dependency-free `DocumentsTelemetry` seam allows optional OpenTelemetry instrumentation without adding runtime telemetry dependencies:
+```ts
+const telemetry: DocumentsTelemetry = {
+  startSpan(name, attributes) {
+    // Maps to tracer.startSpan with allow-listed metadata (kind, format, bytes, counts)
+    // Model content and document text are NEVER passed to telemetry spans.
+    return activeSpan;
+  },
+};
+```
+
+### Decimal Fidelity Ceiling
+Financial worksheets often require exact decimal representations that JavaScript 64-bit binary floating-point numbers cannot represent without precision loss. `@arnilo/prism-office/documents` supports canonical string decimals (`{ type: "decimal", value: "1500000.00" }`), preserving exact numerical formatting without IEEE 754 precision artifacts.
+
+## Security and performance notes
+
+- **Pure In-Memory Operation**: No temporary files, no shell execution, no binary spawning, and zero network sockets.
+- **ZIP Signature Gating**: Buffers must begin with PK zip container signatures (`0x50, 0x4B, 0x03, 0x04`). Extension-based type inference is strictly prohibited.
+- **Fail-Closed Caps**: Input size and element count caps are evaluated before entering XML translation passes, preventing zip-bomb and decompression amplification attacks.
+- **Sanitize-by-Construction HTML**: `renderPreviewHtml` strictly entity-encodes all text fields, neutralizes dangerous protocols (`javascript:`, `http://`, `https://`), strips raw script/image tags, and caps output size to prevent DOM-based XSS and memory exhaustion.
+- **Performance Budget**: Warm generation of 200-block documents completes in under 15 ms; parse and round-trip equality checks complete in under 100 ms.
+
+## Related APIs
+
+- [`@arnilo/prism-coding-tools/document-reader`](./document-reader.md): Bounded literal text extraction from PDF and DOCX documents for coding agent tools.
+- [`@arnilo/prism-core/integrations/work`](./work-tools.md): Microsoft 365 and Google Workspace identity-scoped connectors.
+- [`@arnilo/prism-coding-tools/agent`](./coding-agent-tools.md): Coding tools and file operations.
+- [`@arnilo/prism-core/governance/observability`](./observability.md): OpenTelemetry instrumentation and trace adapters.

package/docs/embeddings.md

@@ -0,0 +1,112 @@
+# Embeddings
+
+## What it does
+
+`EmbeddingsProvider` is the provider-neutral embeddings contract: one-shot batch
+text→vector generation with usage accounting, per-item error mapping, and
+capability-gated models. First-party adapters ship in
+[`@arnilo/prism-providers/openai`](providers/openai.md) (OpenAI-compatible
+`POST {base}/embeddings`) and [`@arnilo/prism-providers/alibaba`](providers/alibaba.md)
+(DashScope `compatible-mode/v1/embeddings`); offline conformance runs via
+`runEmbeddingsConformance` from `@arnilo/prism/testing/provider-conformance`.
+
+## When to use it
+
+Use it when a host owns its embedding pipeline (RAG ingestion, memory indexing,
+semantic search) and wants a portable contract instead of per-provider HTTP code.
+Do not use it as a live integration runner: the contract is caller-gated, adapters
+perform no network on construction, and batch caps are enforced with typed errors
+rather than silent auto-chunking — chunk callers themselves (for example,
+`@arnilo/prism-memory`'s `embedBatched`).
+
+## Inputs / request
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `model` | `string` | Embedding model id, e.g. `text-embedding-3-small` or `text-embedding-v4`. |
+| `inputs` | `readonly string[]` | Texts to embed; order is preserved on `result.vectors`. Non-empty and within the provider batch cap. |
+| `dimensions` | `number?` | Output-dimensions override; only for models that support reduced dimensions (OpenAI `dimensions` param, DashScope 64–2048). |
+| `signal` | `AbortSignal?` | Cancellation; observed by the adapter transport. |
+
+Adapter options: `apiKey` (`CredentialValueSource` — the existing credential seam,
+resolved per call and redacted from errors), `baseUrl`/`preset` (Alibaba),
+`fetch` (inject a fake transport for offline tests), `headers`, and default
+`dimensions`/`encodingFormat` where the provider supports them.
+
+## Outputs / response / events
+
+| Field | Type | Meaning |
+| --- | --- | --- |
+| `vectors` | `readonly (readonly number[])[]` | Vectors in input order; `vectors[i]` corresponds to `inputs[i]`. |
+| `usage` | `Usage` | Provider-reported token usage (`inputTokens`, `totalTokens`); empty object when the provider omits it. |
+| `dimensions` | `number` | Actual vector dimensionality reported by the response. |
+
+Failures throw `EmbeddingsError` with a stable `code`:
+`empty_input` (no inputs), `batch_too_large` (over the provider cap — OpenAI 2048,
+DashScope 10), `request_failed` (non-2xx, secret-redacted message),
+`response_malformed` (missing index, wrong dimensionality), `unsupported_model`
+(via `assertEmbeddingsSupported` when the host checks
+`ModelCapabilities.embeddings`).
+
+## Request/response example
+
+```json
+{ "model": "text-embedding-3-small", "input": ["hello", "world"], "dimensions": 256 }
+```
+
+## Implementation example
+
+```ts
+import { createOpenAIEmbeddingsProvider } from "@arnilo/prism-providers/openai";
+import { runEmbeddingsConformance } from "@arnilo/prism/testing/provider-conformance";
+
+const embeddings = createOpenAIEmbeddingsProvider({ apiKey: process.env.OPENAI_API_KEY });
+const result = await embeddings.embedMany({
+  model: "text-embedding-3-small",
+  inputs: ["hello", "world"],
+});
+// result.vectors.length === 2; result.usage.inputTokens reported
+
+// Offline conformance (fake transport, no network):
+await runEmbeddingsConformance({
+  provider: createOpenAIEmbeddingsProvider({ apiKey: "sk-test", fetch: fakeFetch }),
+  model: "text-embedding-3-small",
+  maxBatchSize: 2048,
+  sample: { inputs: ["a", "b"], dimensions: 2 },
+});
+```
+
+## Extension and configuration notes
+
+- Implement `EmbeddingsProvider` (`{ id, embedMany }`) for other vendors; the
+  contract is structural — no base class, no registry.
+- Models declare support with `capabilities.embeddings`; hosts gate with
+  `modelSupportsEmbeddings(capabilities)` / `assertEmbeddingsSupported(model)`,
+  mirroring the structured-output guard pattern.
+- `@arnilo/prism-memory` keeps its dependency-free `Embedder` host seam; adapters
+  bridge structurally (`createAlibabaEmbedder` remains assignable to `Embedder`
+  without importing it). The contract is a superset: it adds usage and per-item
+  error mapping.
+- Adapters never auto-chunk: a batch over the provider cap rejects with
+  `batch_too_large`, so `embedBatched`-style callers own batching and preserve
+  per-item error attribution.
+
+## Security and performance notes
+
+- API keys resolve through the existing `CredentialValueSource` seam and are
+  redacted from every thrown error (`redactSecrets`); no new secret paths.
+- Responses are read through the bounded transport (`readBoundedResponseJson` /
+  `readBoundedResponseText`) — response bodies cannot exhaust memory.
+- Input text is never logged; error messages carry status and redacted body only.
+- One HTTP request per `embedMany` call; response mapping allocates one vector
+  copy per input and nothing else. Per-item token caps are server-enforced;
+  the local cap is batch count.
+
+## Related APIs
+
+- [`@arnilo/prism-memory`](working-and-semantic-memory.md): `Embedder` host seam and `embedBatched` —
+  consumption side of the structural bridge.
+- [Provider conformance](provider-conformance.md): `runEmbeddingsConformance` and
+  the offline conformance matrix this contract joins.
+- [Provider packages](provider-packages.md): subpath import rules for
+  `@arnilo/prism-providers/openai` and `@arnilo/prism-providers/alibaba`.

package/docs/enterprise-postgres-state.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-enterprise-postgres` is one optional PostgreSQL composition for existing enterprise state seams:
+`@arnilo/prism-core/enterprise/postgres` is one optional PostgreSQL composition for existing enterprise state seams:
 
 | State | Composition property | Durable behavior |
 | --- | --- | --- |
@@ -14,7 +14,7 @@
 | ERP messaging | `erpMessaging` | Transactional outbox/inbox markers plus bounded, tenant-scoped at-least-once dispatch (migration 004). |
 | Multi-party approvals | `createPostgresApprovalStore({ pool, schema, authority })` | Immutable approval requests, role/quorum decisions, revocation, bounded delegation, and atomic grant consumption (migration 005). |
 
-`createPostgresEnterpriseState()` opens a host-supplied or adapter-owned `pg` pool, verifies/applies checksum-protected enterprise migrations (`001_enterprise_state`, `002_tool_effects`, `003_router_reservations`, `004_erp_messaging`, `005_erp_approvals`), and returns those stores plus explicit cleanup and close operations. Importing it performs no I/O. It is separate from session/run persistence in [`@arnilo/prism-session-store-postgres`](postgres-persistence.md).
+`createPostgresEnterpriseState()` opens a host-supplied or adapter-owned `pg` pool, verifies/applies checksum-protected enterprise migrations (`001_enterprise_state`, `002_tool_effects`, `003_router_reservations`, `004_erp_messaging`, `005_erp_approvals`), and returns those stores plus explicit cleanup and close operations. Importing it performs no I/O. It is separate from session/run persistence in [`@arnilo/prism-core/sessions/postgres`](postgres-persistence.md).
 
 ## When to use it
 
@@ -25,7 +25,7 @@ Use memory/file stores only for tests, demos, or a deliberately single-process h
 ## Inputs / request
 
 ```ts
-import { createPostgresEnterpriseState } from "@arnilo/prism-enterprise-postgres";
+import { createPostgresEnterpriseState } from "@arnilo/prism-core/enterprise/postgres";
 import { Pool } from "pg";
 
 const pool = new Pool({
@@ -95,7 +95,7 @@ A migration creates `prism_policy_decisions`, `prism_evaluations`, `prism_work_i
 ## Implementation example
 
 ```ts
-import { createPostgresErpMessaging } from "@arnilo/prism-enterprise-postgres";
+import { createPostgresErpMessaging } from "@arnilo/prism-core/enterprise/postgres";
 
 const messaging = createPostgresErpMessaging({ pool, schema: "prism" });
 const client = await pool.connect();
@@ -141,7 +141,7 @@ await messaging.dispatcher.replay({
 
 ```ts
 import type { AgentIdentity } from "@arnilo/prism";
-import { createPostgresEnterpriseState, type PostgresEnterpriseState } from "@arnilo/prism-enterprise-postgres";
+import { createPostgresEnterpriseState, type PostgresEnterpriseState } from "@arnilo/prism-core/enterprise/postgres";
 
 const identity: AgentIdentity = {
   tenantId: "tenant-1",
@@ -202,13 +202,13 @@ export async function recordEnterpriseState(state: PostgresEnterpriseState) {
 
 ## Extension and configuration notes
 
-- `createModelRouter({ resolver, stateStore: state.modelRouter })` keeps allow-list, residency, fallback, and diagnostics behavior in `@arnilo/prism-model-router`; this package only supplies durable state. Router admission reservations (`reserveBudget`/`commitBudget`/`releaseBudget` on `state.modelRouter`) live in the `reservations` JSONB column of `prism_model_router_budgets`: one atomic UPSERT per admission, fencing-token-guarded commit/release in a SERIALIZABLE transaction, and TTL reconciliation as unknown usage; see [Model routing](model-routing.md).
+- `createModelRouter({ resolver, stateStore: state.modelRouter })` keeps allow-list, residency, fallback, and diagnostics behavior in `@arnilo/prism-core/governance/model-router`; this package only supplies durable state. Router admission reservations (`reserveBudget`/`commitBudget`/`releaseBudget` on `state.modelRouter`) live in the `reservations` JSONB column of `prism_model_router_budgets`: one atomic UPSERT per admission, fencing-token-guarded commit/release in a SERIALIZABLE transaction, and TTL reconciliation as unknown usage; see [Model routing](model-routing.md).
 - `createPostgresErpMessaging({ pool, schema? })` is the direct messaging composition. `outbox.append` and `inbox.record` accept a caller-owned `PoolClient`; the host must put them in the same transaction as its local mutation. The dispatcher owns only short claim/transition transactions and never invokes business callbacks or stores executable handlers.
 - `createPostgresApprovalStore({ pool, schema?, authority })` is the direct approval composition (migration 005). `authority.resolveRoles(actor, request)` and `policyRevision` are host-owned; Prism persists only accepted role grants and delegation chains. `decide`/`revoke` lock the request row and revision-check the terminal transition in one transaction. `consume` accepts an optional caller-owned `client`; grant consumption and the protected action commit (or roll back) together.
 - Rate/budget/circuit tables are capped like the memory store: `consumeRate`/`readBudget`/`addUsage`/`reserveBudget` accept `maxRateKeys`/`maxBudgetKeys` (the router passes its resolved limits) and evict the least-recently-used row on new-key insert — never the row just inserted, never a budget row holding an active reservation — else fail closed with `ERR_PRISM_MODEL_ROUTER_STATE`. Cleanup prunes expired reservations within its bounded batch.
 - Policy/evaluation/query public contracts stay in their owning packages. This package exports `createPostgresEnterpriseState`, `createPostgresApprovalStore`, `createPostgresErpMessaging`, their options/result/types, and `EnterprisePostgresError`; it has no SQL, DDL, codec, queryable, or migration subpath.
 - The fixed schema has no generic key/value table and no background cleanup scheduler. Schedule `state.cleanup()` from an authorized host job, size its bounded batch for the deployment, and monitor unknown work rows for reconciliation. Run protected integration checks with `PRISM_TEST_POSTGRES_URL="$DATABASE_URL" npm run test:postgres`; the command rejects an absent URL instead of silently skipping database coverage.
-- The OPA adapter (`@arnilo/prism-policy/opa`, 0.0.28) records decisions into the same `state.policy` store unchanged via `evaluateAndAppend` — see [Policy and audit](policy-and-audit.md#opa-external-policy-adapter-arniloprism-policyopa-008).
+- The OPA adapter (`@arnilo/prism-core/governance/policy/opa`, 0.0.28) records decisions into the same `state.policy` store unchanged via `evaluateAndAppend` — see [Policy and audit](policy-and-audit.md#opa-external-policy-adapter-arniloprism-policyopa-008).
 - Request-path state SQL uses `SELECT`, `INSERT`, `UPDATE`, and `DELETE` on the eight state tables. The open/migration lifecycle additionally needs schema/catalog/advisory-lock and DDL permissions. Use a deployment migration principal for that lifecycle and a least-privilege request role for request traffic; this release intentionally does not ship a migration CLI or worker.
 
 ## Security and performance notes

package/docs/evaluations.md

@@ -2,11 +2,11 @@
 
 ## What it does
 
-`@arnilo/prism-evals` adds optional deterministic scorers, immutable datasets, bounded persistence-trace grading, explicit host model judges, pairwise comparisons, CI thresholds, live post-run scoring, and batch experiments over `AgentRunResult`. Scores are finite numbers in `[0, 1]` with optional reason/metadata and linkage to run/session/trace/experiment IDs.
+`@arnilo/prism-core/governance/evals` adds optional deterministic scorers, immutable datasets, bounded persistence-trace grading, explicit host model judges, pairwise comparisons, CI thresholds, live post-run scoring, and batch experiments over `AgentRunResult`. Scores are finite numbers in `[0, 1]` with optional reason/metadata and linkage to run/session/trace/experiment IDs.
 
 ## When to use it
 
-Use this package when a host needs offline quality checks or sampled live scoring without coupling scorers into core agent execution. Install it directly or through `@arnilo/prism-all`; installation does not attach scorers to runs.
+Use this package when a host needs offline quality checks or sampled live scoring without coupling scorers into core agent execution. Install it directly or through the `@arnilo/prism-core` family package; installation does not attach scorers to runs.
 
 ## Inputs / request
 
@@ -19,6 +19,7 @@ Use this package when a host needs offline quality checks or sampled live scorin
 | `createMemoryEvaluationStore` | optional seed records |
 | `appendEvaluationFeedback` | `RunFeedbackStore`, `EvaluationStore`, feedback fields, and 1–64 known evaluation IDs |
 | `createPersistenceTraceResolver` | explicit `ProductionPersistenceStore`, exact session/run/ownership, page/byte bounds |
+| `datasetFromRuns` | `runIds` and/or `sessionIds`, existing dataset, `ProductionPersistenceStore`, ownership, redactor/secrets, optional `toItem` |
 | `createModelJudge` | host judge callback, stable rubric/version, timeout/attempt/output bounds |
 | `runComparison` | immutable dataset, 2–8 named candidates by default, pairwise scorers |
 | `assertEvaluationThreshold` / `serializeEvaluationReport` | mean/failure/per-scorer gates and bounded redacted JSON |
@@ -32,6 +33,7 @@ Use this package when a host needs offline quality checks or sampled live scorin
 | `runExperiment` | `ExperimentReport` with stable item order, evaluations, and aggregates |
 | `EvaluationStore.query` | cursor-paginated, ownership-filtered page |
 | `appendEvaluationFeedback` | immutable `RunFeedbackRecord` containing only evaluation/scorer IDs |
+| `datasetFromRuns` | `{ dataset, version, added, skipped }` — new immutable dataset version with one item per added run; skips carry reasons (missing run, ownership mismatch, empty output) |
 
 ## Request/response example
 
@@ -58,7 +60,7 @@ import {
   defineScorer,
   runExperiment,
   scoreRunLive,
-} from "@arnilo/prism-evals";
+} from "@arnilo/prism-core/governance/evals";
 
 const scorer = defineScorer({
   id: "contains-citation",
@@ -121,6 +123,7 @@ console.log(report.aggregate.meanScore, linked.evaluationIds);
 - Model judges are host callbacks, not providers: Prism passes rubric/version plus bounded target only—never credential resolvers, tools, or workspace. Defaults are one attempt, 30 seconds, and 16 KiB output; failures become redacted evaluation records.
 - Pairwise candidates are sorted by name, executed once per item, compared in stable item/pair/scorer order, and record ties/failures without choosing a winner. Candidate and scorer outputs have byte caps.
 - `assertEvaluationThreshold()` throws `ERR_PRISM_EVAL_THRESHOLD`; an uncaught error gives CI a non-zero exit. Keep model-judge/live gates credential-gated and outside the network-free default suite. `serializeEvaluationReport()` bounds/redacts checked-in artifacts.
+- `datasetFromRuns` redacts before item construction and never stores an unredactable item; curated items are byte-capped (`ERR_PRISM_EVAL_CURATE`), ownership is verified exactly per run, and a failed append leaves the prior dataset version untouched.
 
 ## Trace, judge, comparison, and CI example
 
@@ -137,14 +140,45 @@ assertEvaluationThreshold(report, { minimumMean: 0.9, maximumFailures: 0 });
 
 `traceResolver` is explicit; no arbitrary run search occurs. `baseline`/`candidate` are host functions returning `AgentRunResult`. See `examples/evaluation-gate.ts` for a network-free gate and `examples/coding-browser-evaluation.ts` for coding/browser adversarial fixtures.
 
+## Curating datasets from production runs
+
+Plan 043 adds `datasetFromRuns`: it turns recorded runs (production incident transcripts included) into dataset items in one call — resolve through `createPersistenceTraceResolver`, redact, map through an optional host `toItem`, and append as a **new immutable dataset version**. The prior version object is never mutated.
+
+```ts
+import { datasetFromRuns, defineDataset } from "@arnilo/prism-core/governance/evals";
+
+const result = await datasetFromRuns({
+  runIds: ["run_9f2", "run_a71"],
+  dataset: defineDataset({ id: "support-regressions", items: [] }),
+  store: persistence,
+  ownership: { tenantId: "t1", userId: "u1" },
+  redactor,
+  toItem: (run) => ({
+    input: run.input,
+    expected: run.feedback?.metadata?.expected, // human-graded gold from the feedback seam
+    metadata: { graded: run.feedback?.rating },
+  }),
+});
+// result.added === 2, result.dataset.version === "2"
+```
+
+Omit `toItem` and the default mapping is used: `input` = first user message, `expected` = the recorded feedback's `metadata.expected` when present, `output` = final assistant text under `metadata.output`.
+
+- Session ids (`sessionIds`) expand to every run recorded under the session; bare `runIds` are located by one ownership-bounded scan of the run ledger (page/byte caps apply).
+- Each resolved run maps to one item keyed by the run id. The default `toItem` carries the first user message as `input`; `expected` comes **only from the feedback seam** (`metadata.expected` of the latest human-graded `RunFeedbackRecord`) — never re-derived from untrusted outputs, and omitted entirely (not fabricated) when a run has no feedback. The recorded output rides `metadata.output` for provenance. A host `toItem` always wins and may return `undefined` to drop a run (`host filter` skip).
+- Feedback costs one bounded, owner-scoped query per curation batch (`store.feedback`); records are read id-only per the feedback linkage contract — scorer payloads are never copied. A feedback query failure (e.g. scope contract mismatch) omits `expected` instead of aborting the batch.
+- Every item field passes the host `redactor` after host mapping — fail closed: a redactor failure or an item over the frozen 4 MiB cap (`ERR_PRISM_EVAL_CURATE`) skips the run or aborts the append before anything is persisted. Cross-tenant runs are never readable (resolver ownership check → `ownership mismatch` skip).
+
+Prompt versions can ride the same primitives: [`assertPromptPromotion`](prompt-registry.md#eval-gated-promotion) in `@arnilo/prism-core/governance/prompts` resolves two prompt versions, runs them through `runComparison`, and returns a `promote`/`hold` verdict with per-scorer aggregates and a bounded report — never applying the change itself.
+
 ## Coding and browser adversarial evaluations (0.0.9)
 
 Release 0.0.9 ships curated network-free adversarial fixtures in package tests:
 
-- `@arnilo/prism-coding-agent` `eval-fixtures.test.ts`: safe native list vs shell, Git path/ref injection, dirty-tree rollback, unknown named-check failure, PR-handoff artifact completeness, and prompt-injection file content under read-only tools.
-- `@arnilo/prism-browser` `eval-fixtures.test.ts`: stale snapshot refs, side-effect approval, private/loopback/file deny, upload/download/screenshot policy, CSS/evaluate target rejection, and hostile accessible-name text.
+- `@arnilo/prism-coding-tools/agent` `eval-fixtures.test.ts`: safe native list vs shell, Git path/ref injection, dirty-tree rollback, unknown named-check failure, PR-handoff artifact completeness, and prompt-injection file content under read-only tools.
+- `browser` `eval-fixtures.test.ts`: stale snapshot refs, side-effect approval, private/loopback/file deny, upload/download/screenshot policy, CSS/evaluate target rejection, and hostile accessible-name text.
 
-Fixtures reuse `@arnilo/prism-evals` (`defineDataset` / `defineScorer` / `scoreRun` / `assertEvaluationThreshold` / `serializeEvaluationReport`). Optional SWE-bench-compatible or live-browser harnesses remain host adapters — they are not default dependencies or quality claims. Protected real Docker/Playwright gates stay env-gated (`PRISM_TEST_DOCKER_SANDBOX`, `PRISM_LIVE_PLAYWRIGHT`) and never enter `sdk:ready`.
+Fixtures reuse `@arnilo/prism-core/governance/evals` (`defineDataset` / `defineScorer` / `scoreRun` / `assertEvaluationThreshold` / `serializeEvaluationReport`). Optional SWE-bench-compatible or live-browser harnesses remain host adapters — they are not default dependencies or quality claims. Protected real Docker/Playwright gates stay env-gated (`PRISM_TEST_DOCKER_SANDBOX`, `PRISM_LIVE_PLAYWRIGHT`) and never enter `sdk:ready`.
 
 ## PostgreSQL enterprise state (0.0.23)
 
@@ -180,7 +214,7 @@ The protected runner (`scripts/phase27-erp-journey.test.mjs`) carries the journe
 ### Hard-gate usage
 
 ```ts
-import { createErpInvariantScorers, erpInvariantDataset, scoreRun } from "@arnilo/prism-evals";
+import { createErpInvariantScorers, erpInvariantDataset, scoreRun } from "@arnilo/prism-core/governance/evals";
 
 const scorers = createErpInvariantScorers();
 const records = await scoreRun({

package/docs/extensions.md

@@ -140,9 +140,9 @@ await kernel.middleware.run("provider_request", { metadata: {} });
 - [Compaction and retry policies](compaction-and-retry.md): compaction strategy/retry policy contributions and `compaction`/`retry` middleware runtime behavior.
 - [LLM compaction package](compaction-llm.md): optional extension helper that registers a provider-backed compaction strategy.
 - [Observational memory compaction package](compaction-observational-memory.md): optional extension helper that registers an inert fast memory compaction strategy.
-- [Caveman behavior integration](caveman.md): optional `@arnilo/prism-caveman` upstream Caveman skills, commands, level injector, and session `caveman-level` persistence.
-- [Ponytail behavior integration](ponytail.md): optional `@arnilo/prism-ponytail` upstream Ponytail skills, commands, mode injector, and session `ponytail-mode` persistence.
-- [Impeccable behavior integration](impeccable.md): optional `@arnilo/prism-impeccable` upstream Impeccable skill and `load_skill` command.
+- [Caveman behavior integration](caveman.md): optional `@arnilo/prism-coding-tools/caveman` upstream Caveman skills, commands, level injector, and session `caveman-level` persistence.
+- [Ponytail behavior integration](ponytail.md): optional `@arnilo/prism-coding-tools/ponytail` upstream Ponytail skills, commands, mode injector, and session `ponytail-mode` persistence.
+- [Impeccable behavior integration](impeccable.md): optional `@arnilo/prism-coding-tools/impeccable` upstream Impeccable skill and `load_skill` command.
 - [Public contracts](public-contracts.md): `Extension`, `ExtensionAPI`, and contribution contract types.
 - [Credentials and redaction](credentials-and-redaction.md): secret-redaction behavior used for extension errors.
 

package/docs/forge-integration.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`createGitHubForge` is an optional host-activated adapter in `@arnilo/prism-coding-agent` for the six **proven** GitHub operations: issue context read, authenticated push, pull-request create/update, review comments, and check/status retrieval, plus bounded handoff reconciliation. It is GitHub-first by freeze decision — there is no multi-forge generic abstraction, and no octokit dependency: HTTP uses Node's global `fetch` with a bounded streaming reader, timeout, and rate-limit backoff; push reuses the existing `BoundGitRunner` with the token injected via `GIT_CONFIG_*` environment variables (`http.extraHeader`) — never argv, never persisted, never in logs/events. Every mutation flows through Phase 8 approval (`ExecutionPolicy`) and Phase 7 `ToolEffectStore` idempotency keys; a retry after a completed call returns the existing record instead of duplicating the PR or comment.
+`createGitHubForge` is an optional host-activated adapter in `@arnilo/prism-coding-tools/agent` for the six **proven** GitHub operations: issue context read, authenticated push, pull-request create/update, review comments, and check/status retrieval, plus bounded handoff reconciliation. It is GitHub-first by freeze decision — there is no multi-forge generic abstraction, and no octokit dependency: HTTP uses Node's global `fetch` with a bounded streaming reader, timeout, and rate-limit backoff; push reuses the existing `BoundGitRunner` with the token injected via `GIT_CONFIG_*` environment variables (`http.extraHeader`) — never argv, never persisted, never in logs/events. Every mutation flows through Phase 8 approval (`ExecutionPolicy`) and Phase 7 `ToolEffectStore` idempotency keys; a retry after a completed call returns the existing record instead of duplicating the PR or comment.
 
 | Export | Purpose |
 | --- | --- |
@@ -18,7 +18,7 @@
 Use when a coding agent needs to open/update PRs, comment on reviews, push a branch with scoped credentials, or verify handoff state against GitHub before deciding the next step. Do not use as a general GitHub SDK, an auto-merge engine (`reconcileHandoff` never merges), or a replacement for host-owned App installation flows — the adapter resolves credentials through the host's `CredentialResolverSource`-compatible resolver and never stores them.
 
 ```ts
-import { createGitHubForge } from "@arnilo/prism-coding-agent";
+import { createGitHubForge } from "@arnilo/prism-coding-tools/agent";
 
 const forge = createGitHubForge({
   credentials: { name: "github", resolver: myCredentialResolver }, // App installation token preferred; PAT allowed
@@ -75,7 +75,7 @@ Every mutation result is recorded in the `effectStore` with a stable key derived
 ## Implementation example
 
 ```ts
-import { createGitHubForge } from "@arnilo/prism-coding-agent";
+import { createGitHubForge } from "@arnilo/prism-coding-tools/agent";
 
 const forge = createGitHubForge({
   credentials: { name: "github-app", resolver },

package/docs/graft.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-graft` is an optional package that wires [nanonets/graft](https://github.com/nanonets/graft) — a repository context-graph CLI (`graft/` directory, INDEX.md orientation, symbol-level wiring graph) — into Prism contribution contracts.
+`@arnilo/prism-memory/graft` is an optional subpath that wires [nanonets/graft](https://github.com/nanonets/graft) — a repository context-graph CLI (`graft/` directory, INDEX.md orientation, symbol-level wiring graph) — into Prism contribution contracts.
 
 It registers six pull tools backed by the graft CLI (`--json`, argv-safe), a push-mode retrieval-pack context provider plus first-turn orientation injector carried on the `graft` skill, commands (`graft`, `graft-build`, `graft-check`, `graft-viz`), and an edit-watch middleware that computes blast radius after mutating tool calls. Import is inert; a missing graft CLI fails closed at `setup` with a bounded redacted error.
 
@@ -14,7 +14,7 @@ Use it when a host wants agents to locate code by architecture, callers, and cou
 - `"push"` — per-turn retrieval pack (pointers only) + first-turn orientation, injected automatically.
 - `"both"` — everything.
 
-Install optional peer `@nanonets/graft@^0.13.0` **or** pass `packageRoot`/`cliPath` explicitly. Pair with progressive disclosure: the `graft` skill body stays small; tool schemas carry the details. Graft complements indexed code search (`repository_search`): graph/semantic locators vs literal search — neither replaces the other.
+Install optional peer `@nanonets/graft@^0.16.0` **or** pass `packageRoot`/`cliPath` explicitly. Pair with progressive disclosure: the `graft` skill body stays small; tool schemas carry the details. Graft complements indexed code search (`repository_search`): graph/semantic locators vs literal search — neither replaces the other.
 
 Zero-code alternative (L0): hosts can skip this package entirely and let agents call `graft <command> --json` through their shell tool, optionally seeding context with graft's own generated instruction files. This package exists for native-tool ergonomics, budgeted subprocesses, session persistence, and push mode.
 
@@ -74,7 +74,7 @@ Tool call (pull):
 Status event:
 
 ```json
-{ "type": "graft:status", "extension": "@arnilo/prism-graft", "metadata": { "fresh": true, "missing": 0, "stale": 2 } }
+{ "type": "graft:status", "extension": "@arnilo/prism-memory/graft", "metadata": { "fresh": true, "missing": 0, "stale": 2 } }
 ```
 
 ## Implementation example
@@ -83,7 +83,7 @@ See [`examples/graft-extension.ts`](../examples/graft-extension.ts) — network-
 
 ```ts
 import { createExtensionKernel, createMemorySessionStore } from "@arnilo/prism";
-import { createGraftExtension } from "@arnilo/prism-graft";
+import { createGraftExtension } from "@arnilo/prism-memory/graft";
 
 const store = createMemorySessionStore();
 const kernel = createExtensionKernel({ errorPolicy: "throw" });
@@ -105,7 +105,7 @@ await kernel.load([
 - Import alone registers nothing (`sideEffects: false`); no timers, watchers, or network. The only child processes are budgeted graft CLI calls.
 - Retrieval happens in-process via Prism primitives (context provider, injector, tool_result middleware) — no external hook shims.
 - Ask result shape is parsed tolerantly (`nodes|results|matches|hits`) because graft is pre-1.0; formatters emit pointers (`title` + `file:line` + `[[wikilink]]`), never source bodies.
-- Not included in `@arnilo/prism-code`, `@arnilo/prism-sdk`, or the `prism-all` umbrella (deliberate opt-out, like Caveman/Ponytail) — opt-in install only.
+- Not included in `@arnilo/prism-coding-tools`, `@arnilo/prism-core`, or the `prism-all` umbrella (deliberate opt-out, like Caveman/Ponytail) — opt-in install only.
 - Multi-repo layouts work as upstream graft defines them (workspaces, submodules with `--follow-submodules`, sibling repos); point `projectDir` at the graft root that owns the target repo.
 
 ## Security and performance notes

package/docs/guardrails.md

@@ -76,9 +76,9 @@ Guardrails are callbacks supplied by the host. Prism does not discover, load, re
 
 ## Security and performance notes
 
-Optional `@arnilo/prism-policy` can record guardrail outcomes via `recordGuardrailDecision` (evidence refs only; see [Policy and audit](policy-and-audit.md)).
+Optional `@arnilo/prism-core/governance/policy` can record guardrail outcomes via `recordGuardrailDecision` (evidence refs only; see [Policy and audit](policy-and-audit.md)).
 
-Output buffering prevents blocked provider content from reaching subscribers, session entries, ledgers, parsers, delegation, or tools. Tool-output checks receive raw results but Prism discards blocked raw output before event, ledger, transcript, or MCP exposure. Redaction replaces exact known values only; it is not general secret detection. Parallel checks receive an abort signal, but callback code must honor it to stop in-flight work. Browser snapshots and page text from `@arnilo/prism-browser` are untrusted external content: never allow them to modify tools, permissions, credentials, or policy. Browser mutations still require host `ExecutionPolicy`/approval; prompt-injection text in a page cannot grant upload/download release.
+Output buffering prevents blocked provider content from reaching subscribers, session entries, ledgers, parsers, delegation, or tools. Tool-output checks receive raw results but Prism discards blocked raw output before event, ledger, transcript, or MCP exposure. Redaction replaces exact known values only; it is not general secret detection. Parallel checks receive an abort signal, but callback code must honor it to stop in-flight work. Browser snapshots and page text from the `browser` subpath are untrusted external content: never allow them to modify tools, permissions, credentials, or policy. Browser mutations still require host `ExecutionPolicy`/approval; prompt-injection text in a page cannot grant upload/download release.
 
 ## Related APIs