@arnilo/prism 0.5.4 → 0.5.6

package/docs/runs-and-usage.md

@@ -59,9 +59,9 @@ await session.run("Summarize", {
 });
 ```
 
-Defaults are the unconfigured fence (OWASP LLM10): turns 16, provider attempts 24, tool rounds 8, tool calls 32, wall time 120 seconds, request and response bytes 8 MiB each, input tokens 40,000, output tokens 10,000, total tokens 50,000. Hard process ceilings exist only for request/response bytes (64 MiB each), so a bug cannot OOM the host through a giant provider frame; those two axes reject `null`. Every other axis is host policy (0.5.4): omit a key for the default, set a positive safe integer sized to the workload, or set `null` to disable the axis — overnight sessions raise turns/wall/tokens, and a disabled wall still honors `RunOptions.signal`. Resolution stays narrowing-only: `RunOptions.limits` may lower `AgentConfig.limits`, `null` acts as +Infinity (agent 16 + run `null` → 16), and a raised/disabled `maxTurns` lifts an omitted `maxProviderAttempts` (default 24) to at least `maxTurns` so attempts cannot undercut turns; explicitly set attempts values are lifted only when both are finite. Cumulative token counters are billed usage across the whole run, not the context window (`contextBudget` governs window compaction). For production, prefer an explicit `maxCost`: cost needs a finite non-negative amount plus one currency, and when cost is limited, absent, non-finite, or mixed-currency provider cost fails closed. Vendors that omit usage charge zero to the token counters (local/Ollama report none), so a configured `maxCost` is the fail-closed envelope for usage-less vendors.
+Defaults are the unconfigured fence (OWASP LLM10): turns 16, provider attempts 24, tool rounds 8, tool calls 32, wall time 120 seconds, request and response bytes 8 MiB each, input tokens 40,000, output tokens 10,000, total tokens 50,000. Hard process ceilings exist only for request/response bytes (64 MiB each), so a bug cannot OOM the host through a giant provider frame; those two axes reject `null` and are charged **per frame** (request payload, provider event), not as a run-lifetime sum — a 2 MiB prompt sent forty times is 2 MiB frames, not an 80 MiB parse. Snapshots still report the cumulative `requestBytes`/`responseBytes` counters for telemetry. Every other axis is host policy (0.5.4): omit a key for the default, set a positive safe integer sized to the workload, or set `null` to disable the axis — overnight sessions raise turns/wall/tokens, and a disabled wall still honors `RunOptions.signal`. Resolution stays narrowing-only: `RunOptions.limits` may lower `AgentConfig.limits`, `null` acts as +Infinity (agent 16 + run `null` → 16), and a raised/disabled `maxTurns` lifts an omitted `maxProviderAttempts` (default 24) to at least `maxTurns` so attempts cannot undercut turns; explicitly set attempts values are lifted only when both are finite. Cumulative token counters are billed usage across the whole run, not the context window (`contextBudget` governs window compaction). For production, prefer an explicit `maxCost`: cost needs a finite non-negative amount plus one currency, and when cost is limited, absent, non-finite, or mixed-currency provider cost fails closed. Vendors that omit usage charge zero to the token counters (local/Ollama report none), so a configured `maxCost` is the fail-closed envelope for usage-less vendors.
 
-Prism charges turns before assembly, provider attempts and request bytes before generation, response bytes per provider event, tool rounds before a batch, tool calls before dispatch, and usage before another turn. A breach stops new work, aborts active work through the run signal, emits exactly one redacted `run_limit_exceeded` event/ledger row, and throws `AgentRunError` with `result.limit` (`limit`, `maximum`, `observed`, optional `currency`). Provider-reported token/cost totals arrive after generation, so that completed provider turn can be the unavoidable overshoot boundary.
+Prism charges turns before assembly, provider attempts before generation, request bytes per request payload, response bytes per provider event (each frame must fit the byte cap on its own), tool rounds before a batch, tool calls before dispatch, and usage before another turn. A breach stops new work, aborts active work through the run signal, emits exactly one redacted `run_limit_exceeded` event/ledger row, and throws `AgentRunError` with `result.limit` (`limit`, `maximum`, `observed`, optional `currency`). Provider-reported token/cost totals arrive after generation, so that completed provider turn can be the unavoidable overshoot boundary.
 
 `createRunLimitTracker()` and `resolveRunLimits()` are public for adapters that need the same validation and accounting semantics. Workflow agent nodes forward `RunWorkflowOptions.limits`; supervisor delegation narrows its step/tool/token/timeout budget into core limits; MCP tool calls use a per-call tracker.
 

package/docs/session-stores-and-branching.md

@@ -1,124 +1,3 @@
 # Session stores and branching
 
-> Compatibility page: the canonical session-store overview now lives at [Session stores](session-stores.md). This page retains the detailed branch-helper reference and is kept to avoid breaking existing links.
-
-## What it does
-
-Session store helpers define branch-aware session entries and pure utilities for creating entries, listing branch leaves, reading a leaf path, and rebuilding provider context from a selected leaf. For atomic append options, `SessionAppendConflictError`, branch handles, and production `readBranchPath` guidance, start with [Session stores](session-stores.md#atomic-append-and-branch-handles).
-
-Public helpers:
-
-- `createSessionEntry(options)`
-- `createMemorySessionStore(initialEntries?)`
-- `getSessionBranchEntries(entries, options)` and `getSessionBranchEntries(reader, query)`
-- `listSessionBranches(entries)`
-- `rebuildSessionContext(entries, options)` and `rebuildSessionContext(reader, query)`
-
-## When to use it
-
-Use these helpers when a host, runtime session, or store adapter needs durable, branch-aware session data without coupling stores to providers, tools, credentials, or files.
-
-Do not use them as a database layer, migration system, lock service, compaction strategy, retry policy, CLI/RPC protocol, or hidden global store registry.
-
-## Inputs / request
-
-`SessionEntry` has stable branch fields plus typed payloads:
-
-| Field | Purpose |
-| --- | --- |
-| `id` | Unique entry id. |
-| `parentId` | Previous entry on the branch, if any. |
-| `sessionId` | Session that owns the entry. |
-| `timestamp` | ISO timestamp chosen by the caller/helper. |
-| `kind` | `message`, `event`, `summary`, `metadata`, `model_change`, `label`, `custom`, or `compaction`. |
-| `runId` | Optional run id. |
-| `message`, `event`, `model`, `previousModel`, `label`, `summary`, `data`, `metadata` | Optional payload fields for the entry kind. |
-
-`rebuildSessionContext()` and `getSessionBranchEntries()` accept an optional `leafId`. If omitted, the last entry is used as the leaf. The async reader overloads accept `SessionBranchRead { sessionId, leafId?, cursor?, limit? }` and call a `BranchReader` / `readBranchPath` implementation so database adapters can return one ancestor chain without `list(sessionId)`. `createMemorySessionStore()` accepts optional initial entries.
-
-## Outputs / response / events
-
-| Helper | Output |
-| --- | --- |
-| `createSessionEntry()` | A `SessionEntry` with generated `id` and `timestamp` when omitted. |
-| `getSessionBranchEntries()` | Ordered entries from root to selected leaf (deep copies). |
-| `listSessionBranches()` | Leaf ids and their root-to-leaf entry paths (deep copies). |
-| `rebuildSessionContext()` | `{ leafId, entries, messages, summaries }` for provider input rebuild; with a compaction entry, raw `entries` stay intact while `messages` becomes recent context and `summaries` includes the compaction summary. All arrays and objects are deep copies. |
-| `createMemorySessionStore()` | Async `SessionStore` with `append()`, `list(sessionId)`, and `get(id)`. `list()` and `get()` return deep copies. |
-
-Helpers throw on duplicate entry ids, unknown leaves, or missing parents. They do not mutate input arrays.
-
-For `kind: "compaction"`, `data` may contain `throughEntryId`, `keepEntryIds`, `strategy`, and `trigger`. The latest valid compaction entry on a branch is used as the provider-context boundary; raw history remains in `entries`.
-
-## Request/response example
-
-```json
-{
-  "leafId": "entry_2",
-  "messages": [{ "role": "user", "content": [{ "type": "text", "text": "Hi" }] }]
-}
-```
-
-## Implementation example
-
-```ts
-import { createMemorySessionStore, createSessionEntry, rebuildSessionContext } from "@arnilo/prism";
-
-const first = createSessionEntry({
-  id: "entry_1",
-  sessionId: "s1",
-  kind: "message",
-  message: { role: "user", content: [{ type: "text", text: "Hi" }] },
-});
-const label = createSessionEntry({
-  id: "entry_2",
-  parentId: first.id,
-  sessionId: "s1",
-  kind: "label",
-  label: "investigation",
-});
-
-const store = createMemorySessionStore([first]);
-await store.append(label);
-
-const context = rebuildSessionContext(await store.list("s1"), { leafId: label.id });
-```
-
-## Extension and configuration notes
-
-Stores and extensions can use these data helpers directly. Store adapters only need append/list/get behavior; branch queries are derived in memory from listed entries unless the store implements `readBranchPath`. See [Session stores](session-stores.md) for `SessionAppendOptions`, `SessionAppendConflictError`, and `(sessionId, leafId)` branch-handle guidance.
-
-`createMemorySessionStore()` is the built-in in-memory implementation. It preserves append order per session, isolates session ids, returns entries by id in O(1), rejects duplicate entry ids, and returns deep copies from `list()` and `get()`. It is process memory only; hosts that need durability should pass another `SessionStore`.
-
-`getSessionBranchEntries()` and `rebuildSessionContext()` also return deep copies of entries and messages, so callers cannot mutate the input arrays or the memory store by editing returned objects.
-
-`AgentSession` uses `AgentSessionConfig.store` before `AgentConfig.store`, otherwise a private memory store. It appends user, assistant, tool-result, and model-change entries, resumes from `leafId`, rebuilds provider history from the selected branch, checks out old leaves, forks by selecting a leaf in the same session, and clones the selected branch to a new session id.
-
-Node hosts that need simple file durability can import `createJsonlSessionStore()` from the explicit `@arnilo/prism/node/session-store-jsonl` subpath.
-
-Use `createDefaultCompactionStrategy()` to create compaction entries that `rebuildSessionContext()` understands. Compaction adds summaries; it does not delete or rewrite raw store entries.
-
-`createSessionEntry()` accepts injectable `createId` and `now` functions for deterministic tests or host id policy. Prism does not create a global store or id service.
-
-## Security and performance notes
-
-- Helpers are pure data functions: no provider calls, tool calls, settings reads, credential resolution, filesystem access, network access, timers, or dependencies.
-- Store only host-approved session entries. Do not put provider credentials, credential resolvers, provider objects, full provider requests, or secrets in entries.
-- Branch rebuild is linear over listed entries for development stores. Production stores should implement `readBranchPath` and return one branch ancestor chain so large sessions are not fully loaded.
-- Compaction-aware rebuild keeps raw branch entries in `entries`; only provider-context `messages`/`summaries` are reduced.
-- Memory store lookup by id is O(1); list is O(n) for that session.
-- Duplicate ids and missing parents fail clearly instead of guessing a branch.
-
-## Related APIs
-
-- [Migration guide](migration.md): moving branch-handle reads from the dev `list(sessionId)` path to a database-backed `readBranchPath`.
-- [Session stores](session-stores.md): canonical overview for `SessionAppendOptions`, `SessionAppendConflictError`, branch handles, and dev-vs-production branch reads.
-- [Public contracts](public-contracts.md): `SessionEntry`, `SessionStore`, `StoreFactory`, and session contracts.
-- [Agent/session runtime](agent-session-runtime.md): runtime sessions use these branch helpers for store-backed history, checkout, fork, and clone.
-- [Node JSONL session store](node-jsonl-session-store.md): optional Node filesystem store for caller-named JSONL files.
-- [Compaction and retry policies](compaction-and-retry.md): default strategy for creating compaction entries.
-- [Input and prompt assembly](input-and-prompt-assembly.md): provider input assembly consumes rebuilt `messages` and `summaries`.
-- [Credentials and redaction](credentials-and-redaction.md): security boundary for secrets that must not enter session entries.
-- [Workflows](workflows.md): optional orchestration that reuses session `leafId` on resume rather than reloading full transcripts into the scheduler.
-
-Session stores persist the entries they receive. Configure `AgentConfig.redactor` or `RunOptions.redactor` before a run when known secrets must be removed before entries reach durable stores.
+> Compatibility stub: the canonical session-store overview — `SessionStore`, atomic append, `SessionAppendConflictError`, branch handles, and `readBranchPath` — lives at [Session stores](session-stores.md).

package/docs/sqlite-persistence.md

@@ -126,7 +126,7 @@ SQLite applies migrations **006**/**007** for per-run event sequence compatibili
 - [Database persistence](database-persistence.md): shared schema model, conditional append pattern, indexes.
 - [Session store conformance](session-store-conformance.md): `assertSessionStoreConforms` / `runSessionStoreConformance`.
 - [Run ledger conformance](run-ledger-conformance.md): `assertRunLedgerConforms` / `runRunLedgerConformance`.
-- [Persistence, credentials, and multimodality primitives](persistence-credentials-multimodality-primitives.md): package matrix and threat model.
+- [Persistence, credentials, and multimodality primitives](history/persistence-credentials-multimodality-primitives.md): package matrix and threat model.
 - [Node JSONL session store](node-jsonl-session-store.md): dev-only single-process alternative.
 - [Workflows](workflows.md): adapt `persistence.checkpoints` and pass `persistence.leases` to `createWorkflowCoordinator()` and `createWorkflowSchedules()` for durable background execution and schedules.
 - [Migration guide](migration.md): moving from JSONL/in-memory to database-backed persistence.

package/docs/wiki.md

@@ -17,7 +17,7 @@ It integrates Tobias Lütke's [`qmd`](https://github.com/tobi/qmd) on-device hyb
 
 The Karpathy LLM Wiki pattern is structured into 3 distinct tiers:
 
-1. **Raw Sources (Immutable)**: Source code files, design docs, transcripts, journals, and Markdown notes. Raw sources are strictly read-only and never mutated.
+1. **Raw Sources (Immutable)**: Source code files, design docs, transcripts, journals, and Markdown notes. Raw sources are strictly read-only and never mutated. New external material arrives through the ingest staging area (`raw/ingest/<utc>-<slug>/`): an immutable `source.*` original plus a UTF-8 `extract.md` the maintainer skill files into the wiki.
 2. **Compiled Wiki (`.wiki/`)**: Persistent, cross-linked Markdown documents containing synthesized architecture models, entity descriptions, decision records, and line-anchored claims.
 3. **Schema & Protocols (`SCHEMA.md`)**: Operational guidelines governing OKF v0.2 emission, entity categorization, citation rules (`file:///path#Lxx-Lyy`), catalog indexing (`index.md`), and chronological change logging (`log.md`).
 
@@ -39,12 +39,14 @@ The Karpathy LLM Wiki pattern is structured into 3 distinct tiers:
 - `wiki_search`: `{ query: string, mode?: "search" | "vsearch" | "query", maxResults?: number }`
 - `wiki_read_page`: `{ pagePath: string }` — `pagePath` must resolve inside the wiki root (lexical + `fs.realpath` containment). Traversal (sibling-prefix, `..`, absolute paths) and symlinks pointing outside the wiki throw an access-denied error; a missing contained page returns `found: false`.
 - `wiki_record_insight`: `{ title: string, content: string, category?: "decision" | "concept" | "entity" }` — title and content must be non-empty; titles are capped at 200 characters, content at 65,536 bytes, and control characters/newlines in titles are collapsed to spaces so titles cannot inject Markdown headings, index entries, or log entries.
+- `wiki_ingest`: `{ text?: string, path?: string, url?: string, title?: string }` — stages one raw source and returns a filing brief for the `wiki-maintainer` skill (see `/wiki-ingest` below). Exactly one of `text`/`path`/`url` must be provided; `url` requires a host `fetchUrl` hook.
 
 ### Slash Commands
 
 - `/wiki-init`: Scaffolds `.wiki/`, instantiates `SCHEMA.md`, `index.md`, and `log.md`, deploys skills, and adds the `qmd` collection.
 - `/wiki-refresh`: Detects modified source files via SHA-256 Merkle diffing, compiles updates to affected entity pages, reconciles contradictions in `log.md`, and runs `qmd update`.
 - `/wiki-lint`: Checks OKF frontmatter (`type`, ISO `generated.at`), leftover `[[wikilinks]]`, unresolved relative markdown links, dead line anchors, and orphan pages.
+- `/wiki-ingest`: `{ text?, path?, url?, title? }` — stages one external source into `raw/ingest/<utc>-<slug>/` (`source.*` original + `extract.md`), then returns a brief (staged paths, extract preview, source URL when applicable, Karpathy filing checklist). When the host injects `drivers`, the command calls `drivers.startRun(brief, { activeSkills: ["wiki-maintainer"] })` so the maintainer skill files the source into the wiki; without drivers it stages only and reports `runStarted: false`. Results are labeled `metadata.trust: "untrusted_external"`.
 
 ### Standalone CLI Commands
 
@@ -60,12 +62,49 @@ npx prism-wiki lint
 
 # Search wiki from terminal
 npx prism-wiki search "How does authentication work?" --mode query
+
+# Stage an external source for the wiki
+npx prism-wiki ingest --path notes/paper.pdf --title "Paper"
+
+# `--url` is a usage error in the standalone CLI:
+# the wiki package never fetches — URL ingest needs a host fetchUrl hook
+npx prism-wiki ingest --url https://example.com/rfc.pdf   # → exit 1
 ```
 
 ## Outputs / response / events
 
 - `wiki_search` returns a structured markdown payload containing section breadcrumbs, conceptual summaries, and clickable source line links (`file:///path#Lxx-Lyy`).
 - Lifecycle commands return status objects (`{ status: "initialized" | "refreshed" | "clean", ok: boolean }`).
+- `wiki_ingest` / `/wiki-ingest` return the staged paths, the (capped) extract preview, and a filing brief; `value.runStarted` reports whether a driver run was started.
+
+### `ingestWikiSource(input, options)`
+
+The staging primitive behind `/wiki-ingest`, `wiki_ingest`, and the CLI. Accepts `{ text?, path?, bytes?, url?, filename?, title? }` (precedence `path` > `bytes` > `url` > `text`) and returns `{ id, rawDir, sourcePath, extractPath, mediaType?, extract, truncated, url? }` (`url` present only for URL-staged sources).
+
+| Input | Parse behavior |
+| :--- | :--- |
+| Text-like files and `text` | Decoded as UTF-8 (RAG text/markdown/html parsers) |
+| Uncompressed PDF | Parsed by the RAG PDF parser (bounded pages/bytes) |
+| Compressed PDF / DOCX | Throws a named error unless the host supplies `options.extractDocument` (e.g. wire `createDocumentReader()` from `@arnilo/prism-coding-tools/document-reader`) |
+| `url` | `assertSsrfAllowedUrl` runs first (private/link-local hosts rejected before any fetch); then the host `fetchUrl` hook supplies the bytes/text — missing or empty hook output fails closed. Staged filename comes from the hook, the URL extension (`doc.pdf`), or `source.md` |
+| Images | Staged as-is; stub extract points at the staged `source.*` — no OCR; view the file |
+| Unknown binary | Fails closed unless `extractDocument` claims it |
+
+Caps: 32 MiB per staged input, 2 MiB per extract. `path` must resolve inside the workspace root (realpath containment). `log.md` gains an `**Ingested**` entry only when the wiki root exists.
+
+Wiring a `fetchUrl` hook (the wiki package ships no HTTP client — hosts bring their own, e.g. Obscura):
+
+```ts
+import { runObscuraCli, validateObscuraWebUrl } from "@arnilo/prism-web-tools/obscura";
+
+const wiki = createWikiExtension({
+  fetchUrl: async ({ url }) => {
+    validateObscuraWebUrl(url); // host-side SSRF gate of its own
+    const run = await runObscuraCli({ command: "obscura", args: ["fetch", url, "--dump", "markdown"] });
+    return { text: run.stdout, filename: "source.md" };
+  },
+});
+```
 
 ## Request/response example
 
@@ -135,12 +174,17 @@ Emitted `.wiki/` trees are [OKF v0.2](https://github.com/GoogleCloudPlatform/ope
 
 ## Extension and configuration notes
 
-- The wiki subpath registers tools (`wiki_search`, `wiki_read_page`, `wiki_record_insight`), commands (`wiki-init`, `wiki-refresh`, `wiki-lint`), skills (`wiki-maintainer`, `wiki-searcher`), and instruction injectors (`wiki-guidance`) into Prism registries.
+- The wiki subpath registers tools (`wiki_search`, `wiki_read_page`, `wiki_record_insight`, `wiki_ingest`), commands (`wiki-init`, `wiki-refresh`, `wiki-lint`, `wiki-ingest`), skills (`wiki-maintainer`, `wiki-searcher`), and instruction injectors (`wiki-guidance`) into Prism registries.
 - It operates with zero core modifications and can be used with any `@arnilo/prism` agent.
 - `qmd` is optional but recommended. When `@tobilu/qmd` is not installed, the search engine falls back to catalog matching against `index.md`.
 
+## Ingest filing rules (Karpathy/OKF)
+
+After staging, the `wiki-maintainer` skill files the source: read `.wiki/SCHEMA.md` and `index.md` first, read the staged `extract.md` (and view `source.*` for images/PDFs), integrate claims into existing entity/concept/decision pages — create pages only for genuinely new concepts — then emit OKF v0.2 frontmatter (`type` required; `sources[].resource` pointing at the staged `source.*` — for URL-staged sources the original URL is also legitimate; `generated.by: prism-wiki/ingest`), add per-claim footnotes keyed to `sources[].id` for source-specific claims, update `index.md`, and prepend an `**Ingested**` entry to `log.md`. Never copy raw bodies into wiki pages; one source per ingest; contradictions update the existing page and are logged. The same rules ship in every scaffolded `SCHEMA.md` (`## Ingest Protocol`).
+
 ## Security and performance notes
 
+- **Ingest boundaries**: the wiki package never fetches — `url` inputs require a host `fetchUrl` hook, and `assertSsrfAllowedUrl` rejects private/link-local hosts before the hook runs; the standalone CLI rejects `--url` with a usage error. `path` inputs are contained inside the workspace root via realpath; extracts are untrusted data (`trust: "untrusted_external"`), never instructions; the raw layer stays read-only for the LLM; images get a stub extract (no OCR).
 - **Source Immutability**: Raw source files are read-only and never modified by wiki operations.
 - **Subprocess Safety**: All `qmd` subprocess calls use argument arrays (`execFile`) to prevent shell injection.
 - **Path Containment**: Wiki and raw source paths are confined to the workspace root; directory traversal (`../`) is rejected.

package/docs/workflows.md

@@ -432,7 +432,7 @@ Use workflows for known, durable, replayable graphs. Use optional supervisor del
 ## Related APIs
 
 - Examples: `examples/workflow-research-and-review.ts`, `examples/workflow-parallel-research.ts`, `examples/workflow-tool-approval.ts`, `examples/workflow-multimodal-document.ts`, `examples/workflow-sqlite-resume.ts`, `examples/workflow-postgres-resume.ts`, `examples/workflow-event-sink.ts`, `examples/workflow-rpc-cancel.ts`, `examples/workflow-distributed-coordinator.ts`, `examples/autonomous-coding-loop.ts` (host-loop iterate-until-done) — offline runnable demos; PostgreSQL safely skips unless `PRISM_TEST_POSTGRES_URL` is set.
-- [Workflow orchestration primitives](workflow-orchestration-primitives.md): Task 0–1 inventory and locked adapter contracts
+- [Workflow orchestration primitives](history/workflow-orchestration-primitives.md): Task 0–1 inventory and locked adapter contracts
 - [Agent/session runtime](agent-session-runtime.md): `AgentSession.run()`/`stream()`, abort, subscribe
 - [Guardrails](guardrails.md): `RunWorkflowOptions.guardrails` routes tool nodes through core dispatch before policy and side effects.
 - [Supervisor delegation](supervisors.md): bounded dynamic child selection.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arnilo/prism",
-  "version": "0.5.4",
+  "version": "0.5.6",
   "description": "Agent harness for AI providers, agents, sessions, and tools.",
   "type": "module",
   "main": "./dist/index.js",