@arnilo/prism 0.3.1 → 0.4.0

package/docs/workflows.md

@@ -2,14 +2,14 @@
 
 ## What it does
 
-`@arnilo/prism-workflows` is an optional package for typed, bounded DAG orchestration over Prism sessions, tools, events, and persistence seams. Hosts define acyclic workflows with agent/function/tool/conditional/fan-out/join/nested-workflow nodes; the package runs a Kahn-style scheduler with a bounded worker pool, emits package-local `WorkflowEvent`s, checkpoints progress, can coordinate queued runs across multiple host processes using durable leases and fencing, and can run bounded linear sagas with durable compensation.
+`@arnilo/prism-workflows` is an optional package for typed, bounded DAG orchestration over Prism sessions, tools, events, and persistence seams. Hosts define acyclic workflows with agent/function/tool/conditional/fan-out/join/nested-workflow/loop nodes; the package runs a Kahn-style scheduler with a bounded worker pool, emits package-local `WorkflowEvent`s, checkpoints progress, can coordinate queued runs across multiple host processes using durable leases and fencing, and can run bounded linear sagas with durable compensation.
 
 Primary exports:
 
 | Export | Purpose |
 | --- | --- |
 | `defineWorkflow` / `buildGraph` | Validate definitions (acyclicity, edge refs, limits) and build deterministic successor/indegree maps |
-| `agentNode`, `functionNode`, `toolNode`, `conditionalNode`, `fanOutNode`, `joinNode`, `workflowNode` | Typed node factories, including composition through the same runner |
+| `agentNode`, `functionNode`, `loopNode`, `toolNode`, `conditionalNode`, `fanOutNode`, `joinNode`, `workflowNode` | Typed node factories, including bounded iterative refinement and composition through the same runner |
 | `runWorkflow` / `resumeWorkflow` / `suspend` / `replayWorkflow` | Execute, durably suspend, exactly-once resume, or create an immutable-lineage replay from a succeeded node |
 | `createMemoryWorkflowCheckpoints` | In-process `WorkflowCheckpointAdapter` over core `createMemoryCheckpointStore()` |
 | `createWorkflowCheckpoints` | Adapt core `CheckpointStore` (including SQLite/PostgreSQL persistence capabilities) to workflow checkpoint shapes |
@@ -50,8 +50,22 @@ Use `defineSaga`/`runSaga` for a linear business sequence whose remote effects n
 | `limits.maxStateBytes` / hard cap | 64 KiB / 512 KiB |
 | `limits.maxStateHistory` / hard cap | 32 / 128 state snapshots; updates stop before evidence would be discarded |
 | `limits.maxReplayDepth` / hard cap | 8 / 32 lineage generations |
+| loop `maxIterations` | Required per loop / hard cap 64 |
 | `state.initial` / `state.schema` | Initial shared JSON object and optional host-validated schema |
 
+### Node kinds
+
+| Kind | Factory | Behavior |
+| --- | --- | --- |
+| `agent` | `agentNode` | Runs `AgentSession` from `agentFactory` |
+| `function` | `functionNode` | Runs one host async function |
+| `loop` | `loopNode` | Runs one bounded inline or function/tool body repeatedly until `until(ctx)` is true |
+| `tool` | `toolNode` | Dispatches one registered tool, optionally behind durable approval |
+| `conditional` | `conditionalNode` | Evaluates a predicate and skips configured successors |
+| `fan_out` | `fanOutNode` | Maps a bounded list with workflow concurrency |
+| `join` | `joinNode` | Reduces an upstream array |
+| `workflow` | `workflowNode` | Runs a nested workflow with inherited capabilities |
+
 All workflow limits and runtime `concurrency` reject non-safe integers, zero, negatives, NaN, `Infinity`, and values above the named hard cap. Node retries allow 0–100; an explicit node timeout allows 1–86,400,000 ms. Omitting `timeoutMs` remains an explicit host choice.
 
 `runWorkflow(workflow, input, options?)`:
@@ -76,6 +90,16 @@ All workflow limits and runtime `concurrency` reject non-safe integers, zero, ne
 
 A function node returns `suspend({ reason, data?, resumeSchema? })` to persist `status: "suspended"`. Its next invocation receives `ctx.resume` only after an approved resume. `resumeWorkflow(workflow, { runId }, options)` validates schema/version/ownership/`definitionHash`, claims the checkpoint before node execution, and continues the suspended node. Denial persists terminal `denied` status without invoking it. Existing failed/aborted checkpoint resume remains available without a human decision.
 
+> **Contract — resume-aware nodes.** After an approved resume, the **same** node's `execute` is re-invoked with `ctx.resume`. Returning `suspend(...)` unconditionally re-suspends silently; downstream nodes never run. Branch on `ctx.resume`:
+>
+> ```ts
+> execute: async (ctx) => ctx.resume
+>   ? handle(ctx.resume)
+>   : suspendAskUserDecision({ ... }),
+> ```
+>
+> Live demo: [`examples/autonomous-coding-loop.ts`](../examples/autonomous-coding-loop.ts) (`gate` node).
+
 Coding-agent ask-user glue (opt-in, no Goal DB): `suspendAskUserDecision(request)` wraps `suspend` with durable question/options/`selectionMode`/`allowCustom` data + resume schema; resume with `createAskUserDecisionResumeValidator()` or `validateAskUserDecisionResume`. Goal→verify: `runCodingGoalVerify` / `createCodingGoalVerifyWorkflow` compose plan Markdown → named checks → approve suspend → bounded handoff over the same primitives (`examples/coding-goal-verify.ts`). When a workflow node wraps a durable agent run, that run's shared pending-decision batch (Task 2) is the approval authority — workflow `suspend`/`resume` stay workflow-scoped and do not mint a parallel decision store.
 
 Every node receives bounded `ctx.state`, `ctx.stateVersion`, and async `ctx.updateState(patch, { mode: "merge" | "replace" })`. Updates serialize, validate, redact, and snapshot before checkpoint save. A rejected state or checkpoint write stays rejected (nothing committed) and recovers the per-run chain so a later valid write can run. `workflowNode({ workflow })` runs its child with the same ownership, agent/tool registries, execution policy, redactor, signal, checkpoints, and event bus; child state replaces parent state after success.
@@ -120,7 +144,7 @@ Saga `onEvent` callbacks receive metadata-only `saga_transition` events with ten
 
 Schedule `onEvent` receives bounded-attribution `schedule_fired` or metadata-only `schedule_failed`; schedule input is never copied into these events.
 
-Package-local `WorkflowEvent` types: `workflow_started`, `workflow_suspended`, `workflow_resumed`, `workflow_finished`, `node_started`, `node_finished`, `node_failed`, `node_skipped`, `checkpoint_saved`, `agent_event` (wraps a redacted `AgentEvent`), `workflow_event_overflow`. Sequences are monotonic; drain/order is deterministic by `(sequence, nodeId)`.
+Package-local `WorkflowEvent` types: `workflow_started`, `workflow_suspended`, `workflow_resumed`, `workflow_finished`, `node_started`, `node_finished`, `node_iteration_started`, `node_iteration_finished`, `node_failed`, `node_skipped`, `checkpoint_saved`, `agent_event` (wraps a redacted `AgentEvent`), `workflow_event_overflow`. Loop iteration-finished events carry bounded/redacted output and stable `iterationId`. Sequences are monotonic; drain/order is deterministic by `(sequence, nodeId)`.
 
 ## Request/response example
 
@@ -154,6 +178,7 @@ import {
   runWorkflow,
   resumeWorkflow,
   functionNode,
+  loopNode,
   agentNode,
   createWorkflowCheckpoints,
   createWorkflowCommands,
@@ -300,11 +325,79 @@ runRpcServer({
 });
 ```
 
+## Iterative refinement (`loopNode`)
+
+`loopNode` keeps the workflow graph acyclic while executing one body repeatedly. `ctx.iteration` is zero-based, `ctx.iterationId` is a stable compensation key (tenant-prefixed when ownership is supplied), and the body receives the prior body output as `ctx.previousOutput`. `until(ctx)` receives the current body output through that same property. Body and predicate can use `ctx.updateState()` for durable accumulation.
+
+Use inline `execute` for pure refinement, or `body` for one interior function/tool sub-step. A tool body uses the normal durable approval gate; approval suspends before its side effect and an approved resume re-enters the same iteration. Completed prior iterations remain in the checkpoint ledger and are not re-executed.
+
+`maxIterations` is required and capped at 64 (`HARD_MAX_LOOP_ITERATIONS`). The scheduler enforces the cap even when `until` never passes. Exhaustion throws `WorkflowLoopLimitError` with code `ERR_PRISM_WORKFLOW_LOOP_LIMIT`, `iterations`, and a bounded/redacted `lastOutput`. Each completed iteration stores a versioned, bounded/redacted output record before the next body starts; `maxNodeOutputBytes` applies to every body output.
+
+### Frozen budget accounting
+
+`maxNodes` counts declared DAG nodes once. `maxIterations` independently caps loop body executions; iterations never consume `maxNodes`. Both limits are validated before execution and fail closed at their hard caps.
+
+`node_iteration_started` and `node_iteration_finished` events expose `iteration` and `iterationId`; finished events also expose `done` and bounded/redacted output. A replay started from a completed loop re-runs its body and emits the same iteration sequence for the new run. Hosts that persist events should treat `iterationId` as the idempotency key.
+
+```ts
+const refine = loopNode({
+  execute: async (ctx) => ({
+    iteration: ctx.iteration,
+    draft: improve((ctx.previousOutput as { draft?: string } | undefined)?.draft),
+  }),
+  until: (ctx) => (ctx.previousOutput as { passed?: boolean } | undefined)?.passed === true,
+  maxIterations: 5,
+});
+
+const approvedRefine = loopNode({
+  body: toolNode({
+    tool: publishDraft,
+    args: () => ({ action: "refine" }),
+    approval: { reason: "approve refinement side effect" },
+  }),
+  until: (ctx) => ctx.previousOutput === "accepted",
+  maxIterations: 3,
+});
+
+const workflow = defineWorkflow({
+  id: "refine-draft",
+  revision: "1",
+  nodes: { refine, approvedRefine },
+});
+```
+
+### Saga compensation boundary
+
+A loop remains one DAG node and one host saga step. Persist the loop's `iterations` as that step's aggregate output, and register external compensation under each record's `iterationId`; compensate records in reverse iteration order. The workflow runner does not invoke saga handlers implicitly, so the host retains ownership of side-effect policy and audit records while replay/resume stay deterministic.
+
+## Bounded iterate-until-done (host-loop pattern)
+
+Workflows can now use `loopNode` for bounded in-graph refinement. A host `for`/`while` over `runWorkflow` remains useful when each iteration must be a separate run id, use a different workflow definition, or run on versions before this node kind. Runnable proof: [`examples/autonomous-coding-loop.ts`](../examples/autonomous-coding-loop.ts) (N runs, mid-loop human gate with simulated restart, typed budget exhaustion).
+
+1. Keep the DAG acyclic (roadmap → execute → validate → gate → compact).
+2. Pass `{ goal, iteration }` as `runWorkflow` input — never a back-edge.
+3. Bound the host loop (`MAX_ITERATIONS`). Per-child tool/token caps stay on `supervisor.delegate` / `RunOptions`.
+4. Explicit predicate (`passed(outputs)`). Exhaustion throws a typed error — fail-closed, never hang.
+5. Human gate is ordinary `suspend` / `resumeWorkflow` (CAS `expectedVersion`). Restart = new runtime, same checkpoint store.
+6. Audit **each iteration** with `replayWorkflow({ sourceRunId, fromNodeId })`. The host loop is N run ids — `listWorkflowRuns` lists them; `replayWorkflow` does not replay the `for`.
+
+```ts
+for (let i = 0; i < MAX_ITERATIONS; i++) {
+  const run = await runWorkflow(phase, { goal, iteration: i }, { checkpoints, ownership });
+  if (run.status === "suspended") break; // resumeWorkflow later with expectedVersion
+  if (run.status !== "succeeded") throw new Error(run.status);
+  if (passed(run.outputs)) break;
+}
+if (!passed(last.outputs)) throw new BudgetExhaustedError(MAX_ITERATIONS);
+```
+
+For a single bounded refinement, prefer `loopNode`. Keep this host-loop pattern when separate run ids, per-run checkpoints, or a new workflow definition are part of the contract.
+
 ## Extension and configuration notes
 
 - Workflow semantics stay in this optional package; generic checkpoint persistence and bounded event fan-in live in core.
 - `ProductionPersistenceStore.checkpoints` and `.leases` are optional generic capabilities. First-party SQLite/PostgreSQL adapters own `prism_checkpoints` / `prism_leases`; workflows only adapt them. Sagas use the same `WorkflowCheckpointAdapter` and `LeaseStore`; they add no SQL table or scheduler.
-- `createWorkflowEventBus()` delegates queueing, source fan-in, overflow, abort, and close behavior to core `createEventMultiplexer()`, including its single-consumer contract: a second concurrent `subscribe()` is rejected with `EventMultiplexerError` (`ERR_PRISM_EVENT_MULTIPLEXER_SINGLE_CONSUMER`) instead of silently splitting the stream. Graceful `close()` stops new emits/sources and drains already-queued events (in `(sequence, nodeId)` order) before the subscriber completes; overflow `close` still emits one `workflow_event_overflow` notice and terminates.
+- `createWorkflowEventBus()` delegates queueing, source fan-in, overflow, abort, and close behavior to core `createEventMultiplexer()`, including its single-consumer contract: a second concurrent `subscribe()` is rejected with `EventMultiplexerError` (`ERR_PRISM_EVENT_MULTIPLEXER_SINGLE_CONSUMER`) instead of silently splitting the stream. Graceful `close()` stops new emits/sources and drains already-queued events (in `(sequence, nodeId)` order) before the subscriber completes; overflow `close` still emits one `workflow_event_overflow` notice and terminates. Loop iteration events carry bounded/redacted output and stable `iterationId` values for durable sinks.
 - The in-process active-run registry (`registerActiveWorkflowRun` / `getActiveWorkflowRun` / `abortActiveWorkflowRun`) is **non-durable, in-process only — it does not survive restart**; durable active-run recovery is a later milestone. It is bounded: every register sweeps aborted/leaked entries (runs whose promise never settled) and the registry fails closed at `MAX_ACTIVE_WORKFLOW_RUNS` (512) rather than evicting a live run; `sweepActiveWorkflowRuns()` is available for hosts. Cross-tenant lookups stay ownership-isolated.
 - `createWorkflowCommands()` is optional; hosts can drive `workflow.start` / `enqueue` / `replay` / `status` / `list` / `cancel` / `resume`. The six `schedule.*` commands appear only when a scoped `schedules` service is supplied.
 - Hosts may bridge `WorkflowEvent` into OpenTelemetry or custom sinks; there is no built-in TUI.
@@ -313,7 +406,8 @@ runRpcServer({
 
 ## Security and performance notes
 
-- Definitions require a non-empty host-authored `revision` and fail closed on cycles, unknown edges, self-edges, invalid limits, and `maxNodes` overflow. Revision and every nested revision enter the deterministic definition hash; hosts must bump revision when function/tool behavior changes.
+- Definitions require a non-empty host-authored `revision` and fail closed on cycles, unknown edges, self-edges, invalid limits, and `maxNodes` overflow. Revision and every nested revision enter the deterministic definition hash; hosts must bump revision when function/tool behavior changes. Loop `maxIterations` is required and capped at 64.
+- Loop bodies run serially inside one scheduler node; every body output and durable iteration record is bounded/redacted with `maxNodeOutputBytes`, and the scheduler persists the completed-iteration cursor before advancing. Approved durable resumes re-enter only the incomplete iteration.
 - Fan-out length is bounded by `maxFanOut`. Independent `map` items run in a local worker pool capped by the resolved workflow `maxConcurrency` (and `options.concurrency`); output stays in input order. Abort or the first map failure stops further items. There is no extra global admission service.
 - Node outputs, shared state/history, schedule input/records, and checkpoints are byte/count/depth bounded. Checkpoint size remains the final aggregate ceiling.
 - Event buses use a bounded buffer (default 2048) with `close` / `drop_oldest` / `drop_newest` overflow.
@@ -325,19 +419,19 @@ runRpcServer({
 - Active registry identity includes workflow ID, run ID, and exact ownership. Exact duplicates fail instead of overwriting; distinct owners remain isolated in lookup/list/cancel/unregister.
 - Tool nodes attach `workflowId` / `nodeId` on `ExecutionAction.metadata` for approval/audit context.
 - Nested workflows inherit host registries/policies and cannot inject broader tools, agents, ownership, or credentials. Nested depth is inherited; child suspension bubbles to the parent review cursor.
-- Replay source ownership/hash/status/node eligibility are checked before a new checkpoint is created. Source records are immutable, lineage is bounded, and copied approval-bearing paths are rejected.
+- Replay source ownership/hash/status/node eligibility are checked before a new checkpoint is created. Source records are immutable, lineage is bounded, and copied approval-bearing paths are rejected. Replaying from a completed loop starts a fresh loop cursor and emits its per-iteration records; it never mutates source evidence.
 - Schedule services are ownership-scoped and explicitly started. Per-fire leases plus deterministic run IDs/CAS prevent duplicate enqueue across coordinators and crash retry. Host calculator IDs resolve only from the supplied map; no callback or cron expression is persisted.
 - Proactive schedules require an explicit capability grant. Revocation pauses the schedule (never fired by `pollOnce`) and `assertActive` fails closed on missing/revoked/expired tokens; enable/revoke/deny events carry redacted actor refs for the host policy ledger. Capability TTL is capped (default 24h / hard 31d) and the token record is byte-bounded (≤ 16 KiB); tokens are ownership-scoped, so foreign access fails closed rather than leaking existence.
 - Scheduler stores O(nodes + active outputs + bounded state history); ready-node work uses indegree maps, not repeated full scans.
 - Lease acquisition is atomic; opaque tokens protect renew/release; monotonically increasing fencing tokens plus checkpoint compare-and-swap prevent expired workers from committing after takeover. Node functions must honor `ctx.signal` for prompt cooperative cancellation.
 - Saga runs require `tenantId`; checkpoint keys and leases include tenant ownership. Every transition uses checkpoint CAS plus the current lease fence. Forward/compensation retries are capped at 3 by default / 10 hard; ambiguous outcomes require reconciliation and unresolved state becomes `manual_intervention`.
-- Saga input, step outputs, and error text are byte-bounded and passed through the configured `SecretRedactor` before persistence or compensation. Manual resolution requires an active verified actor for the tenant, exact checkpoint version, bounded reason, and a non-empty host audit reference; Prism does not pretend to verify the external audit record.
+- Saga input, step outputs, and error text are byte-bounded and passed through the configured `SecretRedactor` before persistence or compensation. A loop used as one saga step remains one aggregate compensation record; hosts register and compensate its durable iteration IDs in reverse order. Manual resolution requires an active verified actor for the tenant, exact checkpoint version, bounded reason, and a non-empty host audit reference; Prism does not pretend to verify the external audit record.
 
 Use workflows for known, durable, replayable graphs. Use optional supervisor delegation only when child selection must be dynamic at runtime; do not replace deterministic nodes with model routing without a concrete need.
 
 ## Related APIs
 
-- Examples: `examples/workflow-research-and-review.ts`, `examples/workflow-parallel-research.ts`, `examples/workflow-tool-approval.ts`, `examples/workflow-multimodal-document.ts`, `examples/workflow-sqlite-resume.ts`, `examples/workflow-postgres-resume.ts`, `examples/workflow-event-sink.ts`, `examples/workflow-rpc-cancel.ts`, `examples/workflow-distributed-coordinator.ts` — offline runnable demos; PostgreSQL safely skips unless `PRISM_TEST_POSTGRES_URL` is set.
+- Examples: `examples/workflow-research-and-review.ts`, `examples/workflow-parallel-research.ts`, `examples/workflow-tool-approval.ts`, `examples/workflow-multimodal-document.ts`, `examples/workflow-sqlite-resume.ts`, `examples/workflow-postgres-resume.ts`, `examples/workflow-event-sink.ts`, `examples/workflow-rpc-cancel.ts`, `examples/workflow-distributed-coordinator.ts`, `examples/autonomous-coding-loop.ts` (host-loop iterate-until-done) — offline runnable demos; PostgreSQL safely skips unless `PRISM_TEST_POSTGRES_URL` is set.
 - [Workflow orchestration primitives](workflow-orchestration-primitives.md): Task 0–1 inventory and locked adapter contracts
 - [Agent/session runtime](agent-session-runtime.md): `AgentSession.run()`/`stream()`, abort, subscribe
 - [Guardrails](guardrails.md): `RunWorkflowOptions.guardrails` routes tool nodes through core dispatch before policy and side effects.

package/docs/working-and-semantic-memory.md

@@ -2,7 +2,7 @@
 
 ## What it does
 
-`@arnilo/prism-memory` is an optional package for schema/template-backed working memory and embedding-based semantic recall. It owns narrow `Embedder` and `VectorStore` contracts reused by `@arnilo/prism-rag`, plus an in-memory reference path and one PostgreSQL/pgvector production adapter.
+`@arnilo/prism-memory` is an optional package for schema/template-backed working memory and embedding-based semantic recall. It owns narrow `Embedder` and `VectorStore` contracts reused by the `@arnilo/prism-memory/rag` subpath, plus an in-memory reference path and one PostgreSQL/pgvector production adapter.
 
 ## When to use it
 
@@ -26,6 +26,7 @@ Ordinary Prism sessions do not require this package or any vector backend.
 | `limits` | no | top-K, adjacent range, batch, payload, injected-token, export, and rebuild caps |
 | `redactor` / `secrets` | no | Redact text/metadata before persist/inject |
 | `requireConsent` | no | Strict mode: recall/injection excludes entries lacking explicit consent |
+| `importanceFrom` | no | Host-owned hook deriving importance from a redacted reflection payload (write time only; no default, no LLM) |
 
 Semantic indexing (entries carry `MemoryConsent` source/visibility; unset defaults to `{ source: "user", scope: "thread", visible: true }`):
 
@@ -37,13 +38,60 @@ Semantic indexing (entries carry `MemoryConsent` source/visibility; unset defaul
 | `grantedAt` / `revokedAt` | Optional host/audit timestamps; a revocation excludes the record. |
 
 ```ts
-await memory.remember({ entries: [{ id, text, metadata?, consent?, sequence? }] }, { wait?: boolean })
+await memory.remember({ entries: [{ id, text, metadata?, consent?, sequence?, importance?, reflection? }] }, { wait?: boolean })
 ```
 
 Semantic recall (honors consent/visibility at assembly time):
 
 ```ts
-await memory.recall(query, { topK?, messageRange?, requireConsent?, signal? })
+await memory.recall(query, { topK?, messageRange?, requireConsent?, scoring?, signal? })
+```
+
+#### Composite recall scoring (opt-in)
+
+Default recall is pure similarity + lexical scoring and stays unchanged. Hosts opt into blending recency and importance at recall time via `scoring`:
+
+| `RecallScoringOptions` field | Meaning |
+| --- | --- |
+| `recencyWeight` | Weight in `[0,1]` for timestamp half-life decay; requires `halfLifeMs` |
+| `importanceWeight` | Weight in `[0,1]` for the stored record `importance` (neutral `1.0` when absent) |
+| `halfLifeMs` | Positive finite recency half-life in milliseconds |
+
+The resolver validates weights (finite, in `[0,1]`, no extra dependencies) and sum-normalizes: similarity keeps the remainder of `1`; weights overshooting `1` normalize down (similarity → `0`). Hit order becomes the blended score with the same deterministic tie-break (`score` desc, `sequence` asc, `id` asc), and hits expose the `similarity`, `recency`, `importance`, and `score` components. Both adapters converge on one shared pure re-rank — candidates are fetched at `topK × 4`, blended, then cut to `topK` — so pgvector ordering matches the in-memory adapter by construction.
+
+```ts
+const recalled = await memory.recall("preferred response format", {
+  topK: 8,
+  scoring: { recencyWeight: 0.3, importanceWeight: 0.2, halfLifeMs: 7 * 24 * 3600 * 1000 },
+});
+// hits[0]: { text, score, similarity, recency, importance, ... }
+```
+
+Security/performance: importance is host-trusted data clamped to `[0,1]` at write and scoring time; scoring is per-hit arithmetic with no extra queries or LLM calls; recall without `scoring` returns today's ordering and hit shape unchanged.
+
+#### Importance at write (derivation from existing signals)
+
+Stored `importance` never comes from an LLM analysis pass over writes — it derives from existing signals only:
+
+- Direct: pass `importance` on a `remember()` entry (clamped to `[0,1]` at write; wins over derivation).
+- Derived: set `importanceFrom` on `createMemory()` and pass a `reflection` object on the entry. The hook runs once at write time over the reflection **after secret redaction**, and its output is clamped to `[0,1]`; a non-finite output fails the write. Entries without `importance`/`reflection` (or without a hook) score at the neutral `1.0`. The hook is never invoked at recall, and the reflection payload itself is not persisted.
+
+For observational-memory reflections (`@arnilo/prism-memory/compaction/observational-memory`, `MemoryReflection`), spread the record into the entry. The recipe below is an example heuristic — hosts own the real heuristic, and none ships as a default:
+
+```ts
+const memory = createMemory({
+  // ...scope + embedder + stores
+  importanceFrom: (reflection) => {
+    // frequency/prominence recipe example: normalized mention count, no LLM call
+    const mentions = Number(reflection.mentions ?? reflection.supportingObservationIds?.length ?? 1);
+    return Number.isFinite(mentions) ? mentions / 10 : 1;
+  },
+});
+
+await memory.remember(
+  { entries: [{ id: reflection.id, text: reflection.content, reflection: { ...reflection } }] },
+  { wait: true },
+);
 ```
 
 Consent + lifecycle (real grant/correct/delete/retention on stored entries):
@@ -171,14 +219,14 @@ const store = await createPostgresVectorStore({
 // getCurrentGeneration/setCurrentGeneration. close() ends adapter-owned pools.
 ```
 
-`createPostgresVectorStore()` is the production counterpart to `createMemoryVectorStore()` used by `@arnilo/prism-rag`; `createPostgresMemoryStores()` reuses the same vector implementation internally.
+`createPostgresVectorStore()` is the production counterpart to `createMemoryVectorStore()` used by the `rag` subpath; `createPostgresMemoryStores()` reuses the same vector implementation internally.
 
 ## Extension and configuration notes
 
 - Hosts wire the context provider into `AgentConfig.context` or `resolveContextProviders()`.
 - The working-memory processor is opt-in and host-invoked; middleware is not required.
 - `createHashEmbedder()` is for tests/demos only; production hosts supply a real `Embedder`.
-- Observational memory (`@arnilo/prism-compaction-observational-memory`) remains unchanged and composable.
+- Observational memory (`/compaction/observational-memory`) remains unchanged and composable.
 - Consent is enforced at the single `recall()` gate, so both direct recall and `createContextProvider()` injection honor it; `visible: false` (or a revoked grant) keeps an entry out of prompts, events, exports, and telemetry. `setConsent`/`correct` re-upsert in place (consent change does not re-embed); `forget`/`applyRetention` are real deletes, not tombstones. Retention uses indexed oldest-first pages plus a scoped count, deleting one default-500/hard-5000 batch without reading a corpus into memory. The PostgreSQL adapter persists consent in a `consent JSONB` column added by `buildMemoryDdl`.
 - The PostgreSQL vector path owns its DDL in Prism (`buildMemoryDdl`/`buildVectorSearchDdl` exported): the `<table>_rag_scope_generations` per-scope generation pointer table, `text_tsv` tsvector column + GIN index for the lexical RAG leg, and an HNSW index when the embedding dimension is pinned. DDL runs against the host's **knowledge database** — the host names `schema`/`table` (defaults `prism_memory`/`semantic_memory`), owns backup/retention of that database, and can run migrations manually with `skipMigrations: true`. Identifiers are validated/quoted; values stay parameterized.
 - `createPostgresVectorStore({ dimension })` pins the embedding column width before building indexes: pgvector can only build HNSW over `vector(N)` columns, and dimension mismatch fails closed instead of drifting.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@arnilo/prism",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "description": "Agent harness for AI providers, agents, sessions, and tools.",
   "type": "module",
   "main": "./dist/index.js",
@@ -122,34 +122,14 @@
     "CHANGELOG.md"
   ],
   "workspaces": [
-    "packages/provider-*",
-    "packages/memory",
-    "packages/rag",
-    "packages/compaction-*",
-    "packages/observability-*",
-    "packages/tool-validator-*",
-    "packages/session-store-*",
-    "packages/credentials-node",
-    "packages/mcp",
-    "packages/evals",
-    "packages/workflows",
-    "packages/coding-agent",
-    "packages/coding-security",
-    "packages/server",
-    "packages/supervisor",
-    "packages/web-tools",
-    "packages/work-tools",
-    "packages/policy",
-    "packages/model-router",
-    "packages/enterprise-postgres",
-    "packages/browser",
-    "packages/obscura",
-    "packages/ag-ui",
     "packages/acp-agent",
-    "packages/computer-use-linux",
-    "packages/document-reader",
+    "packages/ag-ui",
     "packages/antigravity-agent",
-    "packages/prism-*"
+    "packages/mcp",
+    "packages/memory",
+    "packages/office",
+    "packages/prism-*",
+    "packages/web-tools"
   ],
   "scripts": {
     "build:core": "node scripts/with-build-lock.mjs tsc",
@@ -157,21 +137,21 @@
     "build": "npm run build:core && npm run build --workspaces --if-present",
     "typecheck": "npm run build && npm run typecheck --workspaces --if-present && tsc -p examples --noEmit",
     "sweep:unused": "node scripts/sweep-unused.mjs --json",
-    "test": "npm run build && node scripts/with-build-lock.mjs node --test dist/__tests__/*.test.js && node scripts/with-build-lock.mjs node --test scripts/release-gate.test.mjs scripts/tooling-gate.test.mjs scripts/budget-gate.test.mjs scripts/phase8-conformance.test.mjs scripts/phase9-conformance.test.mjs scripts/phase10-conformance.test.mjs scripts/phase11-conformance.test.mjs scripts/phase11-freeze.test.mjs scripts/phase12-freeze.test.mjs scripts/phase13-freeze.test.mjs scripts/phase14-freeze.test.mjs scripts/phase15-freeze.test.mjs scripts/phase16-freeze.test.mjs scripts/phase17-freeze.test.mjs scripts/phase18-freeze.test.mjs scripts/phase19-freeze.test.mjs scripts/phase20-freeze.test.mjs scripts/phase21-freeze.test.mjs scripts/benchmark-0.1.0.test.mjs scripts/benchmark-multi-agent.test.mjs scripts/sweep-unused.test.mjs scripts/e2e-enterprise-journey.test.mjs scripts/e2e-coding-journey.test.mjs scripts/phase23-quality-gates.test.mjs scripts/phase24-truth.test.mjs scripts/phase25-bounded-accumulation.test.mjs scripts/phase26-freeze.test.mjs scripts/phase27-freeze.test.mjs scripts/phase27-ha.test.mjs scripts/phase27-erp-journey.test.mjs scripts/phase27-release.test.mjs scripts/phase29-freeze.test.mjs scripts/phase30-freeze.test.mjs scripts/phase30-release.test.mjs scripts/phase34-freeze.test.mjs scripts/phase37-provider-matrix.test.mjs scripts/phase26-index-benchmark.test.mjs scripts/obscura-host-conformance.test.mjs && node --test scripts/phase23-build-race.test.mjs && npm run test --workspaces --if-present",
+    "test": "npm run build && node scripts/with-build-lock.mjs node --test dist/__tests__/*.test.js && node scripts/with-build-lock.mjs node --test scripts/release-gate.test.mjs scripts/tooling-gate.test.mjs scripts/budget-gate.test.mjs scripts/phase8-conformance.test.mjs scripts/phase9-conformance.test.mjs scripts/phase10-conformance.test.mjs scripts/phase11-conformance.test.mjs scripts/phase11-freeze.test.mjs scripts/phase12-freeze.test.mjs scripts/phase13-freeze.test.mjs scripts/phase14-freeze.test.mjs scripts/phase15-freeze.test.mjs scripts/phase16-freeze.test.mjs scripts/phase17-freeze.test.mjs scripts/phase18-freeze.test.mjs scripts/phase19-freeze.test.mjs scripts/phase20-freeze.test.mjs scripts/phase21-freeze.test.mjs scripts/benchmark-0.1.0.test.mjs scripts/benchmark-multi-agent.test.mjs scripts/benchmark-tool-search.test.mjs scripts/benchmark-workflow-loop.test.mjs scripts/sweep-unused.test.mjs scripts/e2e-enterprise-journey.test.mjs scripts/e2e-coding-journey.test.mjs scripts/phase23-quality-gates.test.mjs scripts/phase24-truth.test.mjs scripts/phase25-bounded-accumulation.test.mjs scripts/phase26-freeze.test.mjs scripts/phase27-freeze.test.mjs scripts/phase27-ha.test.mjs scripts/phase27-erp-journey.test.mjs scripts/phase27-release.test.mjs scripts/phase29-freeze.test.mjs scripts/phase30-freeze.test.mjs scripts/phase30-release.test.mjs scripts/phase34-freeze.test.mjs scripts/phase37-provider-matrix.test.mjs scripts/phase26-index-benchmark.test.mjs scripts/obscura-host-conformance.test.mjs scripts/phase54-package-map.test.mjs scripts/phase54-legacy-registry.test.mjs && node --test scripts/phase23-build-race.test.mjs && npm run test --workspaces --if-present",
     "test:coverage": "node scripts/with-build-lock.mjs node --test --experimental-test-coverage --test-coverage-lines=60 --test-coverage-functions=70 --test-coverage-branches=75 --test-coverage-exclude='**/__tests__/**' --test-coverage-exclude='**/node_modules/**' --test-coverage-exclude='**/scripts/**' --test-coverage-exclude='**/packages/**' --test-coverage-exclude='**/examples/**' dist/__tests__/*.test.js && node scripts/with-build-lock.mjs node scripts/coverage-summary.mjs && node --test scripts/phase23-coverage.test.mjs && node --test scripts/phase23-skip-manifest.test.mjs",
     "coverage:summary": "node scripts/with-build-lock.mjs node scripts/coverage-summary.mjs",
     "lint": "biome lint . --reporter=sarif --reporter-file=scripts/lint-report.sarif",
     "format": "biome format --write .",
     "format:check": "biome format .",
     "pack:dry-run": "npm pack --dry-run && npm run pack:dry-run --workspaces --if-present",
-    "test:postgres": "node scripts/require-postgres-url.mjs && npm run test:postgres --workspace @arnilo/prism-session-store-postgres && npm run test:postgres --workspace @arnilo/prism-memory && npm run test:postgres --workspace @arnilo/prism-enterprise-postgres && node --test scripts/phase7-conformance.test.mjs scripts/phase12-restart-recovery.test.mjs scripts/phase22-conformance.test.mjs",
+    "test:postgres": "node scripts/require-postgres-url.mjs && npm run test:postgres --workspace @arnilo/prism-core --if-present && npm run test:postgres --workspace @arnilo/prism-memory && node --test scripts/phase7-conformance.test.mjs scripts/phase12-restart-recovery.test.mjs scripts/phase22-conformance.test.mjs",
     "release:dry-run": "npm run sdk:ready",
     "release:check": "node scripts/release.mjs check",
     "release:publish": "node scripts/release.mjs publish",
     "release:evidence": "node scripts/release-skip-manifest.mjs",
     "sdk:ready": "npm run typecheck && npm run lint && npm run format:check && npm test && npm run test:coverage && npm run pack:dry-run && npm run release:gate",
     "release:gate": "node scripts/release-skip-manifest.mjs && node scripts/check-client-neutrality.mjs && node scripts/release.mjs gate",
-    "security:threat-suites": "node --test scripts/phase8-conformance.test.mjs scripts/phase9-conformance.test.mjs scripts/phase10-conformance.test.mjs scripts/phase11-conformance.test.mjs scripts/phase20-security.test.mjs scripts/phase21-security.test.mjs scripts/phase22-security.test.mjs scripts/phase23-security.test.mjs scripts/phase38-codeql-regression.test.mjs"
+    "security:threat-suites": "node --test scripts/phase8-conformance.test.mjs scripts/phase9-conformance.test.mjs scripts/phase10-conformance.test.mjs scripts/phase11-conformance.test.mjs scripts/phase20-security.test.mjs scripts/phase21-security.test.mjs scripts/phase22-security.test.mjs scripts/phase23-security.test.mjs scripts/phase38-codeql-regression.test.mjs scripts/phase40-security.test.mjs scripts/phase46-webhooks-security.test.mjs dist/__tests__/pinned-fetch.test.js packages/prism-core/dist/runtime/server/__tests__/webhooks.test.js"
   },
   "devDependencies": {
     "@biomejs/biome": "^2.5.5",

package/templates/README.md

@@ -0,0 +1,23 @@
+# Prism Template Gallery
+
+Ready-to-run project templates for `prism init --template <name>`.
+
+## Available Templates
+
+| Template | Description | Included Packages |
+| --- | --- | --- |
+| `init` | Minimal starter Prism agent with one selected provider and offline mock test | `@arnilo/prism` |
+| `deep-research` | Flagship deep research agent: plan -> search -> extract -> refine loop -> citations -> HITL clarify | `@arnilo/prism`, `@arnilo/prism-web-tools`, `@arnilo/prism-memory`, `@arnilo/prism-workflows` |
+
+## Usage
+
+```bash
+# Scaffold the flagship deep-research template
+prism init my-research --template deep-research
+
+# List all available templates
+prism init --list-templates
+
+# Scaffold the standard minimal agent
+prism init my-agent
+```

package/templates/deep-research/README.md.tmpl

@@ -0,0 +1,47 @@
+# __PROJECT_NAME__ — Deep Research Agent
+
+A production-grade, modular deep research agent built on **Prism**.
+
+The agent executes an end-to-end research pipeline:
+1. **Plan:** Generates a structured research plan with targeted queries.
+2. **Search & Extract:** Fetches search results and content via `@arnilo/prism-web-tools`.
+3. **Refine (Bounded Loop):** Evaluates coverage and refines queries across bounded iterations.
+4. **Cite & Synthesize:** Synthesizes findings with verifiable, traceable source citations.
+5. **HITL Clarification:** Proposes structured clarification choices when research topics are ambiguous.
+
+## Quick Start
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Run offline tests (100% offline with mock provider)
+npm test
+
+# 3. Start the research agent
+npm start
+
+# 4. Launch local dev inspector
+npm run dev
+```
+
+## Architecture & Component Mapping
+
+| Stage / Seam | Component | Documentation |
+| --- | --- | --- |
+| **Orchestration** | `@arnilo/prism-workflows` DAG with bounded refine loop & checkpoints | [`docs/workflows.md`](https://github.com/arniloy/prism/blob/main/docs/workflows.md) |
+| **Search & Fetch** | `@arnilo/prism-web-tools` Brave/Exa search & Firecrawl fetch | [`docs/web-tools.md`](https://github.com/arniloy/prism/blob/main/docs/web-tools.md) |
+| **Citations & RAG** | Attribution via `@arnilo/prism-web-tools` & `@arnilo/prism-memory` | [`docs/rag.md`](https://github.com/arniloy/prism/blob/main/docs/rag.md) |
+| **HITL Clarification** | Structured decision tool with durable suspend/resume | [`docs/coding-agent-tools.md`](https://github.com/arniloy/prism/blob/main/docs/coding-agent-tools.md) |
+| **Security & Trust** | Untrusted content boundaries for search & web data | [`docs/host-security.md`](https://github.com/arniloy/prism/blob/main/docs/host-security.md) |
+| **Inspector** | Local loopback playground via `prism dev` | [`docs/cli-rpc.md`](https://github.com/arniloy/prism/blob/main/docs/cli-rpc.md) |
+
+## Live Configuration (Opt-in)
+
+To connect real LLM providers and live web search APIs:
+1. Copy `.env.example` to `.env`:
+   ```bash
+   cp .env.example .env
+   ```
+2. Set your provider key (e.g. `OPENAI_API_KEY`) and search key (e.g. `BRAVE_API_KEY`).
+3. Update `src/tools.ts` or `src/agent.ts` to instantiate live providers.

package/templates/deep-research/env.example.tmpl

@@ -0,0 +1,12 @@
+# Deep Research Agent Environment Configuration
+# Copy this file to .env to configure live providers and search tools.
+# By default, tests and starter runs use the offline mock provider (no keys required).
+
+# --- Model Providers (Optional) ---
+# OPENAI_API_KEY=sk-...
+# ANTHROPIC_API_KEY=sk-ant-...
+
+# --- Web Search & Extraction (Optional) ---
+# BRAVE_API_KEY=...
+# FIRECRAWL_API_KEY=...
+# EXA_API_KEY=...

package/templates/deep-research/gitignore.tmpl

@@ -0,0 +1,7 @@
+node_modules/
+dist/
+.env
+.env.local
+.prism/
+coverage/
+*.log

package/templates/deep-research/manifest.json

@@ -0,0 +1,12 @@
+{
+  "name": "deep-research",
+  "description": "Deep research agent: plan -> search -> extract -> refine loop -> citations -> HITL clarify",
+  "version": "0.1.0",
+  "tags": ["research", "workflows", "web-tools", "rag", "hitl"],
+  "packages": [
+    "@arnilo/prism",
+    "@arnilo/prism-web-tools",
+    "@arnilo/prism-memory",
+    "@arnilo/prism-workflows"
+  ]
+}

package/templates/deep-research/package.json.tmpl

@@ -0,0 +1,23 @@
+{
+  "name": "__PROJECT_NAME__",
+  "version": "0.1.0",
+  "private": true,
+  "type": "module",
+  "scripts": {
+    "build": "tsc -p tsconfig.json",
+    "typecheck": "tsc -p tsconfig.json --noEmit",
+    "test": "npm run build && node --test dist/__tests__/research.test.js",
+    "start": "npm run build && node dist/index.js",
+    "dev": "prism dev"
+  },
+  "dependencies": {
+__DEPENDENCIES__
+  },
+  "devDependencies": {
+    "@types/node": "^22.0.0",
+    "typescript": "^5.7.0"
+  },
+  "engines": {
+    "node": ">=20"
+  }
+}

package/templates/deep-research/src/agent.ts.tmpl

@@ -0,0 +1,81 @@
+import {
+  type Agent,
+  createAgent,
+  createMockProvider,
+  providerDone,
+  providerTextDelta,
+} from "@arnilo/prism";
+import { createResearchSearchAdapter, createResearchWebTools } from "./tools.js";
+
+/**
+ * Creates the planning agent responsible for breaking down topics into search queries.
+ */
+export function createPlannerAgent(options?: {
+  readonly model?: { provider: string; model: string };
+  readonly provider?: import("@arnilo/prism").AIProvider;
+}): Agent {
+  const provider =
+    options?.provider ??
+    createMockProvider([
+      providerTextDelta(
+        JSON.stringify({
+          topic: "Prism Architecture",
+          queries: [
+            { query: "Prism workflows orchestration", rationale: "Understand DAG execution", aspect: "workflows" },
+            { query: "Prism web tools citations", rationale: "Check search and citation seam", aspect: "web-tools" },
+          ],
+        }),
+      ),
+      providerDone(),
+    ]);
+
+  return createAgent({
+    model: options?.model ?? { provider: "mock", model: "demo" },
+    provider,
+    instructions:
+      "You are a research planning agent. Given a research topic, produce a structured research plan breaking down the topic into targeted search queries.",
+  });
+}
+
+/**
+ * Creates the synthesis agent responsible for compiling findings into a cited report.
+ */
+export function createSynthesizerAgent(options?: {
+  readonly model?: { provider: string; model: string };
+  readonly provider?: import("@arnilo/prism").AIProvider;
+}): Agent {
+  const provider =
+    options?.provider ??
+    createMockProvider([
+      providerTextDelta(
+        "### Executive Summary\nPrism provides modular agent runtimes with durable workflows and verifiable web-tool citations.\n\n### Findings\n- Workflows execute deterministic DAG pipelines.\n- Web tools generate verifiable cryptographic citations.",
+      ),
+      providerDone(),
+    ]);
+
+  return createAgent({
+    model: options?.model ?? { provider: "mock", model: "demo" },
+    provider,
+    instructions:
+      "You are a research synthesis agent. Review the gathered research findings and compose a detailed, factual summary referencing the attributable citations.",
+  });
+}
+
+/**
+ * Default app agent export for `prism dev` and CLI inspector.
+ */
+export function createAppAgent(): Agent {
+  const adapter = createResearchSearchAdapter();
+  const tools = createResearchWebTools(adapter);
+
+  return createAgent({
+    model: { provider: "mock", model: "demo" },
+    provider: createMockProvider([
+      providerTextDelta("Deep Research Agent ready. Ask me to research any topic."),
+      providerDone(),
+    ]),
+    tools,
+    instructions:
+      "You are a Deep Research Assistant. You break down topics, search the web, refine evidence, and produce cited reports.",
+  });
+}