@arnilo/prism 0.11.0 → 0.12.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +32 -0
- package/README.md +28 -27
- package/dist/cli-dev.js +1 -1
- package/dist/cli-init.js +4 -4
- package/dist/cli-provider-add.js +2 -2
- package/dist/field-evidence.d.ts +53 -0
- package/dist/field-evidence.js +143 -0
- package/dist/index.d.ts +3 -1
- package/dist/index.js +2 -1
- package/dist/run-bundle.js +2 -2
- package/docs/acp-agent.md +2 -2
- package/docs/ag-ui.md +1 -1
- package/docs/attention-compiler.md +1 -1
- package/docs/cli-rpc.md +2 -2
- package/docs/coding-tools.md +3 -3
- package/docs/compaction-observational-memory.md +13 -3
- package/docs/computer-use-linux.md +2 -2
- package/docs/core.md +9 -10
- package/docs/dev-inspector.md +1 -1
- package/docs/disaster-recovery.md +1 -1
- package/docs/document-extraction.md +140 -0
- package/docs/evaluations.md +1 -1
- package/docs/execution-timeline.md +30 -0
- package/docs/guardrails.md +53 -0
- package/docs/history/130-cyclic-workflows-primitive-review.md +293 -0
- package/docs/history/132-document-ingestion-primitive-review.md +124 -0
- package/docs/history/README.md +2 -0
- package/docs/history/migrate-to-0.12.0.md +57 -0
- package/docs/history/retire-node-runtime.md +59 -0
- package/docs/hooks.md +1 -1
- package/docs/host-compositions.md +2 -2
- package/docs/host-security.md +1 -1
- package/docs/index.md +30 -25
- package/docs/live-testing.md +1 -1
- package/docs/mcp-tools.md +2 -2
- package/docs/messaging-channels.md +1 -1
- package/docs/migrate-to-0.7.md +1 -1
- package/docs/migrate-to-0.8.md +2 -2
- package/docs/migrate-to-0.9.md +1 -1
- package/docs/migration.md +2 -2
- package/docs/model-registry.md +1 -1
- package/docs/model-routing.md +21 -0
- package/docs/multi-agent-patterns.md +77 -4
- package/docs/obscura.md +1 -1
- package/docs/openapi-tools.md +1 -1
- package/docs/operations.md +2 -2
- package/docs/peer-dependencies.md +18 -17
- package/docs/performance.md +32 -32
- package/docs/prompt-registry.md +1 -1
- package/docs/provider-packages.md +23 -21
- package/docs/providers/decisions.md +134 -0
- package/docs/providers/laya.md +116 -0
- package/docs/providers/typesafe.md +148 -0
- package/docs/rag.md +17 -5
- package/docs/release-and-install.md +119 -113
- package/docs/run-bundle.md +4 -0
- package/docs/runs-and-usage.md +1 -1
- package/docs/signal-channel.md +1 -1
- package/docs/sqlite-persistence.md +5 -5
- package/docs/synapta-integration-review.md +238 -0
- package/docs/telegram-channel.md +1 -1
- package/docs/testing.md +33 -14
- package/docs/wiki.md +12 -11
- package/docs/work-sandbox.md +2 -0
- package/docs/work-tools.md +3 -3
- package/docs/workflows.md +213 -11
- package/package.json +22 -21
- package/templates/business-worker/README.md.tmpl +3 -3
- package/templates/business-worker/package.json.tmpl +4 -5
- package/templates/business-worker/src/tests/agent.test.ts.tmpl +1 -1
- package/templates/business-worker/tsconfig.json.tmpl +2 -1
- package/templates/deep-research/README.md.tmpl +4 -4
- package/templates/deep-research/package.json.tmpl +4 -5
- package/templates/deep-research/src/tests/research.test.ts.tmpl +1 -1
- package/templates/deep-research/tsconfig.json.tmpl +2 -1
- package/templates/init/README.md.tmpl +3 -3
- package/templates/init/optional/evals-example.ts.tmpl +1 -1
- package/templates/init/optional/workflows-example.ts.tmpl +1 -1
- package/templates/init/package.json.tmpl +4 -5
- package/templates/init/src/tests/agent.test.ts.tmpl +1 -1
- package/templates/init/tsconfig.json.tmpl +2 -1
- package/templates/personal-assistant/README.md.tmpl +3 -3
- package/templates/personal-assistant/package.json.tmpl +4 -5
- package/templates/personal-assistant/src/tests/agent.test.ts.tmpl +1 -1
- package/templates/personal-assistant/tsconfig.json.tmpl +2 -1
- package/templates/provider/README.md.tmpl +2 -2
- package/templates/provider/package.json.tmpl +3 -4
- package/templates/provider/src/tests/provider.test.ts.tmpl +1 -1
- package/templates/provider/tsconfig.json.tmpl +1 -1
|
@@ -0,0 +1,293 @@
|
|
|
1
|
+
# Cyclic Workflows Primitive Review: Loop/Graph Primitives and Superstep Engine Justification
|
|
2
|
+
|
|
3
|
+
Plan: [130-Cyclic-Workflow-Graphs.md](../../plans/130-Cyclic-Workflow-Graphs.md) Task 1
|
|
4
|
+
Date: 2026-09-25
|
|
5
|
+
Baseline: Release `0.11.0` / `0.12.0` line in progress
|
|
6
|
+
Scope: Primitive inventory, capability gap analysis, alternative evaluation, and architectural justification for cyclic workflow execution in `@arnilo/prism-core/runtime/workflows`.
|
|
7
|
+
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
## 1. Executive Summary & Review Scope
|
|
11
|
+
|
|
12
|
+
This review freezes the architectural inventory, capability gap analysis, alternative evaluation, and design justification for **Plan 130: Cyclic Workflow Graphs (Superstep Execution + Dynamic Routing)**.
|
|
13
|
+
|
|
14
|
+
### 1.1 Context and Problem Statement
|
|
15
|
+
|
|
16
|
+
Modern agentic architectures heavily depend on cyclic patterns:
|
|
17
|
+
- **Reflection loops**: A worker agent drafts a deliverable, a reviewer agent critiques or tests it, and back-edges route revisions to the worker until acceptance criteria are satisfied.
|
|
18
|
+
- **Planner ↔ Executor interleavings**: A planner decomposes an evolving problem, an executor carries out sub-tasks and observes environmental feedback, and control cycles back to the planner to adjust future actions.
|
|
19
|
+
- **Model-driven dynamic routing**: A node inspects intermediate state or LLM output and dynamically chooses which branch to activate next, creating conversational multi-turn workflows.
|
|
20
|
+
|
|
21
|
+
In `@arnilo/prism-core/runtime/workflows` today, all workflow definitions are enforced to be Directed Acyclic Graphs (DAGs). Attempting to declare an edge that introduces a cycle triggers an immediate, unconditional validation failure:
|
|
22
|
+
```
|
|
23
|
+
WorkflowDefinitionError: Workflow graph contains a cycle
|
|
24
|
+
```
|
|
25
|
+
(`packages/prism-core/src/runtime/workflows/define.ts:L141`).
|
|
26
|
+
|
|
27
|
+
Hosts seeking cyclic behavior are forced to either:
|
|
28
|
+
1. Collapse multi-agent cycles into a single monolithic `loopNode` body, destroying per-node observability, telemetry, and checkpointing.
|
|
29
|
+
2. Externalize the loop to host application code by repeatedly invoking `runWorkflow`, sacrificing durable single-run lifecycle, lease coordination, and unified state history.
|
|
30
|
+
|
|
31
|
+
### 1.2 Review Objectives
|
|
32
|
+
|
|
33
|
+
1. **Inventory existing primitives**: Rigorously analyze `loopNode`, `conditionalNode`, `fanOutNode`/`joinNode`, `suspend`/`resumeWorkflow`, `replayWorkflow`, and supervisor delegation to document what is already achievable today without new runtime machinery.
|
|
34
|
+
2. **Name the capability gaps precisely**: Articulate why existing primitives cannot cleanly express multi-agent cyclic collaboration.
|
|
35
|
+
3. **Evaluate and reject alternatives**: Critically assess candidate approaches (monolithic `loopNode` bodies, host-side recursion, event-driven reactive topologies, and Kahn scheduler indegree resets) with definitive architectural rationale.
|
|
36
|
+
4. **Justify the chosen design**: Detail why a Bulk Synchronous Parallel (BSP) / Pregel-inspired **superstep activation engine** with explicit bounds (`limits.maxSupersteps`), per-node activation semantics (`"all" | "any"`), and dynamic routing (`routeNode`) provides the minimal, fail-closed, and industry-aligned primitive set.
|
|
37
|
+
5. **Restate fail-closed invariants**: Ensure budget bounding, secret redaction, execution policy enforcement, and bit-for-bit backward compatibility remain strictly preserved.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## 2. Inventory of Existing Primitives (What is Achievable Today)
|
|
42
|
+
|
|
43
|
+
Before introducing any new runtime primitive, Prism requires an exhaustive audit of existing primitives to prove that the proposed capability cannot be cleanly composed from what already ships.
|
|
44
|
+
|
|
45
|
+
| Primitive / Surface | Source Location | Existing Contract & Capabilities | Achievable Workload Patterns | Inherent Boundary / Limitation |
|
|
46
|
+
|---|---|---|---|---|
|
|
47
|
+
| **`loopNode` (inline body)** | `packages/prism-core/src/runtime/workflows/types.ts:L155`, `run/node-execution.ts:L360-L420` | Executes an inline async function `execute(ctx)` repeatedly until `until(ctx)` evaluates to `true` or `maxIterations` (default/hard cap 64) is reached. Checkpoints iteration records (`WorkflowLoopIterationRecord[]`) into the node's state. | - Single-agent iterative refinement (e.g. LLM self-correction prompt loop within one session).<br>- Polling external resources until ready.<br>- Numerical relaxation / convergence loops. | The entire loop is encapsulated inside **one single graph node**. Downstream edges cannot be reached until the loop completely terminates. Cannot host multiple distinct graph nodes per iteration. |
|
|
48
|
+
| **`loopNode` (sub-step body)** | `packages/prism-core/src/runtime/workflows/types.ts:L163`, `run/node-execution.ts:L420-L460` | Executes a single interior `function` or `tool` node definition repeatedly with `until(ctx)`. Supports durable suspension before tool side effects. | - Bounded tool retries with approval gates.<br>- Repeated execution of a single atomic tool or function. | Restricted to exactly **one** child function or tool definition (`WorkflowLoopBodyDefinition`). Cannot express multi-node sub-graphs (e.g. Worker -> Tool -> Reviewer). |
|
|
49
|
+
| **`conditionalNode`** | `packages/prism-core/src/runtime/workflows/types.ts:L175`, `run/skip.ts:L6-L25` | Evaluates predicate `when(ctx)`. Activates static `then?: string[]` successors when true, or `else?: string[]` successors when false. Unchosen branches are transitively marked `skipped`. | - Binary branching in DAGs (e.g., pass vs fail paths).<br>- Bypassing expensive downstream stages.<br>- Static forward routing. | Edges must be strictly forward and acyclic. Routing targets are static configuration lists, not dynamic expressions. Cannot route backwards or repeat nodes. |
|
|
50
|
+
| **`fanOutNode` & `joinNode`** | `packages/prism-core/src/runtime/workflows/types.ts:L184,L191`, `run/node-execution.ts:L330-L360` | `fanOutNode` dynamically maps an array of items across concurrent workers within `concurrency` and `maxFanOut`. `joinNode` reduces upstream outputs into an aggregated value. | - Parallel role specialist execution (Hierarchical Crew pattern, `docs/multi-agent-patterns.md`).<br>- Map-reduce pipelines. | Strictly one-shot forward DAG execution. Re-running the fan-out based on join output requires an external loop. |
|
|
51
|
+
| **`suspend()` & `resumeWorkflow()`** | `packages/prism-core/src/runtime/workflows/run/main.ts`, `run/checkpoint.ts` | Allows any node to return `suspend({ reason, data?, resumeSchema? })`, saving a `status: "suspended"` checkpoint. `resumeWorkflow` claims the checkpoint with CAS `expectedVersion` and re-invokes the node with `ctx.resume`. | - Durable human-in-the-loop approval gates.<br>- Asynchronous webhooks and external confirmations.<br>- Zero-process suspension while awaiting input. | Re-enters the **same** suspended node only. It does not rewind, loop back, or branch to earlier nodes in the graph. |
|
|
52
|
+
| **`replayWorkflow()`** | `packages/prism-core/src/runtime/workflows/replay.ts` | Forks an existing succeeded run from a specific node (`fromNodeId`), copying terminal evidence outside the downstream closure and executing downstream nodes anew under an immutable lineage record. | - Re-evaluating a downstream DAG segment with modified inputs.<br>- Auditing and scenario counterfactual testing. | Spawns a **new independent run** (`runId` changes, separate checkpoint stream). It is a post-hoc replay utility, not an in-flight control-flow loop. |
|
|
53
|
+
| **Supervisor delegation** | `packages/prism-core/src/runtime/supervisor/`, `src/contracts.ts` | A parent agent session delegates tasks to allow-listed child agent sessions via `supervisor.delegate()` or `createSpawnAgentTool`. | - Dynamic sub-agent spawning during a conversational turn.<br>- Hierarchical tool dispatch. | Operates at the agent/session runtime level, not the workflow graph level. Lacks graph-level checkpoints, workflow state transactions, DAG event scheduling, and topological visualization. |
|
|
54
|
+
|
|
55
|
+
### 2.1 Pattern Coverage Matrix
|
|
56
|
+
|
|
57
|
+
Evaluating common multi-agent workflow patterns against the existing primitive set confirms what can and cannot be built today:
|
|
58
|
+
|
|
59
|
+
| Workflow Pattern | Expressible with Existing Primitives? | How it is Composed Today | Limitation / Compromise Incurred |
|
|
60
|
+
|---|:---:|---|---|
|
|
61
|
+
| **Single-Agent Self-Correction** | **Yes** | `loopNode` with inline `execute` or `generateValidateReviseLoop` (`src/agent-loops.ts:L37`). | Confined to one prompt/turn loop within one agent session. |
|
|
62
|
+
| **Static Branching Pipeline** | **Yes** | `conditionalNode` with `then` / `else` edge lists. | Forward only; cannot loop back on quality failure. |
|
|
63
|
+
| **Hierarchical Crew (One-Shot)** | **Yes** | Manager `agentNode` -> `fanOutNode` specialists -> `joinNode` aggregator (`examples/crew-hierarchy.ts`). | Complete DAG execution; revision requires re-instantiating the pipeline. |
|
|
64
|
+
| **Human-in-the-Loop Review Gate** | **Yes** | `functionNode` calling `suspend()`; resumed via `resumeWorkflow()`. | Pauses pipeline cleanly, but cannot send the deliverable back to an earlier worker node upon rejection. |
|
|
65
|
+
| **Multi-Agent Reflection Loop**<br>*(Worker -> Reviewer -> Worker until approved)* | **No** | **Fails validation** (`Workflow graph contains a cycle`). | Forces embedding all roles into a single `loopNode.execute`, losing distinct graph nodes, per-role telemetry, and discrete checkpoints. |
|
|
66
|
+
| **Planner ↔ Executor Cyclic Interleaving** | **No** | **Fails validation** (`Workflow graph contains a cycle`). | Requires host-side while-loop wrapping `runWorkflow`, fragmenting state across multiple disconnected runs. |
|
|
67
|
+
| **Model-Driven Dynamic Successor Selection** | **No** | Static `conditionalNode` branching only. | Cannot dynamically compute a target successor set from LLM output without pre-declaring rigid combinations. |
|
|
68
|
+
|
|
69
|
+
---
|
|
70
|
+
|
|
71
|
+
## 3. The Capability Gap
|
|
72
|
+
|
|
73
|
+
The audit identifies three fundamental capability gaps that prevent `@arnilo/prism-core/runtime/workflows` from serving as a complete substrate for cyclic multi-agent systems:
|
|
74
|
+
|
|
75
|
+
### Gap 1: Per-Node Checkpointing, Events, and Attribution Inside an Iteration
|
|
76
|
+
|
|
77
|
+
In multi-agent systems, collaboration occurs between distinct roles with distinct configurations (different system instructions, different models, different tool allow-lists, and distinct permissions).
|
|
78
|
+
- In a reflection cycle, Worker and Reviewer must be observable as distinct steps on the execution timeline (`projectWorkflowTimeline`).
|
|
79
|
+
- When all logic is crammed into a single `loopNode`, the workflow engine emits only `node_iteration_started` / `node_iteration_finished` for the outer loop node. There are no discrete `node_started`, `node_finished`, or `agent_event` records distinguishing the Worker phase from the Reviewer phase within an iteration.
|
|
80
|
+
- If a crash or timeout occurs during Reviewer execution in iteration 4, checkpoint recovery cannot identify that the Worker succeeded and only the Reviewer needs re-execution; the entire composite iteration is lost or must be re-run from the beginning.
|
|
81
|
+
- Furthermore, per-node resource limits (such as `timeoutMs`, `retries`, and `maxNodeOutputBytes`) apply to the whole loop rather than being enforced independently on Worker and Reviewer.
|
|
82
|
+
|
|
83
|
+
### Gap 2: Mixed Node Kinds per Iteration
|
|
84
|
+
|
|
85
|
+
A realistic iterative cycle rarely consists of a single function. A standard software engineering agent workflow consists of:
|
|
86
|
+
1. `agentNode` ("planner"): analyzes requirements and drafts changes.
|
|
87
|
+
2. `toolNode` ("linter_and_test_runner"): executes local tests or build commands in a sandbox.
|
|
88
|
+
3. `agentNode` ("code_reviewer"): reviews code diffs and test output.
|
|
89
|
+
4. `conditionalNode` or routing decision: decides whether to commit or loop back to "planner".
|
|
90
|
+
|
|
91
|
+
Today's `loopNode` supports only two shapes:
|
|
92
|
+
- `InlineLoopNodeDefinition`: a single raw JavaScript function `execute(ctx)`.
|
|
93
|
+
- `NodeLoopNodeDefinition`: a single child `WorkflowLoopBodyDefinition` (either one function or one tool).
|
|
94
|
+
|
|
95
|
+
It is syntactically and architecturally impossible to declare a heterogeneous graph of agents, tools, and conditionals inside a `loopNode` without manually simulating a mini-orchestrator inside the `execute` callback.
|
|
96
|
+
|
|
97
|
+
### Gap 3: Dynamic Successor Activation
|
|
98
|
+
|
|
99
|
+
Prism's `conditionalNode` requires pre-defining static arrays of successors: `then?: readonly string[]` and `else?: readonly string[]`.
|
|
100
|
+
- In dynamic agentic workflows (e.g. LangGraph's `Command(goto=...)` or AutoGen's dynamic routing), an agent or classifier inspects data and selects one or more destinations from a set of declared valid routes (e.g., `["revise_code"]`, `["request_clarification"]`, `["escalate_to_human"]`, `["deploy"]`).
|
|
101
|
+
- Expressing dynamic multi-way routing using binary `conditionalNode`s requires cascading trees of dummy nodes that clutter the graph, complicate Mermaid/DOT exports, and degrade execution latency.
|
|
102
|
+
- There is currently no primitive that allows a node to evaluate a dynamic selector function (`select: (ctx) => string[]`) and activate only the chosen subset of declared successor edges.
|
|
103
|
+
|
|
104
|
+
---
|
|
105
|
+
|
|
106
|
+
## 4. Evaluation of Rejected Alternatives
|
|
107
|
+
|
|
108
|
+
Four architectural alternatives were considered to address these gaps. Each was rejected based on Prism's design principles:
|
|
109
|
+
|
|
110
|
+
### Alternative A: Monolithic `loopNode.execute` Bodies (Do Nothing / Status Quo)
|
|
111
|
+
|
|
112
|
+
- **Description**: Recommend that hosts implement reflection loops by putting all role coordination inside a single `loopNode` whose `execute` function imperatively invokes multiple `AgentSession`s or functions.
|
|
113
|
+
- **Why Rejected**:
|
|
114
|
+
1. **Destroys Observability**: Telemetry events (`WorkflowEvent`) are emitted only at node boundaries. The inner multi-agent dialogue becomes an opaque black box to event subscribers, AG-UI, and OpenTelemetry spans.
|
|
115
|
+
2. **Coarse-Grained Checkpointing**: Checkpoint writes occur at the node or loop-iteration boundary. Individual agent steps within the iteration are uncheckpointed.
|
|
116
|
+
3. **Loss of Graph Topology**: Graph serialization (`serializeWorkflowGraph`), Mermaid diagrams (`workflowGraphToMermaid`), and DOT views cannot visualize the agent interaction topology because it exists only as procedural TypeScript code inside a closure.
|
|
117
|
+
4. **Violation of Declarative Orchestration**: Forces hosts to reinvent agent execution, error handling, and parameter passing instead of relying on the workflow engine's proven lifecycle.
|
|
118
|
+
|
|
119
|
+
### Alternative B: Host-Side Recursion Over `runWorkflow`
|
|
120
|
+
|
|
121
|
+
- **Description**: Hosts execute a workflow representing one iteration (e.g. Worker -> Reviewer). If the reviewer requests revisions, the host application code calls `runWorkflow` again with the previous run's output passed as input.
|
|
122
|
+
- **Why Rejected**:
|
|
123
|
+
1. **Fragmented Durability**: Each iteration becomes a separate `runId`. There is no unified checkpoint history, no aggregate execution timeline, and no single source of truth for the complete business task.
|
|
124
|
+
2. **Broken Coordination and Fencing**: `createWorkflowCoordinator` leases and executes single workflow runs. If an iterative task is split into multiple independent runs, workers cannot maintain atomic lease ownership or guarantee fair scheduling across the complete task lifecycle.
|
|
125
|
+
3. **Suspension Failure**: If a human-in-the-loop review is requested in iteration 3, `suspend()` pauses only that discrete sub-run. The host application must build custom stateful machinery to remember which iteration it was on and restart the outer loop when the sub-run resumes.
|
|
126
|
+
4. **Lineage Overhead**: Tracking provenance requires external bookkeeping or chaining dozens of `replayWorkflow` lineage records, multiplying database records and memory overhead.
|
|
127
|
+
|
|
128
|
+
### Alternative C: Event-Driven Topology (Reactive / Typed Event Bus as Edges)
|
|
129
|
+
|
|
130
|
+
- **Description**: Replace explicit graph edges with a pub/sub event bus where nodes emit typed domain events and downstream nodes subscribe to event patterns (similar to LlamaIndex Workflows).
|
|
131
|
+
- **Why Rejected / Deferred**:
|
|
132
|
+
1. **Authoring Model Divergence**: Prism's workflow system is built around explicit, auditable graph topologies (`WorkflowDefinition.edges`). An event-driven architecture is a fundamentally different authoring paradigm, not merely a missing primitive.
|
|
133
|
+
2. **Static Auditability**: Security and compliance teams inspect Prism workflow graphs to verify dataflow boundaries and authorization constraints. In a purely event-driven model, finding the complete topology requires runtime tracing or complex static analysis.
|
|
134
|
+
3. **Cycle Bound Complexity**: Detecting cycles and enforcing fail-closed iteration bounds (`maxSupersteps`) becomes significantly harder when connections are dynamic event channels rather than static adjacency lists.
|
|
135
|
+
4. **Composability**: Hosts needing event-driven routing can already compose it cleanly today using `conditionalNode`s inspecting shared JSON state (`ctx.state`).
|
|
136
|
+
|
|
137
|
+
### Alternative D: Extending the Kahn Scheduler with Indegree Resets
|
|
138
|
+
|
|
139
|
+
- **Description**: Maintain the existing Kahn topological sort algorithm in `packages/prism-core/src/runtime/workflows/run/scheduler.ts`, but allow backward edges. When a backward edge fires, decrement or reset the target node's `remainingIndegree` so it re-enters the `state.ready` queue.
|
|
140
|
+
- **Why Rejected**:
|
|
141
|
+
1. **The Kahn Deadlock Trap**: Kahn's algorithm relies on the invariant that indegree reaches zero only when *all* upstream dependencies have completed. On a cyclic graph, a node that has both an initial entry edge (from a start node) and a back-edge (from a reviewer) has an initial indegree of 2. In iteration 1, it can never start because the back-edge has not run yet (deadlock). If the scheduler ignores back-edges on entry, it must know which edges belong to which iteration wave—introducing ad-hoc state tracking that breaks down when multiple concurrent branches exist.
|
|
142
|
+
2. **Indegree Reset Ambiguity**: When multiple predecessors exist in a cycle, resetting indegrees on completion of any single predecessor causes premature activations; waiting for all causes deadlocks. This is the precise deadlock flaw identified in early AutoGen and solved in Microsoft Agent Framework (MAF) by moving to explicit activation semantics.
|
|
143
|
+
3. **Regression Risk to DAG Execution**: Mutating the battle-tested, zero-overhead Kahn drain loop threatens bit-for-bit backward compatibility and performance on existing acyclic workflows.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## 5. Industry Survey & Baseline Comparisons
|
|
148
|
+
|
|
149
|
+
To ensure Prism adopts proven, interoperable, and industry-standard primitives, we benchmarked the three leading agent workflow frameworks:
|
|
150
|
+
|
|
151
|
+
| Framework | Cyclic Execution Mechanism | Loop Bound Guardrail | Node Activation Semantics | Dynamic Routing Primitive |
|
|
152
|
+
|---|---|---|---|---|
|
|
153
|
+
| **LangGraph** (`StateGraph`) | Cyclic edges allowed in graph definition. Nodes write to shared state. Superstep execution rounds synchronize parallel node executions. | `recursion_limit` (default: 25). Throws `GraphRecursionError` on breach. | Superstep wave: all nodes activated in step $N$ execute concurrently; their outputs trigger step $N+1$. | `Command(goto="node_id")` and conditional edges returning dynamic node strings. |
|
|
154
|
+
| **Microsoft Agent Framework (MAF)** | Bulk Synchronous Parallel (BSP) / Pregel-inspired supersteps. Iterations advance through synchronization barriers. Run succeeds on **idle drain**. | `max_iterations` (default: 100). Fails closed on breach. | Activation conditions (`all` vs `any`) configured per edge group. | Dynamic executor routing via message destinations. |
|
|
155
|
+
| **AutoGen** (GraphFlow) | Directed cyclic graphs with message-passing semantics between agents. | `max_turns` / iteration limits configured per conversation pattern. | Per-node `activation_condition`: `"all"` (all predecessors must arrive) vs `"any"` (any predecessor arrival fires node). | Dynamic speaker selection via selector functions / LLM judge. |
|
|
156
|
+
|
|
157
|
+
### 5.1 Key Industry Lessons Adopted by Prism
|
|
158
|
+
|
|
159
|
+
1. **The Superstep / Wave Execution Model is the Standard**: Both LangGraph and Microsoft Agent Framework execute cyclic graphs in discrete superstep waves. All active nodes in wave $K$ execute up to configured concurrency; completed nodes post activations for wave $K+1$; an execution barrier synchronizes the transition.
|
|
160
|
+
2. **Termination via Idle Drain**: In a cyclic graph, there is no single fixed "leaf" node. A run naturally and successfully terminates when all pending activations are exhausted and no nodes are currently running (the MAF model).
|
|
161
|
+
3. **Explicit Activation Semantics (`"all"` vs `"any"`) Eliminate Deadlocks**: To allow a node to serve both as a workflow entry point and as the recipient of a cycle back-edge, nodes must support `activation: "any"`. Nodes requiring multi-branch synchronization (joins within a cycle) use `activation: "all"`.
|
|
162
|
+
4. **Constrained Dynamic Routing (`routeNode`)**: Dynamic routing must be constrained to a statically declared successor set. This provides model-driven flexibility (LangGraph `Command(goto)`) while maintaining topological auditability and preventing unvetted arbitrary jumps.
|
|
163
|
+
|
|
164
|
+
---
|
|
165
|
+
|
|
166
|
+
## 6. Justification of the Superstep Engine & Prism Design Synthesis
|
|
167
|
+
|
|
168
|
+
Based on the primitive review, capability gap analysis, and industry survey, Plan 130 synthesizes these requirements into a cohesive, minimal design:
|
|
169
|
+
|
|
170
|
+
### 6.1 Architecture Overview
|
|
171
|
+
|
|
172
|
+
```
|
|
173
|
+
Workflow Definition
|
|
174
|
+
│
|
|
175
|
+
Is graph cyclic OR opt-in?
|
|
176
|
+
┌────┴────┐
|
|
177
|
+
No Yes
|
|
178
|
+
│ │
|
|
179
|
+
Kahn DAG Scheduler Superstep Engine
|
|
180
|
+
(Bit-for-bit unchanged) (run/superstep.ts)
|
|
181
|
+
│
|
|
182
|
+
┌─────────────────────┴─────────────────────┐
|
|
183
|
+
│ Wave Loop: superstep = 0 .. maxSupersteps │
|
|
184
|
+
│ 1. Check budget: superstep < max │
|
|
185
|
+
│ 2. Drain pendingActivations into wave │
|
|
186
|
+
│ 3. Execute wave within concurrency pool │
|
|
187
|
+
│ 4. Nodes complete -> onNodeSucceeded │
|
|
188
|
+
│ - routeNode / conditional evaluate │
|
|
189
|
+
│ - Post activations (all vs any rule) │
|
|
190
|
+
│ 5. Barrier: wait for wave completion │
|
|
191
|
+
│ 6. Batch persist checkpoint (Schema v2) │
|
|
192
|
+
│ 7. Check Idle Drain -> Success │
|
|
193
|
+
└───────────────────────────────────────────┘
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
### 6.2 Core Design Pillars
|
|
197
|
+
|
|
198
|
+
#### 1. Opt-In Cycle Policy with Fail-Closed Bounding
|
|
199
|
+
- In `defineWorkflow`, graphs containing cycles are rejected with `WorkflowDefinitionError: Workflow graph contains a cycle` unless `limits.maxSupersteps` is explicitly declared.
|
|
200
|
+
- `limits.maxSupersteps` is validated via the existing `validateWorkflowLimit` helper (positive safe integer, 1 to `HARD_MAX_SUPERSTEPS = 256`).
|
|
201
|
+
- Cyclic runs increment `state.superstep` on each wave. If `state.superstep >= limits.maxSupersteps` and pending work remains, execution immediately aborts and fails closed with `WorkflowSuperstepLimitError` (`ERR_PRISM_WORKFLOW_SUPERSTEP_LIMIT`).
|
|
202
|
+
|
|
203
|
+
#### 2. Dual-Engine Dispatch (Zero DAG Regressions)
|
|
204
|
+
- The execution engine is selected deterministically at run start:
|
|
205
|
+
- If the definition is acyclic and does not declare `maxSupersteps`, the engine uses the existing `executeSchedulerBody` Kahn path bit-for-bit. Existing tests, event sequences, and checkpoint formats remain completely undisturbed.
|
|
206
|
+
- If the definition contains cycles or opts into supersteps, the run dispatches to `run/superstep.ts`.
|
|
207
|
+
- Both engines share a unified completion seam: `onNodeSucceeded(state, nodeId, options, emit)`.
|
|
208
|
+
|
|
209
|
+
#### 3. Per-Node Activation Semantics (`"all"` vs `"any"`)
|
|
210
|
+
- Every node definition inherits `activation?: "all" | "any"` on `WorkflowNodeBase` (default `"all"`).
|
|
211
|
+
- **`"all"`**: The node activates only when *every* declared predecessor has fired and contributed an activation since the node's last execution.
|
|
212
|
+
- **`"any"`**: The node activates whenever *any* predecessor fires. This enables cycle re-entry nodes (e.g. `worker` receiving entry from `start` or feedback from `reviewer`) to re-fire smoothly without deadlocking.
|
|
213
|
+
- Fail-closed validation: Declaring `activation: "any"` on a node with zero predecessors is rejected at definition time as invalid configuration.
|
|
214
|
+
|
|
215
|
+
#### 4. Dynamic Routing Primitive (`routeNode`)
|
|
216
|
+
- Exported factory: `routeNode({ select })`.
|
|
217
|
+
- Definition shape: `RouteNodeDefinition` (`kind: "route"`, `select: (ctx) => readonly string[] | Promise<readonly string[]>`).
|
|
218
|
+
- Execution contract:
|
|
219
|
+
- Evaluates `select(ctx)` against current state and upstream data.
|
|
220
|
+
- Returns a subset of declared successor node IDs to activate next wave.
|
|
221
|
+
- Runtime validation: All returned target IDs must be a subset of the node's declared successor edges in `workflow.edges`. An unknown target throws `WorkflowRuntimeError` (`ERR_PRISM_WORKFLOW_ROUTE_TARGET`) fail closed, preventing unauthorized control flow jumps.
|
|
222
|
+
- Returning `[]` activates nothing, contributing to natural idle drain.
|
|
223
|
+
|
|
224
|
+
#### 5. Durable Checkpoints (Schema v2)
|
|
225
|
+
- Extends `WorkflowCheckpointValue` with an additive `execution` block:
|
|
226
|
+
```ts
|
|
227
|
+
execution?: {
|
|
228
|
+
readonly mode: "supersteps";
|
|
229
|
+
readonly superstep: number;
|
|
230
|
+
readonly maxSupersteps: number;
|
|
231
|
+
readonly pending: Readonly<Record<string, { readonly from: readonly string[]; readonly round: number }>>;
|
|
232
|
+
}
|
|
233
|
+
```
|
|
234
|
+
- Increments `WORKFLOW_CHECKPOINT_SCHEMA_VERSION` from `1` to `2`.
|
|
235
|
+
- Checkpoints from cyclic runs record node execution histories through the existing per-node `iterations[]` ledger (`WorkflowLoopIterationRecord`), avoiding unbounded growth while enabling mid-cycle `suspend()` and exact-once CAS resumption via `resumeWorkflow`.
|
|
236
|
+
- Checkpoint persistence cadence: exactly one checkpoint written per wave boundary, ensuring that wave-based execution matches or improves upon today's per-node write overhead.
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
## 7. Universal Invariants, Security Posture, and Threat Model
|
|
241
|
+
|
|
242
|
+
The superstep engine preserves all Prism universal invariants and defends against specific threat vectors introduced by cyclic topologies:
|
|
243
|
+
|
|
244
|
+
| Invariant / Threat Vector | Required Posture & Engine Enforcement |
|
|
245
|
+
|---|---|
|
|
246
|
+
| **Bounded Execution Budgets** | Cyclic workflows can easily become infinite money/resource sinks if unconstrained. The engine enforces an explicit `limits.maxSupersteps` (1–256). Exceeding this ceiling fails closed immediately with `WorkflowSuperstepLimitError` before dispatching any over-budget provider calls or tool actions. |
|
|
247
|
+
| **Fail-Closed Definition Validation** | A definition with cycles and no declared `limits.maxSupersteps` throws `WorkflowDefinitionError("Workflow graph contains a cycle")` with the exact historical error message. Self-loops (`["nodeA", "nodeA"]`) remain strictly forbidden. |
|
|
248
|
+
| **Controlled Control Flow (No Goto-Anywhere)** | `routeNode` cannot execute arbitrary jumps across the system. The selector's returned targets are validated at runtime against declared successors (`targets ⊆ declaredSuccessors`). Any undeclared target fails closed with `ERR_PRISM_WORKFLOW_ROUTE_TARGET`. |
|
|
249
|
+
| **Credential and Identity Isolation** | Node executions in cyclic waves reuse the standard `runNode` harness. Cycles do not widen tool allow-lists, relax `ExecutionPolicy` checks, or grant access to credentials. Tenant, account, and user ownership scopes propagate immutably across waves. |
|
|
250
|
+
| **Secret Redaction Across Waves** | Outputs from cyclic node executions and payloads stored in `pendingActivations` pass through `SecretRedactor` before being written to checkpoint stores or emitted over the `WorkflowEventBus`. |
|
|
251
|
+
| **Checkpoint Integrity & Tamper Resistance** | Checkpoint schema v2 persists `superstep` counters and pending activations under CAS versioning. Resuming a cyclic run requires verifying `definitionHash`, ensuring that workflow topology or limits cannot be tampered with between suspension and resumption. |
|
|
252
|
+
| **Process Safety Hard Caps** | All existing byte caps (`maxNodeOutputBytes` 4 MiB/16 MiB, `maxCheckpointBytes` 1 MiB/8 MiB, `maxStateBytes` 64 KiB/512 KiB) remain strictly enforced on every wave iteration. |
|
|
253
|
+
|
|
254
|
+
---
|
|
255
|
+
|
|
256
|
+
## 8. Plan Mapping & Downstream Task Directives
|
|
257
|
+
|
|
258
|
+
This review establishes the technical baseline and contract requirements for all subsequent tasks in [Plan 130](../../plans/130-Cyclic-Workflow-Graphs.md):
|
|
259
|
+
|
|
260
|
+
- **Task 2: Definition Surface (`types.ts`, `limits.ts`, `define.ts`, `nodes.ts`)**
|
|
261
|
+
- Add `maxSupersteps?: number` to `WorkflowLimits` with `HARD_MAX_SUPERSTEPS = 256`.
|
|
262
|
+
- Add `activation?: "all" | "any"` to `WorkflowNodeBase`.
|
|
263
|
+
- Add `RouteNodeDefinition` and export `routeNode()` factory.
|
|
264
|
+
- Update `defineWorkflow` to permit cycles only when `limits.maxSupersteps` is present, tagging the definition with `execution: "supersteps"`.
|
|
265
|
+
- **Task 3: Superstep Scheduler Engine (`run/superstep.ts`, `run/scheduler.ts`)**
|
|
266
|
+
- Implement wave execution loop with synchronization barrier and idle-drain termination.
|
|
267
|
+
- Implement round-based activation bookkeeping with `"all"` and `"any"` semantics.
|
|
268
|
+
- Extract shared `onNodeSucceeded` seam between Kahn and superstep engines.
|
|
269
|
+
- Introduce `WorkflowSuperstepLimitError` (`ERR_PRISM_WORKFLOW_SUPERSTEP_LIMIT`).
|
|
270
|
+
- **Task 4: `routeNode` Dynamic Routing End-to-End**
|
|
271
|
+
- Wire `routeNode` execution in `run/node-execution.ts`.
|
|
272
|
+
- Enforce declared successor subset validation and runtime fail-closed checks.
|
|
273
|
+
- **Task 5: Checkpoint Schema v2 (`run/checkpoint.ts`, `types.ts`, `limits.ts`)**
|
|
274
|
+
- Bump `WORKFLOW_CHECKPOINT_SCHEMA_VERSION = 2`.
|
|
275
|
+
- Persist `execution` block with `superstep`, `maxSupersteps`, and `pending` activations.
|
|
276
|
+
- Implement CAS resume for mid-cycle suspended runs.
|
|
277
|
+
- **Task 6: Replay, Serialization, and Graph Export (`replay.ts`, `graph-export.ts`)**
|
|
278
|
+
- Support latest-iteration evidence resolution for replay on cyclic graphs.
|
|
279
|
+
- Update Mermaid and DOT exporters to render cycle back-edges cleanly.
|
|
280
|
+
- **Task 7: Example (`examples/cyclic-reflection.ts`)**
|
|
281
|
+
- Build an offline, zero-network multi-agent reflection and dynamic routing demonstration.
|
|
282
|
+
- **Task 8: Documentation Updates (`docs/workflows.md`, `docs/index.md`)**
|
|
283
|
+
- Update public documentation with cyclic workflow guide, limits, and node kind references.
|
|
284
|
+
- **Task 9: Final Verification (`release:gate`)**
|
|
285
|
+
- Execute full test suites, typechecks, and compatibility gates.
|
|
286
|
+
|
|
287
|
+
---
|
|
288
|
+
|
|
289
|
+
## 9. Conclusion
|
|
290
|
+
|
|
291
|
+
Existing Prism primitives (`loopNode`, `conditionalNode`, `fanOutNode`, `suspend`, and `replay`) successfully address single-step iteration, parallel map-reduce, and linear pipelines, but cannot express multi-agent cyclic collaboration without sacrificing per-node observability, granular checkpointing, or declarative graph structure.
|
|
292
|
+
|
|
293
|
+
The superstep activation engine proposed in Plan 130 directly mirrors established industry standards (LangGraph supersteps, Microsoft Agent Framework BSP execution, and AutoGen activation conditions), providing a bounded, fail-closed, durable, and fully backward-compatible solution for cyclic agent workflows.
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# Document ingestion primitive review (plan 132 Task 1)
|
|
2
|
+
|
|
3
|
+
Plan: [132-Anydoc-Docling-Document-Ingestion.md](../../plans/132-Anydoc-Docling-Document-Ingestion.md) Task 1
|
|
4
|
+
Date: 2026-09-26
|
|
5
|
+
Baseline: `@arnilo/prism` 0.11.1, Bun 1.4.2, glibc 2.44, x86_64
|
|
6
|
+
Scope: where a host Markdown converter fits, and what anydoc 0.2.4 / Docling 2.130.0 actually do on this runtime. Research only. No package change.
|
|
7
|
+
|
|
8
|
+
Probes lived in `/tmp/prism-132-probe` and `/tmp/prism-132-docling`. Not committed.
|
|
9
|
+
|
|
10
|
+
## Seams
|
|
11
|
+
|
|
12
|
+
| Seam | What it does today | Converter fit |
|
|
13
|
+
| --- | --- | --- |
|
|
14
|
+
| `parseDocument` / `importDocument` | ZIP-magic OOXML into an editable `DocumentModel` (`doc`/`sheet`/`deck` only). | Do not use. Different product. |
|
|
15
|
+
| `createDocumentReader` / `DocumentParser` | Literal text for the coding read tool. `extract` must return `pages: number` and refuse over `maxPages`. Default peers: pdf-parse, mammoth, in-package xlsx/pptx. Mistral OCR is opt-in and posts to `api.mistral.ai`. | Do not wrap anydoc. Success Markdown has no page count, so a `pages` field would be invented. Leave the reader and Mistral OCR untouched. |
|
|
16
|
+
| Wiki `extractDocument` | `{ bytes, filename, mediaType, title } → { text, format } \| null`. | Host adapter maps `{ markdown, format }` to `{ text, format }`. Do not change the hook signature. `ocrUsed` / `pages` do not fit this return; RAG metadata can carry them. |
|
|
17
|
+
| RAG `Parser` | `replaceDocument({ parser })` calls `parser.parse` and indexes `text` plus metadata. No office parser. `pdfParser` is uncompressed BT/ET text only. | Host parser. No `@arnilo/prism-memory` → `@arnilo/prism-work` dependency (memory peers are only `@arnilo/prism`). |
|
|
18
|
+
| `WorkSandbox.execFile` | `file`, `args`, `cwd`, `env`, `timeout`, `signal`, `onData`. No stdin. Capabilities fail closed to all-false if metadata is incomplete. Env strip is only `M365_` / `GOOGLE_`. Package does not enforce network isolation. | Not the Docling runner. A host callback must supply stdin bytes, kill-on-abort, and stdout/time caps. No new core primitive. |
|
|
19
|
+
|
|
20
|
+
Wiki routing today (`packages/memory/src/wiki/ingest.ts`):
|
|
21
|
+
|
|
22
|
+
- `.txt` `.md` `.json` `.csv` `.html` `.htm`: UTF-8 decode. Hook never runs. `.csv` therefore never becomes an anydoc table unless Task 4 tries the hook first when the host set one.
|
|
23
|
+
- `.pdf`: RAG `pdfParser` first (hard cap 8 MiB, 256 pages). Hook only if that throws. A mixed PDF with text operators succeeds here and never asks for OCR.
|
|
24
|
+
- `.png` `.jpg` `.jpeg` `.gif` `.webp`: stub extract. Hook never runs.
|
|
25
|
+
- anything else (docx, xlsx, pptx, odt, rtf, epub, …): hook, or fail closed.
|
|
26
|
+
|
|
27
|
+
RAG limits that matter: `HARD_MAX_PARSE_MS_CAP` is 30_000 and is not raised by `replaceDocument` itself (built-in parsers enforce it). Do not raise it for OCR. Prefetch models off the request path. Small warm scans fit 30s; a 2 GiB RSS OCR process does not belong in default CI.
|
|
28
|
+
|
|
29
|
+
Trust stays at existing boundaries: wiki tool/command `trust: "untrusted_external"`; RAG `retrieveContext` trust `{ untrusted, inert, injectionCapable }`. The converter does not grant authority.
|
|
30
|
+
|
|
31
|
+
## Frozen result
|
|
32
|
+
|
|
33
|
+
One work-family result. Not a document model.
|
|
34
|
+
|
|
35
|
+
```ts
|
|
36
|
+
interface DocumentExtractionResult {
|
|
37
|
+
readonly markdown: string;
|
|
38
|
+
readonly format: string;
|
|
39
|
+
readonly ocrUsed: boolean;
|
|
40
|
+
readonly pages?: number; // omit unless the engine reported it
|
|
41
|
+
}
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
- anydoc success: `ocrUsed: false`, omit `pages`.
|
|
45
|
+
- anydoc `needsOcr`: throw. No markdown. Carry `pages: number[]` and `pageCount` on the error. Do not copy `error.message` into logs (it is page numbers today; do not depend on that).
|
|
46
|
+
- Docling success only: `ocrUsed: true`, `pages` from `document.num_pages()`.
|
|
47
|
+
- Any other Docling status, including `partial_success`: throw. No text returned.
|
|
48
|
+
|
|
49
|
+
`needsOcr` is the only OCR fallback. `encrypted`, `malformed`, `unsupported`, `resourceLimit`, `missingPart`, `io`, `hosted` never call Docling.
|
|
50
|
+
|
|
51
|
+
## anydoc 0.2.4 on Bun 1.4.2
|
|
52
|
+
|
|
53
|
+
Published package, not GitHub `main`. `engines.node` is `>= 20`. No Bun engine field. N-API optional packages:
|
|
54
|
+
|
|
55
|
+
`darwin-x64`, `darwin-arm64`, `linux-x64-gnu`, `linux-arm64-gnu`, `linux-x64-musl`, `linux-arm64-musl`, `win32-x64-msvc`.
|
|
56
|
+
|
|
57
|
+
No FreeBSD, Android, riscv, or Windows ARM binary. This host loaded `@firecrawl/anydoc-linux-x64-gnu/anydoc.linux-x64-gnu.node` (glibc; ldd: libc, libm, libpthread, libgcc_s, libdl). `bun add` also installed the musl package; the loader did not map it. Do not assume npm's optional-platform filter.
|
|
58
|
+
|
|
59
|
+
`toMarkdownBytes(bytes, format?, options?)` is the JS wrapper. Native conversion ignores `options`. `options.ocr === 'hosted'` catches `needsOcr` and POSTs the whole file to `https://api.firecrawl.dev/v2/parse` (or `FIRECRAWL_API_URL`). Default is reject. Probe monkeypatched `fetch`; default calls made zero network attempts.
|
|
60
|
+
|
|
61
|
+
| Case | Result |
|
|
62
|
+
| --- | --- |
|
|
63
|
+
| `formatFromBytes` DOCX / text PDF / scan PDF | `docx` / `pdf` / `pdf` |
|
|
64
|
+
| `formatFromBytes` CSV and PNG | `null` |
|
|
65
|
+
| `formatFromPath('notes.csv')` / `formatFromExtension('.csv')` | `csv` |
|
|
66
|
+
| DOCX 939 B | `Hello Prism DOCX\n` (17 B). Isolated convert 0.57 ms, VmHWM 58 732 KiB |
|
|
67
|
+
| CSV explicit `'csv'` | GFM table, 55 B, 0.14 ms |
|
|
68
|
+
| CSV omitted or `null` format | `code: 'unsupported'`. No OCR |
|
|
69
|
+
| text PDF 872 B, 2 pages | `## Hello Prism PDF page 1\n\n## Hello Prism PDF page 2\n` (53 B). Isolated convert 2.13 ms, VmHWM 61 872 KiB. Warm in-process 0.78 ms |
|
|
70
|
+
| image-only PDF 3007 B | `needsOcr`, `pages: [1]`, `pageCount: 1`, message `page 1 of 1 needs OCR`. No markdown. Isolated 2.02 ms |
|
|
71
|
+
| mixed PDF (2 text pages + 1 scan) | `needsOcr`, `pages: [3]`, `pageCount: 3`. No partial markdown |
|
|
72
|
+
| PNG / garbage | `unsupported` |
|
|
73
|
+
| `toDocument` DOCX | `{ blocks, notes, assets }` only. No pages |
|
|
74
|
+
| `toDocument` PDF | `unsupported` — PDF has no document model |
|
|
75
|
+
|
|
76
|
+
Import of the native module: ~5–8 ms. Import-only VmHWM 56 212 KiB. No `AbortSignal`. Native work is not preemptable.
|
|
77
|
+
|
|
78
|
+
Not attested by this probe: `encrypted` (no fixture), `resourceLimit` (no bomb), `hosted` (must not be called). Markdown-only API cannot attest success page count, which pages were text vs image, layout boxes, encryption-on-success, truncation, or asset bytes.
|
|
79
|
+
|
|
80
|
+
## Docling 2.130.0
|
|
81
|
+
|
|
82
|
+
`requires_python` is `>=3.10,<4`. This host is CPython 3.14.7. `docling==2.130.0` imports. Standard extra is `docling-slim[standard]==2.130.0` (torch, rapidocr 3.9.2). onnxruntime is not in that extra.
|
|
83
|
+
|
|
84
|
+
Default PyPI torch for cp314 is the CUDA build and pulls nvidia wheels (cublas/nccl/cusolver, multi-GB). Probe killed that install and used `torch==2.14.0+cpu` and `torchvision==0.29.0+cpu` from `https://download.pytorch.org/whl/cpu`. `torch.cuda.is_available()` was false. Task 3 docs must say CPU index unless the host pins a GPU build. Python stays host-provisioned, not an npm dependency.
|
|
85
|
+
|
|
86
|
+
`OcrAutoOptions` (the default, `do_ocr=True`) on Linux tries nemotron, then rapidocr/onnxruntime, then easyocr, then rapidocr/torch. This process logged `Auto OCR model selected rapidocr with torch.` Pin `RapidOcrOptions(backend="torch")`. Do not leave auto: another host extra selects a different engine. Docstring that says "EasyOCR if GPU else Tesseract" is stale.
|
|
87
|
+
|
|
88
|
+
Prefetch: `download_models(artifacts, with_easyocr=False, with_picture_classifier=False)` took 149.9 s and wrote 1 405 021 041 bytes (`du` 1.4 G): layout heron, tableformer, code-formula, RapidOcr torch `ch` models, and RapidOcr onnx `ch` models (downloaded even though the selected backend was torch). Hugging Face warned about unauthenticated requests during prefetch. Runtime must set `artifacts_path` to that directory. `enable_remote_services=False` did not raise.
|
|
89
|
+
|
|
90
|
+
`DocumentStream(name, BytesIO)` works. The name suffix selects format (`scan.pdf` → PDF, `scan.png` → IMAGE). Use `ImageFormatOption` for images and `PdfFormatOption` for PDFs, both with the same `PdfPipelineOptions`. Do not pass a path or URL into the helper.
|
|
91
|
+
|
|
92
|
+
| Case | Result |
|
|
93
|
+
| --- | --- |
|
|
94
|
+
| Liberation Sans scan PDF, 8215 B, "SCAN 132" | `success`, `num_pages()==1`, markdown `SCAN 132`, 4.85 s (weights loaded in this process) |
|
|
95
|
+
| Same page as PNG, 7203 B | `success`, 1 page, markdown `## SCAN 132`, 2.17 s. PDF and PNG markdown are not identical |
|
|
96
|
+
| text PDF 872 B | `success`, 2 pages, correct text, 1.38 s. Not the plan route — anydoc handles text PDFs |
|
|
97
|
+
| `max_num_pages=1` on 3-page PDF, `raises_on_error=True` | `ConversionError`: status `failure`, `Document has 3 pages, exceeding the max_num_pages limit of 1.` 0.05 s. No text |
|
|
98
|
+
| same, `raises_on_error=False` | status `failure`, empty markdown, `FailureCategory.POLICY` |
|
|
99
|
+
| `max_file_size=100` | status `failure`, empty markdown, size message. Defaults are `sys.maxsize` — the helper must set both limits |
|
|
100
|
+
| `document_timeout=0.001` | status `partial_success`, errors `document timeout exceeded` and a pipeline stage timeout, markdown empty this time. `convert()` does not raise on `PARTIAL_SUCCESS` even when `raises_on_error=True` (`document_converter.py` treats it like success). Source also sets `PARTIAL_SUCCESS` when some pages complete (`standard_pdf_pipeline.py`). Refuse every non-`success` |
|
|
101
|
+
| offline rerun | `HF_HUB_OFFLINE=1`, `TRANSFORMERS_OFFLINE=1`, `socket.create_connection` raising. `success`, `networkAttempts: []`. `unshare --net` failed with `Operation not permitted`, so this is a process socket guard, not a network namespace |
|
|
102
|
+
|
|
103
|
+
RSS, one converter, artifacts already on disk: init VmHWM 1 093 704 KiB; after first scan 1 786 460 KiB; after PNG 1 997 992 KiB. A second converter in the same process reached 3 101 088 KiB. Budget one OCR job, not a pool.
|
|
104
|
+
|
|
105
|
+
A tofu-font ImageMagick page (glyphs missing) still returned `success` with markdown `<!-- image -->\n\n00000000`. `SUCCESS` is not correct text. Keep output untrusted. Do not golden-match OCR except a font-rendered fixture.
|
|
106
|
+
|
|
107
|
+
## Thresholds for later tasks
|
|
108
|
+
|
|
109
|
+
This host, small fixtures. Not upstream benchmark claims (anydoc median 4.4 ms, Docling marketing scores).
|
|
110
|
+
|
|
111
|
+
- anydoc office or text PDF under 10 KiB: under 500 ms, process VmHWM under 128 MiB. Measured 0.6–2.1 ms and 57–62 MiB including Bun.
|
|
112
|
+
- anydoc `needsOcr` reject: under 500 ms, zero markdown bytes. Measured ~2 ms.
|
|
113
|
+
- Docling OCR under 100 KiB after artifacts are on disk: under 30 s, one-converter VmHWM under 4 GiB. Measured ~5 s cold / ~2 s warm and ~2.0 GiB. A later real fixture that exceeds 2× without a recorded reason fails the task.
|
|
114
|
+
- Prefetch (150 s, 1.4 GiB) is provisioning, not a request-path budget.
|
|
115
|
+
|
|
116
|
+
## Threat notes
|
|
117
|
+
|
|
118
|
+
- anydoc default path is local. One options flag uploads the whole PDF. The work API must not expose `ocr: 'hosted'` or read `FIRECRAWL_API_KEY`.
|
|
119
|
+
- Docling model bytes come from Hugging Face at prefetch. Pin versions. Runtime uses `artifacts_path` and `enable_remote_services=False`. No `remote-serving` extra.
|
|
120
|
+
- Over-limit Docling input fails with empty markdown (probed). Partial timeout does not raise. Check `status == success` or a timed-out job can be indexed.
|
|
121
|
+
- Mixed PDF: anydoc emits no markdown. Wiki's current PDF built-in can index the text pages and skip the scan. Hook-first when a host extractor is set; null/failure must not fall back.
|
|
122
|
+
- Caps before native anydoc (it accepts no abort and no `maxPages`). Caps inside the Docling `convert()` call (`max_num_pages`, `max_file_size`) plus a worker wall clock (`document_timeout`), because defaults are unbounded.
|
|
123
|
+
- Worker env is not scrubbed by `assertWorkSandboxEnv` beyond two prefixes. Host runner passes an explicit env, no ambient tokens.
|
|
124
|
+
- Helper reads stdin bytes only. No path, no URL, no `HttpSource`.
|
package/docs/history/README.md
CHANGED
|
@@ -8,6 +8,8 @@ page instead and note the archive where provenance matters.
|
|
|
8
8
|
- `release-handoffs.md`: operator publish handoffs per release line (the live [release and install](../release-and-install.md) page keeps current install, inventory, gates, and peer policy).
|
|
9
9
|
- `migrate-to-0.4.md`: retired 0.3-era package reorganization guide (current line: [migrate-to-0.5.md](../migrate-to-0.5.md)).
|
|
10
10
|
- `0.1.0-readiness.md`: frozen 0.1.x readiness record.
|
|
11
|
+
- `retire-node-runtime.md`: the 0.1.x–0.11.x Node 22/24 support story, its two measured CI legs, and the 0.0.16 Node compatibility matrix — retired in 0.12.0 with the `engines.bun` flip.
|
|
12
|
+
- `migrate-to-0.12.0.md`: 0.11 → 0.12 host checklist (Bun runtime, `bun:sqlite` mapping, optional anydoc/Docling).
|
|
11
13
|
- `persistence-credentials-multimodality-primitives.md`, `workflow-orchestration-primitives.md`, `workflow-tui-primitives.md`: plan-era primitive-review inventories.
|
|
12
14
|
- `079-messaging-primitive-review.md`, `080-messaging-followon-primitive-review.md`: messaging-channel primitive reviews (079 shipped runtime; 080 follow-ons + 0.8.0 cut).
|
|
13
15
|
- `081-connected-apps-primitive-review.md`: connected-apps / work-HTTP primitive review (081; page written in 081 Task 1).
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Migrate Prism 0.11 to 0.12
|
|
2
|
+
|
|
3
|
+
> Archived host checklist for the 0.12.0 cut. Current install rules stay in [release and install](../release-and-install.md). Current extraction contract stays in [document extraction](../document-extraction.md).
|
|
4
|
+
|
|
5
|
+
0.12.0 moves all twelve publishable manifests together. The runtime is Bun `>=1.4.2`. Optional document extraction is not part of a default install.
|
|
6
|
+
|
|
7
|
+
## Runtime
|
|
8
|
+
|
|
9
|
+
Install with `bun add`, not `npm install`. Example: `bun add @arnilo/prism @arnilo/prism-core`.
|
|
10
|
+
|
|
11
|
+
Every publishable manifest declares `engines.bun >=1.4.2` and no longer declares `engines.node`. Bun 1.4.2 does not enforce `engines.bun`. A Node process still installs the tarball, then fails when it reaches SQLite:
|
|
12
|
+
|
|
13
|
+
- `@arnilo/prism-core/sessions/sqlite requires the Bun runtime (bun:sqlite).`
|
|
14
|
+
- `@arnilo/prism-core/governance/prompts requires the Bun runtime (bun:sqlite).`
|
|
15
|
+
|
|
16
|
+
A static `import "bun:sqlite"` on Node fails earlier with `ERR_UNSUPPORTED_ESM_URL_SCHEME`. The published 0.11.x line remains the Node host path. Do not mix a 0.12.0 package into a Node process.
|
|
17
|
+
|
|
18
|
+
`prism init` and provider-add scaffolds write `packageManager` `bun@1.4.2` and Bun install commands. The dev hint is `bun add --dev @arnilo/prism-coding-tools`.
|
|
19
|
+
|
|
20
|
+
Release-host exception: `npm pack`, `npm publish`, and `npm sbom` stay on the release host. `bun pack` does not exist, and `bun publish` on 1.4.2 has no `--provenance`. Contributors do not run those commands.
|
|
21
|
+
|
|
22
|
+
## SQLite
|
|
23
|
+
|
|
24
|
+
The `better-sqlite3` optional peer is gone. There is no SQLite package to install. The driver is the runtime's `bun:sqlite`. Existing database files stay; FTS DDL is unchanged.
|
|
25
|
+
|
|
26
|
+
Measured on Bun 1.4.2, against the old driver:
|
|
27
|
+
|
|
28
|
+
| Row | `better-sqlite3` | `bun:sqlite` |
|
|
29
|
+
| --- | --- | --- |
|
|
30
|
+
| Blob read | `Buffer` | `Uint8Array`, not a `Buffer` |
|
|
31
|
+
| `lastInsertRowid` | number | number. `safeIntegers: true` makes it `bigint` |
|
|
32
|
+
| Integer above `2^53-1` | driver-defined | rounds (`9007199254740993n` stores as `9007199254740992`) unless `safeIntegers` |
|
|
33
|
+
| `undefined` bind | throws | stores NULL |
|
|
34
|
+
| `get()` miss | `undefined` | `null` |
|
|
35
|
+
| Constructor options | accepted | no options object. Set `PRAGMA` with `exec`, including `busy_timeout` |
|
|
36
|
+
|
|
37
|
+
Hosts that passed a constructor options object, or that treated a missing row as `undefined`, must update those call sites. Hosts that only used the Prism session store do not.
|
|
38
|
+
|
|
39
|
+
## Optional document extraction
|
|
40
|
+
|
|
41
|
+
Default hosts skip this. Nothing in the default import path installs Python, downloads a model, or calls OCR.
|
|
42
|
+
|
|
43
|
+
Hosts that want local Markdown conversion:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
bun add @arnilo/prism-work @firecrawl/anydoc@0.2.4
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`@firecrawl/anydoc` is an optional native peer. Conversion stays on the host. The subpath never passes hosted OCR options and never reads an API key. A missing peer fails closed. Unrelated work and memory imports do not need the peer.
|
|
50
|
+
|
|
51
|
+
Scanned PDFs and explicit image OCR need a host-provisioned Docling worker. Python and Docling are not npm dependencies. Pin `docling==2.130.0` and the CPU torch index. Prefetch models once (`docling-tools models download`, about 1.4 GiB and 150 s) and point the worker at that directory. The package ships `docling/ocr.py`. The host spawns it with a scrubbed env (`HF_HUB_OFFLINE=1`, telemetry off), one OCR job in flight, and no network egress. `WorkSandbox.execFile` cannot feed stdin, so it is not the runner.
|
|
52
|
+
|
|
53
|
+
OCR never uploads by default. A failed, partial, or over-page conversion returns no Markdown and must not be indexed. One scanned page was about 5 s cold and 2 s warm after prefetch, with about 2 GiB resident. Budget under 30 s and 4 GiB for inputs under 100 KiB. The default no-OCR path does not pay that cost.
|
|
54
|
+
|
|
55
|
+
## Rollback
|
|
56
|
+
|
|
57
|
+
Stay on 0.11.1 if the process must be Node, or if a host still depends on the `better-sqlite3` peer. 0.12.0 packages do not load under Node.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Retiring the Node runtime (0.12.0)
|
|
2
|
+
|
|
3
|
+
> Archived record. Node runtime support retired in **0.12.0**; the live runtime contract is
|
|
4
|
+
> [release and install](../release-and-install.md). Kept verbatim for audit — do not read it as
|
|
5
|
+
> current guidance.
|
|
6
|
+
|
|
7
|
+
## What the 0.1.x–0.11.x line declared
|
|
8
|
+
|
|
9
|
+
Through 0.11.1 Prism described itself as a TypeScript/Node.js agent harness and declared
|
|
10
|
+
`engines.node >=22` in every publishable manifest. The support matrix was frozen by Phase 12 Task 0
|
|
11
|
+
in `scripts/phase12-freeze-manifest.json`:
|
|
12
|
+
|
|
13
|
+
| Runtime | Supported | Measured in CI |
|
|
14
|
+
| --- | --- | --- |
|
|
15
|
+
| Node | 22, 24 (`engines.node >=22`, the 0.x floor) | two declared legs: `verify` imports every public root `exports` default target on Node 24, and `node22-compat` runs the same public export imports on Node 22, for the declared `engines.node >=22` this table recorded. |
|
|
16
|
+
|
|
17
|
+
The floor moved once: Node 20 was dropped in 0.6.0 (`dev-006`; Node 20 reached upstream end-of-life
|
|
18
|
+
2026-04-30) and the `node20-compat` CI leg became `node22-compat`. `@types/node` stayed on the
|
|
19
|
+
published floor (`^22.20.0`) so that an API the published line could not provide failed the build
|
|
20
|
+
instead of compiling clean against a newer type surface. The docs-example path needed Node >=22.6
|
|
21
|
+
native TypeScript stripping, which is why the import smoke (not the full demo run) was the Node 22
|
|
22
|
+
leg.
|
|
23
|
+
|
|
24
|
+
**Recorded compatibility matrix (2026-07-26, release 0.0.16):**
|
|
25
|
+
|
|
26
|
+
| Leg | Node | Result |
|
|
27
|
+
| --- | --- | --- |
|
|
28
|
+
| Full SDK readiness (`bun run sdk:ready`: typecheck, lint, format, test, coverage, pack, release:gate) | 24.18.0 (current) | ✅ green — 1312/1312 tests, lint 0 errors, format clean, coverage 64/72/79 vs 60/70/75 thresholds. |
|
|
29
|
+
| Build toolchain (`tsc` 7.0.2, `biome` 2.5.13) | 20.20.2 (LTS iron) | ✅ both run under Node 20. |
|
|
30
|
+
| Public surface import smoke (all 21 root `exports` default targets) | 20.20.2 | ✅ all import cleanly. |
|
|
31
|
+
| Full core test suite | 20.20.2 | 1311/1312 — the single failure is `examples_demos_run_to_completion_and_emit_no_secret`, which executes `examples/*.ts` via Node's native TypeScript stripping (Node 22.6+). This is a test-harness capability, not an SDK runtime incompatibility, and is exactly why CI scoped Node 20 to build + import smoke. |
|
|
32
|
+
|
|
33
|
+
## What changed
|
|
34
|
+
|
|
35
|
+
Plans 124–129 moved the contributor toolchain, the test runner, the CI runtime, and finally the
|
|
36
|
+
consumer contract to Bun:
|
|
37
|
+
|
|
38
|
+
- **Plan 124** moved every test stage, script wrapper, and CI step to the Bun binary; the only
|
|
39
|
+
surviving Node process is the branch-coverage audit (`scripts/branch-coverage-audit.mjs`), because
|
|
40
|
+
Bun 1.4.2 emits no branch data.
|
|
41
|
+
- **Plan 125 Task 1** replaced `engines.node` with `engines.bun >=1.4.2` in all twelve publishable
|
|
42
|
+
manifests and retired the two declared Node legs together with it. Both legs were *measured*
|
|
43
|
+
(each ran `scripts/public-import-smoke.mjs`, since deleted) rather than dead, so the retirement
|
|
44
|
+
removed a real check and replaced it with the packed-consumer import sweep that imports every
|
|
45
|
+
public subpath of every package on Bun (`scripts/packaging-current.test.mjs`).
|
|
46
|
+
- **Plan 125 Task 4** rewrote the live install and runtime documentation around `bun add`, moved the
|
|
47
|
+
Node story here, and recorded the retirement in `CHANGELOG.md`.
|
|
48
|
+
|
|
49
|
+
`engines.bun` is advisory metadata: neither npm nor Bun 1.4.2 enforces it (a manifest declaring
|
|
50
|
+
`engines.bun >=99.0.0` installs silently under both), so the retirement's compensating control is the
|
|
51
|
+
import-side failure message that plan 126 Task 2 adds when the durable storage layer moves to
|
|
52
|
+
`bun:sqlite`. A Node host running a 0.12.0 package fails closed at import with an actionable
|
|
53
|
+
message instead of at install time. No 0.12.0 tarball reaches the registry before plan 129's cut, so
|
|
54
|
+
no Node host sees the gap without the message.
|
|
55
|
+
|
|
56
|
+
The registry keeps the 0.11.x line for Node hosts; hosts that install with npm can still install
|
|
57
|
+
Prism (the published 0.11.x tarballs declare `engines.node >=22`), but the runtime contract for the
|
|
58
|
+
0.12.0 line is Bun. The release host keeps npm for `pack`/`publish`/`sbom` — runner images ship
|
|
59
|
+
Node, `bun pack` does not exist, and `bun publish` has no `--provenance` on 1.4.2 (plan 125 Task 5).
|
package/docs/hooks.md
CHANGED
|
@@ -175,7 +175,7 @@ event onto the seams above — no Prism internals, `@arnilo/prism` as its only p
|
|
|
175
175
|
runtime dependency. It compiles the five events marked *compiled* in the map.
|
|
176
176
|
|
|
177
177
|
```bash
|
|
178
|
-
|
|
178
|
+
bun add @arnilo/prism @arnilo/prism-hooks
|
|
179
179
|
```
|
|
180
180
|
|
|
181
181
|
```ts
|
|
@@ -102,10 +102,10 @@ Always install containing packages directly. **NPM install never accepts subpath
|
|
|
102
102
|
|
|
103
103
|
```bash
|
|
104
104
|
# Correct — install containing published packages:
|
|
105
|
-
|
|
105
|
+
bun add @arnilo/prism @arnilo/prism-core @arnilo/prism-providers @arnilo/prism-work
|
|
106
106
|
|
|
107
107
|
# Never install subpaths:
|
|
108
|
-
#
|
|
108
|
+
# bun add @arnilo/prism-work/connectors (WRONG: fails with 404 / E404)
|
|
109
109
|
```
|
|
110
110
|
|
|
111
111
|
In your application code, import from documented subpaths:
|
package/docs/host-security.md
CHANGED
|
@@ -171,7 +171,7 @@ Wire those values where they matter: provider adapters receive the resolved cred
|
|
|
171
171
|
- `npm audit --audit-level=high`: 0 vulnerabilities at every severity.
|
|
172
172
|
- Lockfile: 162 registry dependency records, all with `resolved` provenance URL and integrity hash; `npm ls --all` reports a clean graph.
|
|
173
173
|
- License inventory: 160 locked third-party packages; all declare permissive MIT, ISC, BSD, Apache-2.0, or compatible dual licenses. No GPL, AGPL, SSPL, or missing lockfile license metadata.
|
|
174
|
-
- Install scripts:
|
|
174
|
+
- Install scripts: no package in the family runs an install script.
|
|
175
175
|
- Secret scan: source, tests, docs, workflow files, package metadata, built tests, packed-install canary, and tarball deny-list checks found no private-key block or common live-token prefix. Runtime redaction fixtures cover requests, events, ledgers, stores, checkpoints, provider/OAuth errors, and credential ciphertext.
|
|
176
176
|
- Threat suites pass for parameterized SQL/tenant isolation, HTTP URL/SSRF rejection, realpath/symlink containment, shell-metacharacter approval, schema prototype-pollution/remote-reference bounds, OAuth polling/abort/redaction, credential tamper/wrong-key/KDF floors, MCP result bounds/timeouts, and coding approval/path policy. `security:threat-suites` also gates CodeQL-remediation regressions (plan 038): linear `trimTrailingSlashes`/parsers with no environment regex evaluation, single-pass HTML sanitization, crypto (not `Math.random`) fixture identifiers, and no clear-text error logging on password-handling paths.
|
|
177
177
|
|