@db3.ai/app 0.1.0-beta.1

package/src/flows/README.md

@@ -0,0 +1,144 @@
+# Flows
+
+## Runnable flow and testing
+
+The base `App` does not automatically supply a `flows` property. Copy the
+installed package's `src/flows/examples/` into `examples/` to use the explicit
+`FlowExampleApp` service extension, then run `npx tsx examples/runTextFlow.ts`
+with disposable SQL credentials. It creates/removes its own test database and
+file-definition directory. Two blocks normalize and format text; the lab checks
+steps, progress, blank-input failure and original/latest replay after repair.
+The website includes the exact `tests/examples/runTextFlow.test.ts` consumer
+test. A definition snapshot preserves graph/configuration, not executable code;
+version block types when changing runtime behavior for historical graphs.
+
+Flows are durable, observable graphs layered over the existing queue. A flow definition is the source of truth for both the runtime and visual designer. It contains stable ULIDs, named input/output contracts, configured block occurrences, named-port connections, and graph positions.
+
+Definitions are intentionally separate from run data:
+
+- A `FlowDefinitionStore` reads and writes source definitions.
+- `FileFlowDefinitionStore` stores deterministic `flow.json` files suitable for Git.
+- `FlowRun`, `FlowStepRun`, and `FlowRunEvent` persist invocations, every block boundary, logs, failures, timing, and replay lineage.
+- `FlowStepJob` executes one block at a time through the application's existing queue.
+
+The initial compiler accepts one connected sequential path. Branches and merges are rejected until their scheduling and replay semantics are explicit.
+
+## App Service
+
+An app exposes its configured service as `app().flows`:
+
+```ts
+import { fileURLToPath } from 'node:url';
+import { FileFlowDefinitionStore, Flows } from '@db3.ai/app/flows';
+
+class App extends FrameworkApp {
+	get flows(): Flows {
+		return this.service('flows', () => new Flows({
+			queue: this.queue,
+			queueName: 'flows',
+			definitions: new FileFlowDefinitionStore({
+				root: fileURLToPath(new URL('./flows', import.meta.url)),
+			}),
+			blocks: flowBlocks,
+		}));
+	}
+}
+```
+
+`FlowStepJob` resolves that service from the active application when the queue
+worker handles it. No flow-specific queue context or service injection is
+required.
+
+Install `QueuedJob`, `FailedJob`, and `...FLOW_MODELS` in the app database schema.
+
+## Function Blocks
+
+A function-backed block is one file with a serializable contract and one function:
+
+```ts
+import { defineBlock, type FlowValues } from '@db3.ai/app/flows';
+
+type TextValues = FlowValues & {
+	text: string;
+};
+
+export default defineBlock<TextValues, TextValues>({
+	type: 'text.uppercase',
+	name: 'Uppercase',
+	inputs: {
+		text: { type: 'string', required: true },
+	},
+	outputs: {
+		text: { type: 'string', required: true },
+	},
+	run: input => ({
+		text: input.text.toUpperCase(),
+	}),
+});
+```
+
+The block context exposes durable `log(level, message, data)` entries. A debug tap can therefore log input and return it unchanged without introducing a separate instrumentation system.
+
+Database operations should be explicit app-owned block types such as `database.fetch-content-plan` or `database.insert-article`. These blocks use the app's models and ownership rules. The framework deliberately does not expose a generic arbitrary-table or SQL block, because that would bypass model conversion, tenancy, authorization, and reviewable contracts.
+
+## Definition-Driven Inputs
+
+Public flow inputs and block configuration use `FlowValueDefinition`. In addition to runtime type, requirement, description, and default information, a value may carry optional host-agnostic editor hints:
+
+```ts
+const request = {
+	type: 'json',
+	required: true,
+	editor: {
+		component: 'DomJsonInput',
+		label: 'Article request',
+		rows: 8,
+	},
+} satisfies FlowValueDefinition;
+```
+
+Editor metadata never changes runtime validation. A host may map component names to DOM form fields, native controls, or another component registry while keeping the flow definition as the source of truth.
+
+## Flow Boundaries And Subflows
+
+Every `Flows` service registers three framework-owned structural blocks:
+
+- `flow.input` exposes the containing definition's public inputs inside the graph.
+- `flow.output` terminates the graph and exposes the containing definition's public outputs.
+- `flow.subflow` groups another definition behind one generic nested-flow block.
+
+The child definition is the only public-contract source for a subflow. A parent occurrence only needs the generic type and the child ULID:
+
+```json
+{
+	"id": "01KXDNZE8KRZ9W9AKZAXFEQXPG",
+	"type": "flow.subflow",
+	"name": "Greeting Agent",
+	"flowId": "01KXDPRR0NT39S75V942MEV34X",
+	"position": { "x": 625, "y": 44 }
+}
+```
+
+The compiler resolves that occurrence's named ports from the referenced definition's `inputs` and `outputs`. No app-specific wrapper block is required. The child graph connects its `flow.input` boundary through its implementation to its `flow.output` boundary, so changing the child's public contract updates how every parent renders and validates that subflow.
+
+At runtime the parent step enters `waiting`, a linked child run executes through the queue, and the child output or failure resumes the parent. Execution snapshots include the resolved occurrence contracts and every nested definition. Original-definition replay therefore follows the historical child graph, while latest-definition replay resolves the current child source.
+
+## Running And Replaying
+
+```ts
+const run = await app().flows.run(flowId, {
+	name: 'Ada',
+});
+
+const details = await app().flows.runDetails(run.id!);
+
+await app().flows.replay(run.id!, {
+	definition: 'original',
+});
+```
+
+The runtime captures JSON input and output at every boundary with a default one-megabyte limit. This first version does not redact values automatically, so applications must not pass credentials or unfiltered secrets through captured flow values.
+
+## Definition Providers
+
+`FileFlowDefinitionStore` recursively reads `flow.json` and `*.flow.json`, writes tab-formatted deterministic JSON atomically, and uses SHA-256 revisions for optimistic saves. A future database provider should implement the same `FlowDefinitionStore` contract and store the same definition shape; the runtime must continue to execute definitions rather than a separate UI representation.

package/src/flows/blocks/FlowInput.ts

@@ -0,0 +1,32 @@
+import type { FlowValues } from '../contracts';
+import { defineBlock } from '../defineBlock';
+
+/** Stable framework type for the visible public-input boundary node. */
+export const FLOW_INPUT_BLOCK_TYPE = 'flow.input';
+
+/**
+ * Framework-owned input boundary whose effective ports come from its flow.
+ *
+ * The fallback `value` port keeps an unspecialized designer node connectable.
+ * The compiler replaces it with the source definition's complete input contract.
+ */
+export const FlowInput = defineBlock<FlowValues, FlowValues>({
+	type: FLOW_INPUT_BLOCK_TYPE,
+	name: 'Flow Input',
+	description: 'Exposes the public flow inputs to blocks inside this definition.',
+	inputs: {
+		value: { type: 'json', required: true },
+	},
+	outputs: {
+		value: { type: 'json', required: true },
+	},
+	/**
+	 * Passes validated public flow input through its visible output handles.
+	 *
+	 * @param input - Public flow input validated against the resolved contract.
+	 * @returns Unchanged input values.
+	 */
+	run(input) {
+		return input;
+	},
+});

package/src/flows/blocks/FlowOutput.ts

@@ -0,0 +1,32 @@
+import type { FlowValues } from '../contracts';
+import { defineBlock } from '../defineBlock';
+
+/** Stable framework type for the visible public-output boundary node. */
+export const FLOW_OUTPUT_BLOCK_TYPE = 'flow.output';
+
+/**
+ * Framework-owned output boundary whose effective ports come from its flow.
+ *
+ * Reaching this node terminates the current sequential run and exposes its
+ * values through the public contract of the containing flow.
+ */
+export const FlowOutput = defineBlock<FlowValues, FlowValues>({
+	type: FLOW_OUTPUT_BLOCK_TYPE,
+	name: 'Flow Output',
+	description: 'Terminates the graph and exposes named values to parent flows and callers.',
+	inputs: {
+		value: { type: 'json', required: true },
+	},
+	outputs: {
+		value: { type: 'json', required: true },
+	},
+	/**
+	 * Passes terminal values through to the public flow result.
+	 *
+	 * @param input - Values delivered to the output boundary.
+	 * @returns Unchanged public output values.
+	 */
+	run(input) {
+		return input;
+	},
+});

package/src/flows/blocks/Subflow.ts

@@ -0,0 +1,24 @@
+import type { FlowValues } from '../contracts';
+import { defineBlock } from '../defineBlock';
+
+/** Stable framework type for a definition-backed nested flow block. */
+export const SUBFLOW_BLOCK_TYPE = 'flow.subflow';
+
+/**
+ * Generic nested-flow placeholder resolved from its referenced definition.
+ *
+ * The fallback port allows a newly dropped placeholder to be connected before
+ * its child definition is created. Once linked, the child contract replaces it.
+ */
+export const Subflow = defineBlock<FlowValues, FlowValues>({
+	type: SUBFLOW_BLOCK_TYPE,
+	kind: 'flow',
+	name: 'Subflow',
+	description: 'Groups a nested flow behind the inputs and outputs declared by its definition.',
+	inputs: {
+		value: { type: 'json', required: true },
+	},
+	outputs: {
+		value: { type: 'json', required: true },
+	},
+});

package/src/flows/blocks/index.ts

@@ -0,0 +1,11 @@
+export * from './FlowInput';
+export * from './FlowOutput';
+export * from './Subflow';
+
+import type { FlowBlockDefinition } from '../contracts';
+import { FlowInput } from './FlowInput';
+import { FlowOutput } from './FlowOutput';
+import { Subflow } from './Subflow';
+
+/** Framework-owned structural blocks registered for every flow service. */
+export const FLOW_SYSTEM_BLOCKS: FlowBlockDefinition[] = [FlowInput, FlowOutput, Subflow];

package/src/flows/constants.ts

@@ -0,0 +1,62 @@
+/**
+ * Durable lifecycle states for a complete flow invocation.
+ */
+export const FLOW_RUN_STATUS = {
+	queued: 'queued',
+	running: 'running',
+	completed: 'completed',
+	failed: 'failed',
+} as const;
+
+/**
+ * Durable lifecycle states for one block occurrence in a run.
+ */
+export const FLOW_STEP_STATUS = {
+	pending: 'pending',
+	queued: 'queued',
+	running: 'running',
+	waiting: 'waiting',
+	completed: 'completed',
+	failed: 'failed',
+} as const;
+
+/**
+ * Ordered event types emitted while flow runs progress.
+ */
+export const FLOW_RUN_EVENT_TYPE = {
+	runCreated: 'run.created',
+	runStarted: 'run.started',
+	runCompleted: 'run.completed',
+	runFailed: 'run.failed',
+	runReplayed: 'run.replayed',
+	stepQueued: 'step.queued',
+	stepStarted: 'step.started',
+	stepWaiting: 'step.waiting',
+	stepLog: 'step.log',
+	stepCompleted: 'step.completed',
+	stepRetrying: 'step.retrying',
+	stepFailed: 'step.failed',
+	nestedStarted: 'nested.started',
+	nestedCompleted: 'nested.completed',
+	nestedFailed: 'nested.failed',
+} as const;
+
+/**
+ * Supported replay sources for a new run.
+ */
+export const FLOW_REPLAY_DEFINITION = {
+	original: 'original',
+	latest: 'latest',
+} as const;
+
+/** Durable flow-run status value. */
+export type FlowRunStatus = typeof FLOW_RUN_STATUS[keyof typeof FLOW_RUN_STATUS];
+
+/** Durable flow-step status value. */
+export type FlowStepStatus = typeof FLOW_STEP_STATUS[keyof typeof FLOW_STEP_STATUS];
+
+/** Ordered flow-run event type. */
+export type FlowRunEventType = typeof FLOW_RUN_EVENT_TYPE[keyof typeof FLOW_RUN_EVENT_TYPE];
+
+/** Definition source used to create a replayed run. */
+export type FlowReplayDefinition = typeof FLOW_REPLAY_DEFINITION[keyof typeof FLOW_REPLAY_DEFINITION];

package/src/flows/contracts/FlowBlock.ts

@@ -0,0 +1,73 @@
+import type { FlowBlockInstance } from './FlowDefinition';
+import type { FlowValue, FlowValueDefinitions, FlowValues } from './FlowValue';
+
+/**
+ * Severity attached to one durable block log entry.
+ */
+export type FlowLogLevel = 'debug' | 'info' | 'warning' | 'error';
+
+/** Execution strategy owned by a registered block type. */
+export type FlowBlockKind = 'function' | 'flow';
+
+/**
+ * Runtime information and services exposed to an executing block function.
+ */
+export interface FlowBlockContext<TConfig extends FlowValues = FlowValues> {
+	/** Flow definition currently being executed. */
+	flowId: string;
+	/** Durable run ULID. */
+	runId: string;
+	/** Durable step-run ULID. */
+	stepRunId: string;
+	/** Configured block occurrence being executed. */
+	block: FlowBlockInstance;
+	/** Validated block configuration with defaults applied. */
+	config: TConfig;
+	/**
+	 * Records an ordered durable log entry for this block.
+	 *
+	 * @param level - Log severity.
+	 * @param message - Human-readable log message.
+	 * @param data - Optional structured diagnostic data.
+	 */
+	log(level: FlowLogLevel, message: string, data?: FlowValue): Promise<void>;
+}
+
+/**
+ * Executable block type registered with a Flows service.
+ */
+export interface FlowBlockDefinition<
+	TInput extends FlowValues = FlowValues,
+	TOutput extends FlowValues = FlowValues,
+	TConfig extends FlowValues = FlowValues,
+> {
+	/** Stable namespaced type used by serialized flow definitions. */
+	type: string;
+	/** Whether this block executes a function or delegates to a nested flow. */
+	kind?: FlowBlockKind;
+	/** Human-readable block type name. */
+	name: string;
+	/** Optional explanation shown in block palettes and inspectors. */
+	description?: string;
+	/** Whether this block safely preserves a value when automatically inserted on a compatible connection. */
+	insertable?: boolean;
+	/** Named values accepted by the block function. */
+	inputs: FlowValueDefinitions;
+	/** Named values returned by the block function. */
+	outputs: FlowValueDefinitions;
+	/** Serializable configuration fields shown in the block inspector. */
+	config?: FlowValueDefinitions;
+	/**
+	 * Executes the block with validated input and runtime context.
+	 *
+	 * @param input - Values assembled from the flow input or incoming connections.
+	 * @param context - Durable run context, configuration, and logging API.
+	 * @returns Values emitted through the block output ports.
+	 */
+	run?(input: TInput, context: FlowBlockContext<TConfig>): TOutput | Promise<TOutput>;
+}
+
+/**
+ * Designer-safe metadata for one registered executable block type.
+ */
+export type FlowBlockMetadata = Omit<FlowBlockDefinition, 'run'>;

package/src/flows/contracts/FlowCompiler.ts

@@ -0,0 +1,32 @@
+import type { FlowBlockDefinition } from './FlowBlock';
+import type { FlowBlockInstance, FlowConnection, FlowExecutionDefinition } from './FlowDefinition';
+
+/**
+ * Executable block occurrence with its resolved type and graph connections.
+ */
+export interface CompiledFlowBlock {
+	/** Zero-based execution position in the sequential graph. */
+	sequence: number;
+	/** Configured block occurrence from the source definition. */
+	instance: FlowBlockInstance;
+	/** Registered executable block type. */
+	block: FlowBlockDefinition;
+	/** Connections supplying this block's input values. */
+	incoming: FlowConnection[];
+	/** Connections carrying this block's output values forward. */
+	outgoing: FlowConnection[];
+}
+
+/**
+ * Validated sequential execution plan derived from a flow definition snapshot.
+ */
+export interface CompiledFlowDefinition {
+	/** Definition snapshot used to produce this plan. */
+	definition: FlowExecutionDefinition;
+	/** Blocks in deterministic execution order. */
+	blocks: CompiledFlowBlock[];
+	/** Root block receiving the public flow input. */
+	root: CompiledFlowBlock;
+	/** Terminal block producing the public flow output. */
+	terminal: CompiledFlowBlock;
+}

package/src/flows/contracts/FlowDefinition.ts

@@ -0,0 +1,126 @@
+import type { FlowValueDefinitions, FlowValues } from './FlowValue';
+
+/**
+ * Position persisted for a block in compatible graph designers.
+ */
+export interface FlowBlockPosition {
+	/** Horizontal canvas coordinate. */
+	x: number;
+	/** Vertical canvas coordinate. */
+	y: number;
+}
+
+/**
+ * One configured occurrence of a registered block type inside a flow.
+ */
+export interface FlowBlockInstance {
+	/** Stable ULID for this block occurrence. */
+	id: string;
+	/** Registered executable block type. */
+	type: string;
+	/** Optional instance label overriding the block type name in the designer. */
+	name?: string;
+	/** JSON-safe configuration supplied to the block when it runs. */
+	config?: FlowValues;
+	/** Designer position stored with the source-of-truth definition. */
+	position: FlowBlockPosition;
+	/** Optional nested flow id when this instance executes another flow. */
+	flowId?: string;
+}
+
+/**
+ * Serializable input/output contract captured for one resolved block occurrence.
+ *
+ * Generic subflow and flow-boundary blocks derive their ports from a referenced
+ * definition rather than from a statically registered block file. Runs retain
+ * this resolved contract so queued steps and replay do not depend on later edits.
+ */
+export interface FlowResolvedBlockContract {
+	/** Named values accepted by the resolved block occurrence. */
+	inputs: FlowValueDefinitions;
+	/** Named values emitted by the resolved block occurrence. */
+	outputs: FlowValueDefinitions;
+}
+
+/**
+ * One directed value connection between named block ports.
+ */
+export interface FlowConnection {
+	/** Stable ULID for this graph connection. */
+	id: string;
+	/** Source block occurrence ULID. */
+	sourceBlockId: string;
+	/** Named output port on the source block. */
+	sourcePort: string;
+	/** Target block occurrence ULID. */
+	targetBlockId: string;
+	/** Named input port on the target block. */
+	targetPort: string;
+}
+
+/**
+ * Serializable source-of-truth graph executed by the flow runtime.
+ */
+export interface FlowDefinition {
+	/** Definition schema version used for future compatible migrations. */
+	schemaVersion: 1;
+	/** Stable ULID used to invoke and reference this flow. */
+	id: string;
+	/** Human-readable flow name. */
+	name: string;
+	/** Optional longer explanation for developers and AI tooling. */
+	description?: string;
+	/** Public input contract accepted when the flow is invoked. */
+	inputs: FlowValueDefinitions;
+	/** Public output contract returned by the terminal block. */
+	outputs: FlowValueDefinitions;
+	/** Configured block occurrences in the graph. */
+	blocks: FlowBlockInstance[];
+	/** Directed connections between block ports. */
+	connections: FlowConnection[];
+}
+
+/**
+ * Immutable execution snapshot produced from one source flow definition.
+ *
+ * Resolution data is stored only on durable runs. Definition providers continue
+ * to read and write the smaller `FlowDefinition` source shape.
+ */
+export interface FlowExecutionDefinition extends FlowDefinition {
+	/** Effective dynamic contracts keyed by block occurrence ULID. */
+	resolvedBlocks: Record<string, FlowResolvedBlockContract>;
+	/** Immediate child snapshots keyed by flow-backed block occurrence ULID. */
+	nestedDefinitions: Record<string, StoredNestedFlowDefinition>;
+}
+
+/**
+ * Stored child definition captured as part of a parent execution snapshot.
+ */
+export interface StoredNestedFlowDefinition {
+	/** Fully resolved child definition used when the nested block is reached. */
+	definition: FlowExecutionDefinition;
+	/** Definition-provider revision captured with the child source. */
+	revision: string;
+	/** Optional provider-relative source path. */
+	path?: string;
+}
+
+/**
+ * Lightweight definition metadata used by flow lists and selectors.
+ */
+export interface FlowDefinitionSummary {
+	/** Stable flow ULID. */
+	id: string;
+	/** Human-readable flow name. */
+	name: string;
+	/** Optional developer-facing description. */
+	description?: string;
+	/** Public input contract exposed when this flow is used as a subflow block. */
+	inputs: FlowValueDefinitions;
+	/** Public output contract exposed when this flow is used as a subflow block. */
+	outputs: FlowValueDefinitions;
+	/** Current content revision returned by the definition provider. */
+	revision: string;
+	/** Provider-relative source path, when one exists. */
+	path?: string;
+}

package/src/flows/contracts/FlowDefinitionStore.ts

@@ -0,0 +1,52 @@
+import type { FlowDefinition, FlowDefinitionSummary } from './FlowDefinition';
+
+/**
+ * Definition plus provider metadata required for conflict-safe editing.
+ */
+export interface StoredFlowDefinition {
+	/** Parsed and validated source-of-truth definition. */
+	definition: FlowDefinition;
+	/** Content revision used for optimistic concurrency. */
+	revision: string;
+	/** Provider-relative source path, when one exists. */
+	path?: string;
+}
+
+/**
+ * Options used when saving one flow definition.
+ */
+export interface FlowDefinitionWriteOptions {
+	/** Previously-read revision that must still match before replacement. */
+	expectedRevision?: string;
+	/** Optional provider-relative path used when creating a new definition. */
+	path?: string;
+}
+
+/**
+ * Storage boundary shared by local file definitions and future hosted drivers.
+ */
+export interface FlowDefinitionStore {
+	/**
+	 * Lists available definitions without requiring callers to know provider paths.
+	 *
+	 * @returns Available flow summaries.
+	 */
+	list(): Promise<FlowDefinitionSummary[]>;
+
+	/**
+	 * Loads one definition by stable flow ULID.
+	 *
+	 * @param id - Flow ULID.
+	 * @returns Stored definition or null when absent.
+	 */
+	read(id: string): Promise<StoredFlowDefinition | null>;
+
+	/**
+	 * Creates or replaces one definition.
+	 *
+	 * @param definition - Valid flow definition to persist.
+	 * @param options - Revision and optional creation path.
+	 * @returns Newly stored definition metadata.
+	 */
+	write(definition: FlowDefinition, options?: FlowDefinitionWriteOptions): Promise<StoredFlowDefinition>;
+}

package/src/flows/contracts/FlowRun.ts

@@ -0,0 +1,44 @@
+import type { FlowReplayDefinition } from '../constants';
+import type { FlowValue, FlowValues } from './FlowValue';
+
+/**
+ * Structured error snapshot stored on failed runs and block steps.
+ */
+export interface FlowErrorSnapshot {
+	/** Error class or provider name. */
+	name: string;
+	/** Human-readable failure message. */
+	message: string;
+	/** Optional stack trace captured in development/runtime logs. */
+	stack?: string;
+}
+
+/**
+ * Options controlling creation of a new flow run.
+ */
+export interface FlowRunOptions {
+	/** Optional run ULID that this invocation replays. */
+	replayOfRunId?: string;
+}
+
+/**
+ * Options controlling which definition snapshot a replay executes.
+ */
+export interface FlowReplayOptions {
+	/** Whether to execute the original snapshot or the latest stored definition. */
+	definition?: FlowReplayDefinition;
+	/** Optional replacement public input for the replay. */
+	input?: FlowValues;
+}
+
+/**
+ * Durable log entry requested by an executing block.
+ */
+export interface FlowBlockLogInput {
+	/** Log severity. */
+	level: 'debug' | 'info' | 'warning' | 'error';
+	/** Human-readable log message. */
+	message: string;
+	/** Optional structured diagnostic value. */
+	data?: FlowValue;
+}