@pgflow/core 0.0.5-prealpha.2

package/dist/README.md

@@ -0,0 +1,373 @@
+# pgflow SQL Core
+
+PostgreSQL-native workflow engine for defining, managing, and tracking DAG-based workflows directly in your database.
+
+> [!NOTE]
+> This project is licensed under [AGPL v3](./LICENSE.md) license and is part of **pgflow** stack.
+> See [LICENSING_OVERVIEW.md](../../LICENSING_OVERVIEW.md) in root of this monorepo for more details.
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Key Features](#key-features)
+- [Architecture](#architecture)
+  - [Schema Design](#schema-design)
+  - [Execution Model](#execution-model)
+- [Example Flow and its life](#example-flow-and-its-life)
+  - [Defining a Workflow](#defining-a-workflow)
+  - [Starting a Workflow Run](#starting-a-workflow-run)
+  - [Workflow Execution](#workflow-execution)
+    - [Task Polling](#task-polling)
+    - [Task Completion](#task-completion)
+    - [Error Handling](#error-handling)
+    - [Retries and Timeouts](#retries-and-timeouts)
+- [TypeScript Flow DSL](#typescript-flow-dsl)
+  - [Overview](#overview-1)
+  - [Type Inference System](#type-inference-system)
+  - [Basic Example](#basic-example)
+  - [How Payload Types Are Built](#how-payload-types-are-built)
+  - [Benefits of Automatic Type Inference](#benefits-of-automatic-type-inference)
+- [Data Flow](#data-flow)
+  - [Input and Output Handling](#input-and-output-handling)
+  - [Run Completion](#run-completion)
+
+## Overview
+
+The pgflow SQL Core provides the data model, state machine, and transactional functions for workflow management. It treats workflows as Directed Acyclic Graphs (DAGs) of steps, each step being a simple state machine.
+
+This package focuses on:
+
+- Defining and storing workflow shapes
+- Managing workflow state transitions
+- Exposing transactional functions for workflow operations
+- Providing APIs for task polling and status updates
+
+The actual execution of workflow tasks is handled by the [Edge Worker](../edge-worker/README.md), which calls back to the SQL Core to acknowledge task completion or failure.
+
+## Key Features
+
+- **Declarative Workflows**: Define flows and steps via SQL tables
+- **Dependency Management**: Explicit step dependencies with atomic transitions
+- **Configurable Behavior**: Per-flow and per-step options for timeouts, retries, and delays
+- **Queue Integration**: Built on pgmq for reliable task processing
+- **Transactional Guarantees**: All state transitions are ACID-compliant
+
+## Architecture
+
+### Schema Design
+
+[Schema ERD Diagram (click to enlarge)](./schema.svg)
+
+<a href="./schema.svg">
+  <img src="./schema.svg" alt="Schema ERD Diagram" width="25%" height="25%">
+</a>
+
+---
+
+The schema consists of two main categories of tables:
+
+#### Static definition tables
+
+- `flows` (just an identity for the workflow with some global options)
+- `steps` (DAG nodes belonging to particular `flows`, with option overrides)
+- `deps` (DAG edges between `steps`)
+
+#### Runtime state tables
+
+- `runs` (execution instances of `flows`)
+- `step_states` (states of individual `steps` within a `run`)
+- `step_tasks` (units of work for individual `steps` within a `run`, so we can have fanouts)
+
+### Execution Model
+
+The SQL Core handles the workflow lifecycle through these key operations:
+
+1. **Definition**: Workflows are defined using `create_flow` and `add_step`
+2. **Instantiation**: Workflow instances are started with `start_flow`, creating a new run
+3. **Task Management**: The [Edge Worker](../edge-worker/README.md) polls for available tasks using `poll_for_tasks`
+4. **State Transitions**: When the Edge Worker reports back using `complete_task` or `fail_task`, the SQL Core handles state transitions and schedules dependent steps
+
+[Flow lifecycle diagram (click to enlarge)](./flow-lifecycle.svg)
+
+<a href="./flow-lifecycle.svg"><img src="./flow-lifecycle.svg" alt="Flow Lifecycle" width="25%" height="25%"></a>
+
+## Example flow and its life
+
+Let's walk through creating and running a workflow that fetches a website, 
+does summarization and sentiment analysis in parallel steps
+and saves the results to a database.
+
+![example flow graph](./example-flow.svg)
+
+### Defining a Workflow
+
+Workflows are defined using two SQL functions: `create_flow` and `add_step`.
+
+In this example, we'll create a workflow with:
+- `website` as the entry point ("root step")
+- `sentiment` and `summary` as parallel steps that depend on `website`
+- `saveToDb` as the final step, depending on both parallel steps
+
+```sql
+-- Define workflow with parallel steps
+SELECT pgflow.create_flow('analyze_website');
+SELECT pgflow.add_step('analyze_website', 'website');
+SELECT pgflow.add_step('analyze_website', 'sentiment', deps_slugs => ARRAY['website']);
+SELECT pgflow.add_step('analyze_website', 'summary', deps_slugs => ARRAY['website']);
+SELECT pgflow.add_step('analyze_website', 'saveToDb', deps_slugs => ARRAY['sentiment', 'summary']);
+```
+
+> [!WARNING]
+> You need to call `add_step` in topological order, which is enforced by foreign key constraints.
+
+> [!NOTE]
+> You can have multiple "root steps" in a workflow. You can even create a root-steps-only workflow
+> to process a single input in parallel, because at the end, all of the outputs from steps 
+> that does not have dependents ("final steps") are aggregated and saved as run's `output`.
+
+### Starting a Workflow Run
+
+To start a workflow, call `start_flow` with a flow slug and input arguments:
+
+```sql
+SELECT * FROM pgflow.start_flow(
+  flow_slug => 'analyze_website', 
+  input => '{"url": "https://example.com"}'::jsonb
+);
+
+--     run_id  | flow_slug       | status  |  input                         | output | remaining_steps 
+-- ------------+-----------------+---------+--------------------------------+--------+-----------------
+--  <run uuid> | analyze_website | started | {"url": "https://example.com"} | [NULL] |               4
+```
+
+When a workflow starts:
+- A new `run` record is created
+- Initial states for all steps are created
+- Root steps are marked as `started`
+- Tasks are created for root steps
+- Messages are enqueued on PGMQ for worker processing
+
+> [!NOTE]
+> The `input` argument must be a valid JSONB object: string, number, boolean, array, object or null.
+
+### Workflow Execution
+
+#### Task Polling
+
+The Edge Worker continuously polls for available tasks using the `poll_for_tasks` function:
+
+```sql
+SELECT * FROM pgflow.poll_for_tasks(
+  queue_name => 'analyze_website',
+  vt => 60, -- visibility timeout in seconds
+  qty => 5  -- maximum number of tasks to fetch
+);
+```
+
+When a task is polled:
+
+1. The message is hidden from other workers for the specified timeout period
+2. The task's attempts counter is incremented for retry tracking
+3. An input object is built by combining the run input with outputs from completed dependency steps
+4. Task metadata and input are returned to the worker
+
+This process happens in a single transaction to ensure reliability. The worker then executes the appropriate handler function based on the task metadata.
+
+#### Task Completion
+
+After successful processing, the worker acknowledges completion:
+
+```sql
+SELECT pgflow.complete_task(
+  run_id => '<run_uuid>',
+  step_slug => 'website',
+  task_index => 0, -- we will have multiple tasks for a step in the future
+  output => '{"content": "HTML content", "status": 200}'::jsonb
+);
+```
+
+When a task completes:
+1. The task status is updated to 'completed' and the output is saved
+2. The message is archived in PGMQ
+3. The step state is updated to 'completed'
+4. Dependent steps with all dependencies completed are automatically started
+5. The run's remaining_steps counter is decremented
+6. If all steps are completed, the run is marked as completed with aggregated outputs
+
+#### Error Handling
+
+If a task fails, the worker acknowledges this using `fail_task`:
+
+```sql
+SELECT pgflow.fail_task(
+  run_id => '<run_uuid>',
+  step_slug => 'website',
+  task_index => 0,
+  error_message => 'Connection timeout when fetching URL'::text
+);
+```
+
+The system handles failures by:
+
+1. Checking if retry attempts are available
+2. For available retries:
+   - Keeping the task in 'queued' status
+   - Applying exponential backoff for visibility
+   - Preventing processing until the visibility timeout expires
+3. When retries are exhausted:
+   - Marking the task as 'failed'
+   - Marking the step as 'failed'
+   - Marking the run as 'failed'
+   - Archiving the message in PGMQ
+   - Notifying workers to abort pending tasks (future feature)
+
+#### Retries and Timeouts
+
+Retry behavior can be configured at both the flow and step level:
+
+```sql
+-- Flow-level defaults
+SELECT pgflow.create_flow(
+  flow_slug => 'analyze_website',
+  max_attempts => 3,    -- Maximum retry attempts (including first attempt)
+  base_delay => 5,      -- Base delay in seconds for exponential backoff
+  timeout => 60         -- Task timeout in seconds
+);
+
+-- Step-level overrides
+SELECT pgflow.add_step(
+  flow_slug => 'analyze_website',
+  step_slug => 'sentiment',
+  deps_slugs => ARRAY['website']::text[],
+  max_attempts => 5,    -- Override max attempts for this step
+  base_delay => 2,      -- Override base delay for exponential backoff
+  timeout => 30         -- Override timeout for this step
+);
+```
+
+The system applies exponential backoff for retries using the formula:
+```
+delay = base_delay * (2 ^ attempts_count)
+```
+
+Timeouts are enforced by setting the message visibility timeout to the step's timeout value plus a small buffer. If a worker doesn't acknowledge completion or failure within this period, the task becomes visible again and can be retried.
+
+## TypeScript Flow DSL
+
+> [!NOTE]
+> TypeScript Flow DSL is a Work In Progress and is not ready yet!
+
+### Overview
+
+While the SQL Core engine handles workflow definitions and state management, the primary way to define and work with your workflow logic is via the Flow DSL in TypeScript. This DSL offers a fluent API that makes it straightforward to outline the steps in your flow with full type safety.
+
+### Type Inference System
+
+The most powerful feature of the Flow DSL is its **automatic type inference system**:
+
+1. You only need to annotate the initial Flow input type
+2. The return type of each step is automatically inferred from your handler function
+3. These return types become available in the payload of dependent steps
+4. The TypeScript compiler builds a complete type graph matching your workflow DAG
+
+This means you get full IDE autocompletion and type checking throughout your workflow without manual type annotations.
+
+### Basic Example
+
+Here's an example that matches our website analysis workflow:
+
+```ts
+// Provide a type for the input of the Flow
+type Input = {
+  url: string;
+};
+
+const AnalyzeWebsite = new Flow<Input>({
+  slug: "analyze_website",
+  maxAttempts: 3,
+  baseDelay: 5,
+  timeout: 10,
+})
+  .step({ slug: "website" }, async (input) => await scrapeWebsite(input.run.url))
+  .step(
+    { slug: "sentiment", dependsOn: ["website"], timeout: 30, maxAttempts: 5 },
+    async (input) => await analyzeSentiment(input.website.content)
+  )
+  .step(
+    { slug: "summary", dependsOn: ["website"] },
+    async (input) => await summarizeWithAI(input.website.content)
+  )
+  .step(
+    { slug: "saveToDb", dependsOn: ["sentiment", "summary"] },
+    async (input) =>
+      await saveToDb({
+        websiteUrl: input.run.url,
+        sentiment: input.sentiment.score,
+        summary: input.summary,
+      }).status
+  );
+```
+
+### How Payload Types Are Built
+
+The payload object for each step is constructed dynamically based on:
+
+1. **The `run` property**: Always contains the original workflow input
+2. **Dependency outputs**: Each dependency's output is available under a key matching the dependency's ID
+3. **DAG structure**: Only outputs from direct dependencies are included in the payload
+
+This means your step handlers receive exactly the data they need, properly typed, without any manual type declarations beyond the initial Flow input type.
+
+### Benefits of Automatic Type Inference
+
+- **Refactoring safety**: Change a step's output, and TypeScript will flag all dependent steps that need updates
+- **Discoverability**: IDE autocompletion shows exactly what data is available in each step
+- **Error prevention**: Catch typos and type mismatches at compile time, not runtime
+- **Documentation**: The types themselves serve as living documentation of your workflow's data flow
+
+## Data Flow
+
+### Input and Output Handling
+
+Handlers in pgflow **must return** JSON-serializable values that are captured and saved when `complete_task` is called. These outputs become available as inputs to dependent steps, allowing data to flow through your workflow pipeline.
+
+When a step is executed, it receives an input object where:
+- Each key is a step_slug of a completed dependency
+- Each value is that step's output
+- A special "run" key contains the original workflow input
+
+#### Example: `sentiment`
+
+When the `sentiment` step runs, it receives:
+
+```json
+{
+  "run": {"url": "https://example.com"},
+  "website": {"content": "HTML content", "status": 200}
+}
+```
+
+#### Example: `saveToDb`
+
+The `saveToDb` step depends on both `sentiment` and `summary`:
+
+```json
+{
+  "run": {"url": "https://example.com"},
+  "sentiment": {"score": 0.85, "label": "positive"},
+  "summary": "This website discusses various topics related to technology and innovation."
+}
+```
+
+### Run Completion
+
+When all steps in a run are completed, the run status is automatically updated to 'completed' and its output is set. The output is an aggregation of all the outputs from final steps (steps that have no dependents):
+
+```sql
+-- Example of a completed run with output
+SELECT run_id, status, output FROM pgflow.runs WHERE run_id = '<run_uuid>';
+
+--     run_id  | status    | output
+-- ------------+-----------+-----------------------------------------------------
+--  <run uuid> | completed | {"saveToDb": {"status": "success"}}
+```

package/dist/index.js

@@ -0,0 +1,54 @@
+// pkgs/core/src/PgflowSqlClient.ts
+var PgflowSqlClient = class {
+  constructor(sql) {
+    this.sql = sql;
+  }
+  async pollForTasks(queueName, batchSize = 20, visibilityTimeout = 2, maxPollSeconds = 5, pollIntervalMs = 200) {
+    return await this.sql`
+      SELECT *
+      FROM pgflow.poll_for_tasks(
+        queue_name => ${queueName},
+        vt => ${visibilityTimeout},
+        qty => ${batchSize},
+        max_poll_seconds => ${maxPollSeconds},
+        poll_interval_ms => ${pollIntervalMs}
+      );
+    `;
+  }
+  async completeTask(stepTask, output) {
+    await this.sql`
+      SELECT pgflow.complete_task(
+        run_id => ${stepTask.run_id}::uuid,
+        step_slug => ${stepTask.step_slug}::text,
+        task_index => ${0}::int,
+        output => ${this.sql.json(output || null)}::jsonb
+      );
+    `;
+  }
+  async failTask(stepTask, error) {
+    const errorString = typeof error === "string" ? error : error instanceof Error ? error.message : JSON.stringify(error);
+    await this.sql`
+      SELECT pgflow.fail_task(
+        run_id => ${stepTask.run_id}::uuid,
+        step_slug => ${stepTask.step_slug}::text,
+        task_index => ${0}::int,
+        error_message => ${errorString}::text
+      );
+    `;
+  }
+  async startFlow(flow, input) {
+    const results = await this.sql`
+      SELECT * FROM pgflow.start_flow(${flow.slug}::text, ${this.sql.json(
+      input
+    )}::jsonb);
+    `;
+    if (results.length === 0) {
+      throw new Error(`Failed to start flow ${flow.slug}`);
+    }
+    const [flowRun] = results;
+    return flowRun;
+  }
+};
+export {
+  PgflowSqlClient
+};

package/docs/options_for_flow_and_steps.md

@@ -0,0 +1,75 @@
+:::
+
+1. MVP will allow only 1:1 queue:flow mappings, because it simplifies a lot.
+   Queue is created when creating a flow, in pgflow.create_flow() function.
+
+2. No, we will start with either static delay or simple exponential and
+   I do not plan to expand on it further.
+
+   The backoff will be calculated by the retry attempts, that can be deducted
+   from pgmq's "read_ct" counter.
+
+   No need for jitter for now.
+
+3. **Execution Timeouts**
+   I'm not sure about flow timeouts, I think for MVP we should probably
+   skip them and I'm not even sure if I want to have them for the steps too.
+
+   I would be definitely adding them in future.
+
+4. We want everything to by statically typed in TypeScript, because the main
+   way to define flows would be to use TS DSL.
+
+   So a condition must be a JSON-serializable object that will get saved
+   in a JSONB column.
+
+   My initial idea for conditions was to just provide a JSON object that we
+   will use to perform containment check on the step inputs using @> operator.
+
+   I am considering expanding it to more robust condition, but they would need
+   to be defined in JSON-serializable way.
+
+   I do not want to have any SQL snippets in conditions, because I cannot
+   statically type them and they can fail at runtime.
+
+   No, conditions should probably only be able to reference step inputs.
+
+5. Yes it should be able to disable retries per step or override their params.
+   How those circuit breakers should work and what would be the benefit of them?
+   What is retry budget? Do we have a distributed system really? Everything
+   lives in a single postgres instance.
+
+6. No we do not want to support dependency resolution at runtime,
+   but conditions could be used to implement something similar.
+   Steps should not be able to add new steps during runtime,
+   but I plan to have fanout steps that will spawn either multiple tasks
+   or multiple subflows, one per the input array item.
+   Those are meant to be aggregated back to the output array when completed.
+
+   It's not for MVP tho!!
+
+### Cross-Cutting Concerns
+7. We will advise users to not put anything sensitive into flow options,
+   by writing docs and also not having handlers any ability to access the
+   step options.
+
+   We will provide a Context object that users can define in the Flow DSL,
+   that will be passed to the step handlers at runtime and will encourage
+   users to use this Context object to store sensitive data.
+
+8. What you mean by tracked metadata for options and options affecting metric?
+
+9. Versioning is a big problem for my project - I have few ideas how to solve
+   it (basically topologically sort graph and hash it to create a version hash).
+   But for MVP we definitely don't need versioning - users must take care of
+   this on their own by just creating new flows if they change the shape of
+   the flow. There is no way and will be no way to UPDATE flows.
+
+   Graph shape should be immutable after creation, but i'm not sure about
+   the options - maybe it would be a good idea to allow updating retry configs,
+   because only running the flows in production can allow users to gather
+   enough data to make educated decisions about those params.
+
+10. No modification for flows/steps at all, so easy.
+    
+:::

package/docs/pgflow-blob-reference-system.md

@@ -0,0 +1,179 @@
+# PgFlow Blob Reference System
+
+## Overview
+
+PgFlow needs an efficient way to handle large data outputs from workflow steps. The Blob Reference System provides a solution by separating large data payloads from workflow control information while maintaining a seamless developer experience.
+
+## How It Works
+
+### Core Concept
+
+When steps produce large outputs (e.g., HTML content from web scraping, binary data, large API responses), these outputs are stored separately in a dedicated blob storage table. The workflow state maintains references to these blobs rather than storing the actual large data.
+
+### Database Structure
+
+The system uses a dedicated table for blob storage:
+
+```sql
+CREATE TABLE pgflow.output_blobs (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  content JSONB NOT NULL,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+```
+
+### Worker Task Structure
+
+The `poll_for_tasks` function returns tasks with both regular inputs and blob references through a custom type:
+
+```sql
+CREATE TYPE pgflow.step_task_record AS (
+  flow_slug TEXT,
+  run_id UUID,
+  step_slug TEXT,
+  input JSONB,
+  blobs_refs JSONB
+);
+```
+
+This design provides a clean separation between:
+
+- `input`: Regular small data that can be directly included in the task
+- `blobs_refs`: References to large data stored separately in the blob table
+
+### Example Return Value
+
+A task returned by `poll_for_tasks` might look like:
+
+```json
+{
+  "flow_slug": "my_flow",
+  "run_id": "1234-5678-90ab-cdef",
+  "step_slug": "my_step",
+  "input": {
+    "run": "run input",
+    "dependency_a": "dependency_a output"
+  },
+  "blobs_refs": {
+    "dependency_b": "<uuid to the blob saved for dependency_b which returned binary data>"
+  }
+}
+```
+
+In this example:
+
+- `dependency_a` had a small output that's included directly in the `input` object
+- `dependency_b` had a large output (possibly binary data) that's stored as a blob, with only a reference included
+
+### Queue Efficiency
+
+A critical optimization in PgFlow is that the task queue only stores minimal task identification information:
+
+- flow_slug
+- run_id
+- step_slug
+- task_index
+
+This lightweight approach keeps queue messages small and efficient. When a worker picks up a task, it uses these identifiers to:
+
+1. Call `poll_for_tasks` to get the full task data
+2. Receive both the regular `input` and `blobs_refs` in a single query result
+3. Fetch the actual blob content for any referenced blobs
+4. Combine all data to form the complete input for the task handler
+
+## Implementation Flow
+
+### Task Creation
+
+1. When a step completes, its output is analyzed:
+
+   - Outputs below the size threshold remain in the regular output JSONB
+   - Large outputs are stored in the `pgflow.output_blobs` table with a unique ID
+
+2. The `start_ready_steps` function:
+   - Creates task entries with references to any large blob data
+   - Enqueues only the task identifiers (not the actual data) in the task queue
+
+### Task Execution
+
+1. Worker picks up the task identifier from the queue
+2. Worker calls `poll_for_tasks` to get the task details
+3. `poll_for_tasks` returns:
+   - The `input` object with regular data
+   - The `blobs_refs` object with references to any large data outputs
+4. Worker fetches blob content for any references in `blobs_refs`
+5. Worker assembles the complete input (combining regular data and blob data) for the task handler
+6. Task handler executes with the complete data, unaware of the blob reference system
+
+### Example Processing Flow
+
+For a web scraping workflow:
+
+1. `fetch-html` step returns a large HTML string (3MB)
+2. System detects the large output and:
+   - Stores HTML in `pgflow.output_blobs` with ID "abc-123"
+   - Records only the blob reference in the step's output
+3. When `parse-html` step is ready to run:
+   - Queue contains only the task identifier
+   - `poll_for_tasks` returns the task with:
+     ```json
+     {
+       "input": {
+         "run": { "url": "https://example.com" }
+       },
+       "blobs_refs": {
+         "fetch-html": "abc-123"
+       }
+     }
+     ```
+4. Worker:
+   - Detects the blob reference "abc-123" for "fetch-html"
+   - Fetches the actual HTML content from the blob table
+   - Provides the handler with complete input including the HTML content
+
+## Developer Experience
+
+From a workflow developer's perspective, the blob reference system is completely transparent:
+
+```typescript
+// Developer writes code as if all data is directly available
+const parseHtmlHandler: StepHandler<ParseInput, ParseOutput> = async (
+  input
+) => {
+  // input.dependencies["fetch-html"] contains the full HTML content
+  // (the blob reference was automatically resolved)
+  const html = input.dependencies['fetch-html'];
+
+  // Process the HTML...
+  const title = extractTitle(html);
+  const links = extractLinks(html);
+
+  return { title, links };
+};
+```
+
+The developer never needs to:
+
+- Manually resolve blob references
+- Check if data is a reference or actual content
+- Handle storage of large outputs differently
+
+## Benefits and Considerations
+
+### Benefits
+
+1. **Database Efficiency**: Large data is stored separately from workflow metadata
+2. **Queue Performance**: Queue messages remain small and consistent in size
+3. **Separation of Concerns**: Control flow data is separate from large payloads
+4. **Transparent to Developers**: No special code required to handle large data
+5. **Scalability**: Can handle arbitrary data sizes without affecting workflow system performance
+
+### Considerations
+
+1. **Query Optimization**: Ensure `poll_for_tasks` efficiently retrieves both regular data and blob references
+2. **Blob Lifecycle Management**: Implement cleanup for orphaned or expired blobs
+3. **Size Threshold Tuning**: Configure appropriate thresholds for when data should use blob storage
+
+## Conclusion
+
+The Blob Reference System in PgFlow provides an elegant solution for handling large data in workflows. By splitting task data into regular inputs and blob references, the system maintains efficient database usage and queue performance while providing a seamless experience for workflow developers. The design ensures that large data is handled appropriately without requiring developers to write special code for blob resolution or storage.