@pgflow/core 0.0.5-prealpha.2

package/prompts/one_shot.md

@@ -0,0 +1,286 @@
+Your job is to implement required SQL schemas and functions for an MVP of my open source Postgres-native workflow orchestration engine called pgflow.
+
+The main idea of the project is to keep shape of the DAG (nodes and edges) and its runtime state in the database
+and expose SQL functions that will allow to propagate through the state.
+
+Real work is done on the task queue workers and the functions from pgflow are only orchestrating
+the queue messages.
+
+Workers are supposed to call user functions with the input from the queue message,
+and should acknowledge the completion of the task or its failure (error thrown) by
+calling appropriate pgflow SQL functions.
+
+This way the orchestration is decoupled from the execution.
+
+I have a concrete implementation plan for you to follow and will unfold it
+step by step below.
+
+## Assumptions/best practices
+
+### We are building Minimal Viable Product
+
+Remember that we are building MVP and main focus should be on shipping something as soon as possible,
+by cutting scope, simplifying the architectures and code.
+
+But the outlined features are definitely something that we will be doing in the future.
+I am most certain about the foreach-array steps - this is a MUST have.
+So your focus should be on trying to implement the MVP but not closing the doors to the future improvements.
+
+### Slugs
+
+We do not use serial IDs nor UUIDs for static things, we use "slugs" instead.
+A slug is just a string that conforms to following rules:
+
+```sql
+slug is not null
+and slug <> ''
+and length(slug) <= 128
+and slug ~ '^[a-zA-Z_][a-zA-Z0-9_]*$';
+```
+
+We use UUID for identifying particular run of the flow.
+But the states of steps for that particular run are not identified by separate UUIDs,
+but rather by a pair of run_id and step_slug. This pattern allows to easily refer
+to steps and flows by their slugs. **Leverage this pattern everywhere you can!**
+
+### References/fkeys
+
+Use foreign keys everywhere to ensure consistency.
+Use composite foreign keys and composite primary keys composed of flow/step slugs and run_id's if needed.
+
+### Declarative vs procedural
+
+**YOU MUST ALWAYS PRIORITIZE DECLARATIVE STYLE** and prioritize Batching operations.
+
+Avoid plpgsql as much as you can.
+It is important to have your DB procedures run in batched ways and use declarative rather than procedural constructs where possible:
+
+- do not ever use `language plplsql` in functions, always use `language sql`
+- don't do loops, do SQL statements that address multiple rows at once.
+- don't write trigger functions that fire for a single row, use `FOR EACH STATEMENT` instead.
+- don't call functions for each row in a result set, a condition, a join, or whatever; instead use functions that return `SETOF` and join against these.
+
+If you're constructing dynamic SQL, you should only ever use `%I` and `%L` when using `FORMAT` or similar; you should never see `%s` (with the very rare exception of where you're merging in another SQL fragment that you've previously formatted using %I and %L).
+
+Remember, that functions have significant overhead in Postgres - instead of factoring into lots of tiny functions, think about how to make your code more expressive so there's no need.
+
+## Schemas
+
+### pgflow.flows
+
+A static definition of a flow (DAG):
+
+```sql
+CREATE TABLE pgflow.flows (
+    flow_slug text PRIMARY KEY NOT NULL  -- Unique identifier for the flow
+    CHECK (is_valid_slug(flow_slug))
+);
+```
+
+### pgflow.steps
+
+A static definition of a step within a flow (a DAG "nodes"):
+
+```sql
+CREATE TABLE pgflow.steps (
+    flow_slug text NOT NULL REFERENCES flows (flow_slug),
+    step_slug text NOT NULL,
+    step_type text NOT NULL DEFAULT 'single',
+    PRIMARY KEY (flow_slug, step_slug),
+    CHECK (is_valid_slug(flow_slug)),
+    CHECK (is_valid_slug(step_slug))
+);
+```
+
+### pgflow.deps
+
+A static definition of dependencies between steps (a DAG "edges"):
+
+```sql
+CREATE TABLE pgflow.deps (
+    flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug),
+    dep_slug text NOT NULL,  -- The step that must complete first
+    step_slug text NOT NULL,   -- The step that depends on dep_slug
+    PRIMARY KEY (flow_slug, dep_slug, step_slug),
+    FOREIGN KEY (flow_slug, dep_slug)
+    REFERENCES pgflow.steps (flow_slug, step_slug),
+    FOREIGN KEY (flow_slug, step_slug)
+    REFERENCES pgflow.steps (flow_slug, step_slug),
+    CHECK (dep_slug != step_slug)  -- Prevent self-dependencies
+    CHECK (is_valid_slug(step_slug))
+);
+```
+
+### pgflow.runs
+
+A table storing runtime state of given flow.
+A run is identified by a `flow_slug` and `run_id`.
+
+```sql
+CREATE TABLE pgflow.runs (
+    run_id uuid PRIMARY KEY NOT NULL DEFAULT gen_random_uuid(),
+    flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug), -- denormalized
+    status text NOT NULL DEFAULT 'started',
+    input jsonb NOT NULL,
+    CHECK (status IN ('started', 'failed', 'completed'))
+)
+```
+
+There is also `status` that currently can be started, failed or completed.
+);
+
+````
+
+There is also `status` that currently can be pending, failed or completed.
+
+### pgflow.step_states
+
+Represents a state of a particular step in a particular run.
+
+```sql
+
+-- Step states table - tracks the state of individual steps within a run
+CREATE TABLE pgflow.step_states (
+    flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug),
+    run_id uuid NOT NULL REFERENCES pgflow.runs (run_id),
+    step_slug text NOT NULL,
+    status text NOT NULL DEFAULT 'created',
+    PRIMARY KEY (run_id, step_slug),
+    FOREIGN KEY (flow_slug, step_slug)
+    REFERENCES pgflow.steps (flow_slug, step_slug),
+    CHECK (status IN ('created', 'started', 'completed', 'failed'))
+);
+);
+````
+
+### pgflow.step_tasks
+
+This table is really unique and interesting. We are starting the development
+of the flow orchestration engine with a simple step that runs one unit of work.
+
+But I imagine we would suppport additional types of steps, like:
+
+- a step that requires input array and enqueues a task per array item, so they are created in parallel
+- a step that runs some preprocessing/postprocessing in an additional task
+
+So in order to accomodate this, we need an additional layer between step_state and
+an actual task queue, in order to track which messages belong to which steps,
+in case there are more than 1 unit of work for given step.
+
+```sql
+-- Executio logs table - tracks the task of individual steps
+CREATE TABLE pgflow.step_tasks (
+flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug),
+step_slug text NOT NULL,
+run_id uuid NOT NULL REFERENCES pgflow.runs (run_id),
+status text NOT NULL DEFAULT 'queued',
+input jsonb NOT NULL, -- payload that will be passed to queue message
+output jsonb, -- like step_result but for task, can store result or error/stacktrace
+message_id bigint, -- an id of the queue message
+CONSTRAINT step_tasks_pkey PRIMARY KEY (run_id, step_slug),
+FOREIGN KEY (run_id, step_slug)
+REFERENCES pgflow.step_states (run_id, step_slug),
+CHECK (status IN ('queued', 'started', 'failed', 'completed')),
+CHECK (is_valid_slug(flow_slug)),
+CHECK (is_valid_slug(step_slug))
+);
+```
+
+## Typescript DSL, topological ordering and acyclicity validation
+
+The simple typescript DSL will be created that will have string typing
+and will enforce adding steps in a topological order, preventing
+cycles by the strict ordering of the steps addition.
+
+Typescript DSL looks like this:
+
+```ts
+const BasicFlow = new Flow<string>()
+  .step('root', ({ run }) => {
+    return `[${run}]r00t`;
+  })
+  .step('left', ['root'], ({ root: r }) => {
+    return `${r}/left`;
+  })
+  .step('right', ['root'], ({ root: r }) => {
+    return `${r}/right`;
+  })
+  .step('end', ['left', 'right'], ({ left, right, run }) => {
+    return `<${left}> and <${right}> of (${run})`;
+  });
+```
+
+This will be compiled to a simple SQL calling SQL function `pgflow.add_step(flow_slug, step_slug, dep_step_slugs[])`:
+
+```sql
+SELECT pgflow.add_step('basic', 'root', ARRAY[]::text[]);
+SELECT pgflow.add_step('basic', 'left', ARRAY['root']);
+SELECT pgflow.add_step('basic', 'right', ARRAY['root']);
+SELECT pgflow.add_step('basic', 'end', ARRAY['left', 'right']);
+```
+
+## SQL functions API
+
+This describes public SQL functions that are available to developer using pgflow
+and to the workers.
+
+Developer calls `start_flow` and rest is called by the workers.
+
+### pgflow.start_flow(flow_slug::text, input::jsonb)
+
+This function is used to start a flow.
+It should work like this:
+
+- create a new `pgflow.runs` row for given flow_slug
+- create all the `pgflow.step_states` rows corresponding to the steps in the flow
+- find root steps (ones without dependencies) and call "start_step" on each of them
+
+### pgflow.start_step(run_id::uuid, step_slug::text)
+
+This function is called by start_flow but also by complete_step_task (or somewhere near its call)
+when worker acknowledges the step_task completion and it is detected, that there are ready dependant
+steps to be started.
+
+It should probably call start_step_task under the hood, which will:
+
+- updating step_state status/timestamps
+- creating a step_task row
+- enqueueing a queue message for this step_task
+
+For other step types, like array/foreach, it would probably call the step_task
+for each array item, so more than one step task is created and more than one message is enqueued.
+
+### pgflow.start_step_task(run_id::uuid, step_slug::text, task_id::bigint)
+
+I am not yet sure how this will work for other step types that will need more step tasks.
+But probably each step type would have its own implementation of this function,
+and a simple step type will just create a new step_task row and enqueue it.
+
+But an array/foreach step type would need a different implementation.
+Would need to check the input for the step which is an array, and would
+create a new step_task for each array item and enqueue as many messages as there are items in the array.
+
+### pgflow.complete_step_task(run_id::uuid, step_slug::text, output::jsonb)
+
+This will be called by the worker when a step_task is completed.
+It will work like this in the simplified version when one step_state corresponds to one step_task:
+
+- it marks step_task as completed, saving the output
+- it in turns mark step_state as completed, saving the output
+- then it should check for any dependant steps (steps that depend on just completed step) in the same run
+- it should then check if any of those dependant steps are "ready" - meaning, all their dependencies are completed
+- for each of those
+
+I am not yet sure how this will work for other step types that will need more step tasks.
+Probably each step type would have its own implementation of this function,
+so a simple step will just call complete_step_state when complete_step_task is called.
+
+An array/foreach step type would need a different implementation.
+Would probably need to check if other step_tasks are still pending.
+If all are already completed, it would just call complete_step_state,
+otherwise it will just continue, so other (last) step task can complete the step state.
+
+### pgflow.fail_step_task(run_id::uuid, step_slug::text, error::jsonb)
+
+This is very similar to complete_step_task, but it will mark step_task as failed,
+will save error message and will call fail_step_state instead of complete_step_state.

package/prompts/pgtap.md

@@ -0,0 +1,229 @@
+# PGTap Testing Guidelines
+
+## Overview
+
+This document outlines a set of rules, best practices, ideas, and guidelines for writing pgTap tests for the project.
+
+## File Organization
+
+- Store test files under the `supabase/tests/` directory.
+- Use descriptive file names with the `.test.sql` suffix.
+- Organize tests in subfolders, by functionality (e.g., `start_flow`, `create_flow`, `add_step`, `poll_for_tasks`, `complete_task`, etc).
+
+## Transactional Test Structure
+
+Wrap each test in a transaction to ensure isolation:
+
+```sql
+begin;
+select plan(2);
+-- Test queries here
+select finish();
+rollback;
+```
+
+## Setup and Teardown
+
+Reset and prepare the database context at the start of each test:
+
+```sql
+select pgflow_tests.reset_db();
+select pgflow_tests.setup_flow('sequential');
+```
+
+Terminate tests with:
+
+```sql
+select finish();
+rollback;
+```
+
+## Declaring the Test Plan
+
+Declare the number of tests using the `plan()` function:
+
+```sql
+select plan(2);
+```
+
+## Using pgTap Assertions
+
+Use the following assertion functions to verify expected outcomes:
+
+- `is(actual, expected, message)`
+- `results_eq(actual, expected, message)`
+- `set_eq(actual_query, expected_array, message)`
+- `throws_ok(query, expected_error_message, message)`
+- `ok(boolean_expression, message)`
+
+### Example: Validating Run Creation
+
+```sql
+select pgflow.start_flow('sequential', '"hello"'::jsonb);
+
+select results_eq(
+  $$ SELECT flow_slug, status, input FROM pgflow.runs $$,
+  $$ VALUES ('sequential', 'started', '"hello"'::jsonb) $$,
+  'Run should be created with appropriate status and input'
+);
+
+select is(
+  (select remaining_steps::int from pgflow.runs limit 1),
+  3::int,
+  'remaining_steps should be equal to number of steps'
+);
+```
+
+### Example: Testing Error Handling
+
+```sql
+select throws_ok(
+  $$ SELECT pgflow.create_flow('invalid-flow') $$,
+  'new row for relation "flows" violates check constraint "flows_flow_slug_check"',
+  'Should detect and prevent invalid flow slug'
+);
+```
+
+## Idempotence and Duplicate Prevention
+
+Run operations multiple times to ensure idempotency and that no duplicates are created:
+
+```sql
+select pgflow.create_flow('test_flow');
+select pgflow.create_flow('test_flow');
+
+select results_eq(
+  $$ SELECT flow_slug FROM pgflow.flows $$,
+  array['test_flow']::text [],
+  'No duplicate flow should be created'
+);
+```
+
+## Testing Dependencies and Flow Isolation
+
+Ensure that steps and dependencies remain isolated within a flow:
+
+```sql
+select pgflow.create_flow('test_flow');
+select pgflow.add_step('test_flow', 'first_step');
+
+select pgflow.create_flow('another_flow');
+select pgflow.add_step('another_flow', 'first_step');
+select pgflow.add_step('another_flow', 'another_step', array['first_step']);
+
+select set_eq(
+  $$
+      SELECT flow_slug, step_slug
+      FROM pgflow.steps WHERE flow_slug = 'another_flow'
+  $$,
+  $$ VALUES
+       ('another_flow', 'another_step'),
+       ('another_flow', 'first_step')
+  $$,
+  'Steps in second flow should be isolated from first flow'
+);
+```
+
+## Testing Message Queues
+
+Simulate message polling and verify visibility timeouts:
+
+```sql
+select is(
+  (select count(*)::integer from pgflow.poll_for_tasks(
+    queue_name => 'sequential'::text,
+    vt => 5,
+    qty => 1,
+    max_poll_seconds => 1
+  )),
+  1::integer,
+  'First poll should get the available task'
+);
+
+select is(
+  (select count(*)::integer from pgflow.poll_for_tasks(
+    queue_name => 'sequential'::text,
+    vt => 5,
+    qty => 1,
+    max_poll_seconds => 1
+  )),
+  0::integer,
+  'Concurrent poll should not get the same task (due to visibility timeout)'
+);
+```
+
+## Completing Tasks and Flow Progression
+
+Ensure that task completions update state and trigger dependents:
+
+```sql
+select pgflow.complete_task(
+  (select run_id from pgflow.runs limit 1),
+  'first',
+  0,
+  '{"result": "first completed"}'::jsonb
+);
+
+select results_eq(
+  $$ SELECT status, output FROM pgflow.step_tasks
+     WHERE run_id = (SELECT run_id FROM pgflow.runs LIMIT 1)
+       AND step_slug = 'first' $$,
+  $$ VALUES ('completed', '{"result": "first completed"}'::jsonb) $$,
+  'Task should be marked as completed with correct output'
+);
+```
+
+## Archiving Processed Messages
+
+Verify that messages are archived after task completion:
+
+```sql
+select is(
+  (select message ->> 'step_slug' from pgmq.q_sequential limit 1),
+  'first',
+  'First message should be in the queue'
+);
+
+select pgflow.complete_task(
+  (select run_id from pgflow.runs limit 1),
+  'first',
+  0,
+  '"first was successful"'::jsonb
+);
+
+select is(
+  (select count(*)::INT from pgmq.q_sequential where message ->> 'step_slug' = 'first'),
+  0::INT,
+  'There should be no messages in the queue'
+);
+
+select is(
+  (select count(*)::INT from pgmq.a_sequential where message ->> 'step_slug' = 'first' limit 1),
+  1::INT,
+  'The message should be archived'
+);
+```
+
+## Validating Input with Custom Validators
+
+Use custom functions to check input formats:
+
+```sql
+select ok(
+  pgflow.is_valid_slug('valid_slug'),
+  'is_valid_slug returns true for string with underscore'
+);
+```
+
+## Conclusion
+
+Adhere to the following best practices when writing pgTap tests:
+
+- Keep tests self-contained with proper setup and teardown.
+- Use transactions to isolate tests.
+- Declare a clear test plan using `plan()`.
+- Write focused tests with descriptive messages.
+- Ensure idempotence by re-running operations.
+- Validate both positive outcomes and error cases.
+
+Following these guidelines will help maintain consistency, reliability, and clarity in your pgTap tests.

package/prompts/sdk.md

@@ -0,0 +1,59 @@
+# Flow SDK
+
+The purpose of Flow SDK is to allow users to start and observe flow runs in their apps
+and leverage strong typing of the inputs, outputs and dependencies between steps
+in order to improve Developer Experience.
+
+Based on the Flow definition like this:
+
+```ts
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step('table_of_contents', async (payload) => {
+    // Placeholder function
+    return await fetchTableOfContents(payload.run.url);
+  })
+  .step('subpages', ['table_of_contents'], async (payload) => {
+    // Placeholder function
+    return await scrapeSubpages(payload.run.url, payload.table_of_contents.urls_of_subpages);
+  })
+  .step('summaries', ['subpages'], async (payload) => {
+    // Placeholder function
+    return await generateSummaries(payload.subpages.contentsOfSubpages);
+  })
+  .step('sentiments', ['subpages'], async (payload) => {
+    // Placeholder function
+    return await analyzeSentiments(payload.subpages.contentsOfSubpages);
+  })
+  .step('save_to_db', ['subpages', 'summaries', 'sentiments'], async (payload) => {
+    // Placeholder function
+    return await saveToDb(payload.subpages, payload.summaries, payload.sentiments);
+  });
+```
+
+We want to be able to infer the following information somehow:
+
+- The cumulative payload types that are built step-by-step
+- The relationships between steps that are established at runtime
+
+Those are the most important things we need, so users can for example trigger 
+flows and get annotations for the step results etc.
+Given the example flow I would like my users to be able to get their defined flow and do things like:
+
+```ts
+import type { ScrapeWebsiteFlow } from './flows/scrape_website';
+import { createClient } from '@pgflow/sdk';
+
+const { startFlow } = createClient(supabaseClient);
+
+const flowRun = startFlow<ScrapeWebsiteFlow>({
+  url: 'https://example.com', // this is type checked based on the Input to ScrapeWebsiteFlow
+});
+
+// here, 'subpages' (the name of step) would be type checked and only existing steps
+// can be used here, so user cannot await for non existing step
+const subpagesOutput = flowRun.stepCompleted('subpages');
+
+// the subpagesOutput is also type-annotated based on the return type inferred
+// from the handler for 'subpages' step, only based on the ScrapeWebsiteFlow type
+subpagesOutput.forEac() // this is an array because handler for 'subpages' returns an array
+```

package/prompts/step_types.md

@@ -0,0 +1,62 @@
+## Step types in MVP
+
+Regular Steps
+
+- Basic unit of work.
+- Executes a handler that receives outputs from its declared dependencies.
+- Its return value is passed to dependent steps.
+
+## Planned step types
+
+### Map Steps (tasks fanout)
+
+- Designed for when a dependency returns an array.
+- The handler runs once per array element (in parallel).
+- The outputs are collected back into an array (order preserved) and passed downstream.
+
+### Conditional Steps
+
+- Each step will be able to specify a condition, regardless of its type
+- Steps that run only when certain conditions are met
+- A condition is provided (as a JSON fragment)
+- At runtime, the inpuyt input for a step (from all deps) is matched via @> against the condition
+- If the condition is not met, the step does not run and marked as skipped
+- Dependent steps are not run and should probably be marked as skipped as well.
+
+### Manual Approval Steps
+
+- Steps that pause for human intervention.
+- They just differ by NOT immediately queueing a task.
+- Instead, they wait for an external update by calling **complete_step** to set their output and trigger downstream steps.
+
+### Subflow Steps
+
+- Encapsulate an entire subflow (a mini workflow) as a single step.
+- A subflow is defined using the same DSL as the main flow.
+- Each subflow has an automatic final step that gathers the outputs of all leaf steps.
+- The subflow step triggers the subflow and waits until its output is ready.
+- The aggregated output from the subflow becomes the output of the subflow step.
+
+### Fanout subflows step
+
+- Like Map steps, but instead of a task per array item, it runs a subflow per array item.
+- It gathers final steps from subflows into an output array for the fanout subflow step
+
+####
+
+• “Fanout subflow” steps do not have local tasks. Instead, they spawn child subflows and wait for them to finish.
+• You can track subflow completion with the same remaining_tasks field:
+  – Increment remaining_tasks by the number of child subflows.
+  – Decrement it each time a child subflow completes.
+  – When remaining_tasks reaches zero, the fanout subflow step is done.
+• Alternatively, you can add a remaining_subflows column to separate child‐subflow tracking from local tasks.
+  – This gives clearer semantics but requires extra logic to handle multiple completion conditions.
+• Most implementations unify subflow runs under remaining_tasks to reuse existing “remaining_tasks = 0 means done” checks.
+
+### Additional Techniques
+
+We simplify as much as possible and use other tools instead of reinventing the wheel.
+
+- Recurrent tasks are handled externally via Cron triggers.
+- Delays can be implemented using pgmq visibility timeouts.
+- The overall design treats a flow as a single function with one input (parameters) and one output (final aggregated output).

package/prompts/versioning.md

@@ -0,0 +1,16 @@
+# Versioning
+
+Flow Versioning Strategy #[[pgflow/Versioning]]
+Agreed on immutable flow definitions (similar to Temporal)
+Once a flow is uploaded to DB, it remains unchanged
+Versioning handled through flow slugs rather than explicit version numbers
+Users responsible for managing changes in safe, organized manner
+Benefits of immutable approach
+Simplifies implementation
+Provides natural versioning cascade for subflows
+Makes version transitions explicit and intentional
+Avoids "half-upgraded" scenarios
+Consciously decided against "latest" aliases for now
+Could introduce complexity and unpredictable behavior
+Users can implement their own aliasing logic if needed
+Explicit slugs provide clarity about which version is being used

package/queries/fail_permanently.sql

@@ -0,0 +1,17 @@
+select pgflow_tests.reset_db();
+select pgflow_tests.setup_flow('sequential');
+
+-- SETUP
+select pgflow.start_flow('sequential', '{"test": true}'::JSONB);
+
+-- default opt_max_attempts is 3, so failing twice should mark the task as failed
+select pgflow_tests.poll_and_fail('sequential');
+select pg_sleep(1.1);
+select pgflow_tests.poll_and_fail('sequential');
+select * from pgflow.step_tasks;
+select * from pgmq.q_sequential;
+-- select * from pgflow.step_tasks;
+--
+select * from pgflow.step_tasks;
+select * from pgflow.step_states;
+select * from pgflow.runs;

package/queries/fail_task.sql

@@ -0,0 +1,21 @@
+\x
+begin;
+select pgflow_tests.reset_db();
+select pgflow_tests.setup_flow('two_roots_left_right');
+
+--------------------------------------------------------------------------------
+--------------------------------------------------------------------------------
+--------------------------------------------------------------------------------
+
+
+select pgflow.start_flow('two_roots_left_right', '"hello"'::jsonb);
+
+select pgflow_tests.poll_and_complete('two_roots_left_right');
+select pgflow_tests.poll_and_complete('two_roots_left_right');
+select pgflow_tests.poll_and_complete('two_roots_left_right');
+select pgflow_tests.poll_and_complete('two_roots_left_right');
+
+select jsonb_pretty(output) from pgflow.runs;
+select * from pgflow.runs;
+
+rollback;