@pgflow/core 0.0.5 → 0.0.7

package/prompts/dsl-two-arg.ts

@@ -1,61 +0,0 @@
-const ScrapeWebsiteFlow = new Flow<Input>()
-  .step(
-    {
-      slug: 'verify_status',
-    },
-    async (payload) => {
-      // Placeholder function
-      return { status: 'success' };
-    }
-  )
-  .step(
-    {
-      slug: 'when_success',
-      dependsOn: ['verify_status'],
-      runIf: { verify_status: { status: 'success' } },
-    },
-    async (payload) => {
-      // Placeholder function
-      return await scrapeSubpages(
-        payload.run.url,
-        payload.table_of_contents.urls_of_subpages
-      );
-    }
-  )
-  .step(
-    {
-      slug: 'when_server_error',
-      dependsOn: ['verify_status'],
-      runUnless: { verify_status: { status: 'success' } },
-    },
-    async (payload) => {
-      // Placeholder function
-      return await generateSummaries(payload.subpages.contentsOfSubpages);
-    }
-  )
-  .step(
-    {
-      slug: 'sentiments',
-      dependsOn: ['subpages'],
-      maxAttempts: 5,
-      baseDelay: 10,
-    },
-    async (payload) => {
-      // Placeholder function
-      return await analyzeSentiments(payload.subpages.contentsOfSubpages);
-    }
-  )
-  .step(
-    {
-      slug: 'save_to_db',
-      dependsOn: ['subpages', 'summaries', 'sentiments'],
-    },
-    async (payload) => {
-      // Placeholder function
-      return await saveToDb(
-        payload.subpages,
-        payload.summaries,
-        payload.sentiments
-      );
-    }
-  );

package/prompts/dsl.md

@@ -1,119 +0,0 @@
-# Flow DSL
-
-Flow DSL is used do define shape of the flow and tie functions to particular steps.
-
-## Full flow example
-
-```ts
-const ScrapeWebsiteFlow = new Flow<Input>()
-  .step('table_of_contents', async (payload) => {
-    // Placeholder function
-    return await fetchTableOfContents(payload.run.url);
-  })
-  .step('subpages', ['table_of_contents'], async (payload) => {
-    // Placeholder function
-    return await scrapeSubpages(payload.run.url, payload.table_of_contents.urls_of_subpages);
-  })
-  .step('summaries', ['subpages'], async (payload) => {
-    // Placeholder function
-    return await generateSummaries(payload.subpages.contentsOfSubpages);
-  })
-  .step('sentiments', ['subpages'], async (payload) => {
-    // Placeholder function
-    return await analyzeSentiments(payload.subpages.contentsOfSubpages);
-  })
-  .step('save_to_db', ['subpages', 'summaries', 'sentiments'], async (payload) => {
-    // Placeholder function
-    return await saveToDb(payload.subpages, payload.summaries, payload.sentiments);
-  });
-```
-
-## Explanation
-
-This is Fluent API stype DSL but it is very simple:
-
-1. Users create a flow by initializing a `Flow` object with a mandatory
-   type annotation for the Flow `input` - this is the type of the payload
-   users would start flow with and must be serializable to Json:
-
-```ts
-type Input = {
-  url: string; // url of the website to scrape
-};
-
-const ScrapeWebsiteFlow = new Flow<Input>()
-```
-
-2. Then they define steps by calling `.step(stepSlug: string, depsSlugs: string[], handler: Function)` method.
-   The `depsSlugs` array can be ommited if the step has no dependencies.
-   This kind of steps are named "root steps" and are run first and passed only the flow input payload:
-
-```ts
-const ScrapeWebsiteFlow = new Flow<Input>()
-  .step('table_of_contents', async (payload) => {
-    const { run } = payload;
-    // do something
-    // make sure to return some value so next steps can use it
-    return {
-      urls_of_subpages,
-      title
-    }
-  })
-```
-
-The `payload` object always have a special key `run` which is value passed as flow input -
-every step can access and use it.
-
-What the step handler returns is very important!
-We name it `output` and it will be persisted in the the database 
-and used as `input` for the dependent steps.
-
-It must be serializable to json.
-
-3. Then they define dependent steps by calling `.step(stepSlug: string, depsSlugs: string[], handler: Function)` method,
-   now providing an array of dependencies slugs: `['table_of_contents']`.
-
-```ts
-  .step('subpages', ['table_of_contents'], async (payload) => {
-    const { run, urls_of_subpages } = payload;
-    // do something
-    // make sure to return some value so next steps can use it
-    return {
-      contentsOfSubpages
-    }
-  })
-```
-
-Notice how the `payload` object got a new key `urls_of_subpages` - each dependency
-results (the persisted return value from handler) will get passed to `payload` under the dependency slug key.
-
-```ts
-{
-  run: { url: 'https://example.com' },
-  table_of_contents: { 
-    urls_of_subpages: ['https://example.com/subpage1', 'https://example.com/subpage2'] 
-  }
-}
-```
-
-4. There can be multiple steps in parallel:
-
-```ts
-.step('summaries', ['subpages'], async (payload) => await doSomeStuff())
-.step('sentiments', ['subpages'], async (payload) => await doSomeStuff())
-```
-
-5. Steps can also depend on more than one other step:
-
-```ts
-.step('save_to_db', ['subpages', 'summaries', 'sentiments'], async (payload) => await saveToDb())
-```
-
-6. When run finishes, the `output`s of steps that have no dependents will be combined
-   together and saved as the run's `output`. This object will be built in similar
-   way as the step `input` object, but will lack the `run` key.
-
-7. Type Safety - all the step payloads types are inferred from the combination
-   of Flow input, handler inferred return type and the shape of the graph.
-
-   So users will always know that type is their step input.

package/prompts/fanout_steps.md

@@ -1 +0,0 @@
-.file prompts/architect.md -- we are in monorepo for pgflow - postgres native workflow engine. we are not in the root, we are in pkgs/core - an sql part. sql code lives in supabase/migrations/ and tests live in supabase/tests/. you can check some info about step types and declarative sql in prompts/ folder. your jobs is to implement next step type - 'fanout_tasks' type, which just enqueues multiple tasks for a single step - one task per input array item. this step type must have a json path parameter that will tell which part of input is an array of items. you must change the code so steps with this type are handled differently in all the functions. use task_index to indicate which array item the given task is for. when compoleting task, do not proceed with completing steps etc. unless all the tasks for given step state are completed, then use task_index to order their outputs and use all those as a step output array. try to understand what needs to be done first. modify existing migrations, do not create new ones - this is unrealeased source code and we can change migrations.

package/prompts/json_schemas.md

@@ -1,36 +0,0 @@
-# JSON Schemas
-
-JSON schemas can be inferred from the steps `input` types,
-so it is relatively easy to build a JSON schema for each step input.
-
-The same goes for the JSON Schema for the flow input.
-
-## Schema storage
-
-Schemas should be stored in the `pgflow.flows` and `pgflow.steps` tables.
-
-## Schemas in versioning
-
-To make sure that slight changes in the input/output types of steps
-trigger a new version of the flow, we need to use the inferred schemas
-when generating a version hash of the flow.
-
-## Schemas as validation
-
-We can use schemas to do data validation for step handlers:
-
-1. Task executors can validate the runtime input payloads for handler
-   and their output results against the schema.
-2. Core SQL engine can use `pg_jsonschema` to validate the input values to flows
-   and maybe the input values to steps and fail steps if they don't match.
-
-## Problems
-
-Doing any JSON Schema validation in database is probably not a good idea because
-of performance impact it would have.
-
-Using runtime validation in Task Executors is probably good enough,
-with exception of validating the Flow input - you start flows less often than
-steps and it seems like a good idea to validate the input database-wise.
-
-

package/prompts/one_shot.md

@@ -1,286 +0,0 @@
-Your job is to implement required SQL schemas and functions for an MVP of my open source Postgres-native workflow orchestration engine called pgflow.
-
-The main idea of the project is to keep shape of the DAG (nodes and edges) and its runtime state in the database
-and expose SQL functions that will allow to propagate through the state.
-
-Real work is done on the task queue workers and the functions from pgflow are only orchestrating
-the queue messages.
-
-Workers are supposed to call user functions with the input from the queue message,
-and should acknowledge the completion of the task or its failure (error thrown) by
-calling appropriate pgflow SQL functions.
-
-This way the orchestration is decoupled from the execution.
-
-I have a concrete implementation plan for you to follow and will unfold it
-step by step below.
-
-## Assumptions/best practices
-
-### We are building Minimal Viable Product
-
-Remember that we are building MVP and main focus should be on shipping something as soon as possible,
-by cutting scope, simplifying the architectures and code.
-
-But the outlined features are definitely something that we will be doing in the future.
-I am most certain about the foreach-array steps - this is a MUST have.
-So your focus should be on trying to implement the MVP but not closing the doors to the future improvements.
-
-### Slugs
-
-We do not use serial IDs nor UUIDs for static things, we use "slugs" instead.
-A slug is just a string that conforms to following rules:
-
-```sql
-slug is not null
-and slug <> ''
-and length(slug) <= 128
-and slug ~ '^[a-zA-Z_][a-zA-Z0-9_]*$';
-```
-
-We use UUID for identifying particular run of the flow.
-But the states of steps for that particular run are not identified by separate UUIDs,
-but rather by a pair of run_id and step_slug. This pattern allows to easily refer
-to steps and flows by their slugs. **Leverage this pattern everywhere you can!**
-
-### References/fkeys
-
-Use foreign keys everywhere to ensure consistency.
-Use composite foreign keys and composite primary keys composed of flow/step slugs and run_id's if needed.
-
-### Declarative vs procedural
-
-**YOU MUST ALWAYS PRIORITIZE DECLARATIVE STYLE** and prioritize Batching operations.
-
-Avoid plpgsql as much as you can.
-It is important to have your DB procedures run in batched ways and use declarative rather than procedural constructs where possible:
-
-- do not ever use `language plplsql` in functions, always use `language sql`
-- don't do loops, do SQL statements that address multiple rows at once.
-- don't write trigger functions that fire for a single row, use `FOR EACH STATEMENT` instead.
-- don't call functions for each row in a result set, a condition, a join, or whatever; instead use functions that return `SETOF` and join against these.
-
-If you're constructing dynamic SQL, you should only ever use `%I` and `%L` when using `FORMAT` or similar; you should never see `%s` (with the very rare exception of where you're merging in another SQL fragment that you've previously formatted using %I and %L).
-
-Remember, that functions have significant overhead in Postgres - instead of factoring into lots of tiny functions, think about how to make your code more expressive so there's no need.
-
-## Schemas
-
-### pgflow.flows
-
-A static definition of a flow (DAG):
-
-```sql
-CREATE TABLE pgflow.flows (
-    flow_slug text PRIMARY KEY NOT NULL  -- Unique identifier for the flow
-    CHECK (is_valid_slug(flow_slug))
-);
-```
-
-### pgflow.steps
-
-A static definition of a step within a flow (a DAG "nodes"):
-
-```sql
-CREATE TABLE pgflow.steps (
-    flow_slug text NOT NULL REFERENCES flows (flow_slug),
-    step_slug text NOT NULL,
-    step_type text NOT NULL DEFAULT 'single',
-    PRIMARY KEY (flow_slug, step_slug),
-    CHECK (is_valid_slug(flow_slug)),
-    CHECK (is_valid_slug(step_slug))
-);
-```
-
-### pgflow.deps
-
-A static definition of dependencies between steps (a DAG "edges"):
-
-```sql
-CREATE TABLE pgflow.deps (
-    flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug),
-    dep_slug text NOT NULL,  -- The step that must complete first
-    step_slug text NOT NULL,   -- The step that depends on dep_slug
-    PRIMARY KEY (flow_slug, dep_slug, step_slug),
-    FOREIGN KEY (flow_slug, dep_slug)
-    REFERENCES pgflow.steps (flow_slug, step_slug),
-    FOREIGN KEY (flow_slug, step_slug)
-    REFERENCES pgflow.steps (flow_slug, step_slug),
-    CHECK (dep_slug != step_slug)  -- Prevent self-dependencies
-    CHECK (is_valid_slug(step_slug))
-);
-```
-
-### pgflow.runs
-
-A table storing runtime state of given flow.
-A run is identified by a `flow_slug` and `run_id`.
-
-```sql
-CREATE TABLE pgflow.runs (
-    run_id uuid PRIMARY KEY NOT NULL DEFAULT gen_random_uuid(),
-    flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug), -- denormalized
-    status text NOT NULL DEFAULT 'started',
-    input jsonb NOT NULL,
-    CHECK (status IN ('started', 'failed', 'completed'))
-)
-```
-
-There is also `status` that currently can be started, failed or completed.
-);
-
-````
-
-There is also `status` that currently can be pending, failed or completed.
-
-### pgflow.step_states
-
-Represents a state of a particular step in a particular run.
-
-```sql
-
--- Step states table - tracks the state of individual steps within a run
-CREATE TABLE pgflow.step_states (
-    flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug),
-    run_id uuid NOT NULL REFERENCES pgflow.runs (run_id),
-    step_slug text NOT NULL,
-    status text NOT NULL DEFAULT 'created',
-    PRIMARY KEY (run_id, step_slug),
-    FOREIGN KEY (flow_slug, step_slug)
-    REFERENCES pgflow.steps (flow_slug, step_slug),
-    CHECK (status IN ('created', 'started', 'completed', 'failed'))
-);
-);
-````
-
-### pgflow.step_tasks
-
-This table is really unique and interesting. We are starting the development
-of the flow orchestration engine with a simple step that runs one unit of work.
-
-But I imagine we would suppport additional types of steps, like:
-
-- a step that requires input array and enqueues a task per array item, so they are created in parallel
-- a step that runs some preprocessing/postprocessing in an additional task
-
-So in order to accomodate this, we need an additional layer between step_state and
-an actual task queue, in order to track which messages belong to which steps,
-in case there are more than 1 unit of work for given step.
-
-```sql
--- Executio logs table - tracks the task of individual steps
-CREATE TABLE pgflow.step_tasks (
-flow_slug text NOT NULL REFERENCES pgflow.flows (flow_slug),
-step_slug text NOT NULL,
-run_id uuid NOT NULL REFERENCES pgflow.runs (run_id),
-status text NOT NULL DEFAULT 'queued',
-input jsonb NOT NULL, -- payload that will be passed to queue message
-output jsonb, -- like step_result but for task, can store result or error/stacktrace
-message_id bigint, -- an id of the queue message
-CONSTRAINT step_tasks_pkey PRIMARY KEY (run_id, step_slug),
-FOREIGN KEY (run_id, step_slug)
-REFERENCES pgflow.step_states (run_id, step_slug),
-CHECK (status IN ('queued', 'started', 'failed', 'completed')),
-CHECK (is_valid_slug(flow_slug)),
-CHECK (is_valid_slug(step_slug))
-);
-```
-
-## Typescript DSL, topological ordering and acyclicity validation
-
-The simple typescript DSL will be created that will have string typing
-and will enforce adding steps in a topological order, preventing
-cycles by the strict ordering of the steps addition.
-
-Typescript DSL looks like this:
-
-```ts
-const BasicFlow = new Flow<string>()
-  .step('root', ({ run }) => {
-    return `[${run}]r00t`;
-  })
-  .step('left', ['root'], ({ root: r }) => {
-    return `${r}/left`;
-  })
-  .step('right', ['root'], ({ root: r }) => {
-    return `${r}/right`;
-  })
-  .step('end', ['left', 'right'], ({ left, right, run }) => {
-    return `<${left}> and <${right}> of (${run})`;
-  });
-```
-
-This will be compiled to a simple SQL calling SQL function `pgflow.add_step(flow_slug, step_slug, dep_step_slugs[])`:
-
-```sql
-SELECT pgflow.add_step('basic', 'root', ARRAY[]::text[]);
-SELECT pgflow.add_step('basic', 'left', ARRAY['root']);
-SELECT pgflow.add_step('basic', 'right', ARRAY['root']);
-SELECT pgflow.add_step('basic', 'end', ARRAY['left', 'right']);
-```
-
-## SQL functions API
-
-This describes public SQL functions that are available to developer using pgflow
-and to the workers.
-
-Developer calls `start_flow` and rest is called by the workers.
-
-### pgflow.start_flow(flow_slug::text, input::jsonb)
-
-This function is used to start a flow.
-It should work like this:
-
-- create a new `pgflow.runs` row for given flow_slug
-- create all the `pgflow.step_states` rows corresponding to the steps in the flow
-- find root steps (ones without dependencies) and call "start_step" on each of them
-
-### pgflow.start_step(run_id::uuid, step_slug::text)
-
-This function is called by start_flow but also by complete_step_task (or somewhere near its call)
-when worker acknowledges the step_task completion and it is detected, that there are ready dependant
-steps to be started.
-
-It should probably call start_step_task under the hood, which will:
-
-- updating step_state status/timestamps
-- creating a step_task row
-- enqueueing a queue message for this step_task
-
-For other step types, like array/foreach, it would probably call the step_task
-for each array item, so more than one step task is created and more than one message is enqueued.
-
-### pgflow.start_step_task(run_id::uuid, step_slug::text, task_id::bigint)
-
-I am not yet sure how this will work for other step types that will need more step tasks.
-But probably each step type would have its own implementation of this function,
-and a simple step type will just create a new step_task row and enqueue it.
-
-But an array/foreach step type would need a different implementation.
-Would need to check the input for the step which is an array, and would
-create a new step_task for each array item and enqueue as many messages as there are items in the array.
-
-### pgflow.complete_step_task(run_id::uuid, step_slug::text, output::jsonb)
-
-This will be called by the worker when a step_task is completed.
-It will work like this in the simplified version when one step_state corresponds to one step_task:
-
-- it marks step_task as completed, saving the output
-- it in turns mark step_state as completed, saving the output
-- then it should check for any dependant steps (steps that depend on just completed step) in the same run
-- it should then check if any of those dependant steps are "ready" - meaning, all their dependencies are completed
-- for each of those
-
-I am not yet sure how this will work for other step types that will need more step tasks.
-Probably each step type would have its own implementation of this function,
-so a simple step will just call complete_step_state when complete_step_task is called.
-
-An array/foreach step type would need a different implementation.
-Would probably need to check if other step_tasks are still pending.
-If all are already completed, it would just call complete_step_state,
-otherwise it will just continue, so other (last) step task can complete the step state.
-
-### pgflow.fail_step_task(run_id::uuid, step_slug::text, error::jsonb)
-
-This is very similar to complete_step_task, but it will mark step_task as failed,
-will save error message and will call fail_step_state instead of complete_step_state.