@pgflow/core 0.0.5-prealpha.2

package/prompts/architect.md

@@ -0,0 +1,87 @@
+# Architect Mode
+
+## Your Role
+You are a senior software architect with extensive experience designing scalable, maintainable systems. Your purpose is to thoroughly analyze requirements and design optimal solutions before any implementation begins. You must resist the urge to immediately write code and instead focus on comprehensive planning and architecture design.
+
+## Your Behavior Rules
+- You must thoroughly understand requirements before proposing solutions
+- You must reach 90% confidence in your understanding before suggesting implementation
+- You must identify and resolve ambiguities through targeted questions
+- You must document all assumptions clearly
+
+## Process You Must Follow
+
+### Phase 1: Requirements Analysis
+1. Carefully read all provided information about the project or feature
+2. Extract and list all functional requirements explicitly stated
+3. Identify implied requirements not directly stated
+4. Determine non-functional requirements including:
+   - Performance expectations
+   - Security requirements
+   - Scalability needs
+   - Maintenance considerations
+5. Ask clarifying questions about any ambiguous requirements
+6. Report your current understanding confidence (0-100%)
+
+### Phase 2: System Context Examination
+1. If an existing codebase is available:
+   - Request to examine directory structure
+   - Ask to review key files and components
+   - Identify integration points with the new feature
+2. Identify all external systems that will interact with this feature
+3. Define clear system boundaries and responsibilities
+4. If beneficial, create a high-level system context diagram
+5. Update your understanding confidence percentage
+
+### Phase 3: Architecture Design
+1. Propose 2-3 potential architecture patterns that could satisfy requirements
+2. For each pattern, explain:
+   - Why it's appropriate for these requirements
+   - Key advantages in this specific context
+   - Potential drawbacks or challenges
+3. Recommend the optimal architecture pattern with justification
+4. Define core components needed in the solution, with clear responsibilities for each
+5. Design all necessary interfaces between components
+6. If applicable, design database schema showing:
+   - Entities and their relationships
+   - Key fields and data types
+   - Indexing strategy
+7. Address cross-cutting concerns including:
+   - Authentication/authorization approach
+   - Error handling strategy
+   - Logging and monitoring
+   - Security considerations
+8. Update your understanding confidence percentage
+
+### Phase 4: Technical Specification
+1. Recommend specific technologies for implementation, with justification
+2. Break down implementation into distinct phases with dependencies
+3. Identify technical risks and propose mitigation strategies
+4. Create detailed component specifications including:
+   - API contracts
+   - Data formats
+   - State management
+   - Validation rules
+5. Define technical success criteria for the implementation
+6. Update your understanding confidence percentage
+
+### Phase 5: Transition Decision
+1. Summarize your architectural recommendation concisely
+2. Present implementation roadmap with phases
+3. State your final confidence level in the solution
+4. If confidence ≥ 90%:
+   - State: "I'm ready to build! Switch to Agent mode and tell me to continue."
+5. If confidence < 90%:
+   - List specific areas requiring clarification
+   - Ask targeted questions to resolve remaining uncertainties
+   - State: "I need additional information before we start coding."
+
+## Response Format
+Always structure your responses in this order:
+1. Current phase you're working on
+2. Findings or deliverables for that phase
+3. Current confidence percentage
+4. Questions to resolve ambiguities (if any)
+5. Next steps
+
+Remember: Your primary value is in thorough design that prevents costly implementation mistakes. Take the time to design correctly before suggesting to use Agent mode.

package/prompts/condition.md

@@ -0,0 +1,33 @@
+# Conditional Steps in the Flow DSL
+
+Conditional steps allow steps to run only when certain criteria are met based on the incoming payload. Instead of always executing as soon as their dependencies complete, these steps check the provided condition against the input data.
+
+## How It Works
+
+- **Definition**: A condition is supplied as a JSON fragment via the step options (for example, using `runIf` or `runUnless`).
+- **Evaluation**: At runtime, the system evaluates the condition by comparing the step's combined inputs against the JSON fragment.
+- **Mechanism**: Under the hood, the payload is matched against the condition using a JSON containment operator (`@>`), commonly available in PostgreSQL. This operator checks if the input JSON "contains" the condition JSON structure.
+- **Outcome**:
+  - If the condition is met (for `runIf`) or not met (for `runUnless`), the step is executed.
+  - If the condition fails, the step is marked as skipped, and its downstream dependent steps are not executed (or are similarly marked as skipped).
+
+This design helps ensure that unnecessary processing is avoided when prerequisites are not satisfied.
+
+## Type safety
+
+Options object can be strictly type-safe and only allow values that are available in the payload,
+so it is impossible to define invalid condition object.
+
+## Marking as skipped
+
+Skipped steps are not considered a failure but will propagate skipped status to all dependent steps and
+they will not run.
+
+This way we can achieve a kinda robust low level branching logic - users can define branches
+by creating steps with mutually-exclusive conditions, so only one branch will be executed:
+
+```ts
+const ScrapeWebsiteFlow = new Flow<{ input: true }>()
+  .step({ slug: 'run_if_true', runIf: { run: { input: true } } }, handler)
+  .step({ slug: 'run_if_false', runUnless: { run: { input: true } } }, handler);
+```

package/prompts/declarative_sql.md

@@ -0,0 +1,15 @@
+### Declarative vs procedural
+
+**YOU MUST ALWAYS PRIORITIZE DECLARATIVE STYLE** and prioritize Batching operations.
+
+Avoid plpgsql as much as you can.
+It is important to have your DB procedures run in batched ways and use declarative rather than procedural constructs where possible:
+
+- do not ever use `language plplsql` in functions, always use `language sql`
+- don't do loops, do SQL statements that address multiple rows at once.
+- don't write trigger functions that fire for a single row, use `FOR EACH STATEMENT` instead.
+- don't call functions for each row in a result set, a condition, a join, or whatever; instead use functions that return `SETOF` and join against these.
+
+If you're constructing dynamic SQL, you should only ever use `%I` and `%L` when using `FORMAT` or similar; you should never see `%s` (with the very rare exception of where you're merging in another SQL fragment that you've previously formatted using %I and %L).
+
+Remember, that functions have significant overhead in Postgres - instead of factoring into lots of tiny functions, think about how to make your code more expressive so there's no need.

package/prompts/deps_in_payloads.md

@@ -0,0 +1,20 @@
+currently the 'input' jsonb that we are building contains only the 'run' input
+
+we really want to have this jsonb to contain all the deps outputs
+by dep output i mean a step_tasks.output value that corresponds to step_states row that is a dependency of the given updated_step_tasks (via step_states->steps)
+the step_slug of a given dependency should be used as key and its output as value in the input jsonb
+
+so, if a given updated_step_task belongs to step_state, that have 2 dependencies:
+
+step_slug=dep_a output=123
+step_slug=dep_b output=456
+
+we would like the final 'input' jsonb to look like this:
+
+{
+  "run": r.input,
+  "dep_a": dep_a_step_task.output,
+  "dep_b": dep_b_step_task.output
+}
+
+write appropriate joins and augment this code with this requirements

package/prompts/dsl-multi-arg.ts

@@ -0,0 +1,48 @@
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step('verify_status', async (payload) => {
+    // Placeholder function
+    return { status: 'success' };
+  })
+  .step(
+    'when_success',
+    ['verify_status'],
+    async (payload) => {
+      // Placeholder function
+      return await scrapeSubpages(
+        payload.run.url,
+        payload.table_of_contents.urls_of_subpages
+      );
+    },
+    { runIf: { verify_status: { status: 'success' } } }
+  )
+  .step(
+    'when_server_error',
+    ['verify_status'],
+    async (payload) => {
+      // Placeholder function
+      return await generateSummaries(payload.subpages.contentsOfSubpages);
+    },
+    { runUnless: { verify_status: { status: 'success' } } }
+  )
+
+  .step(
+    'sentiments',
+    ['subpages'],
+    async (payload) => {
+      // Placeholder function
+      return await analyzeSentiments(payload.subpages.contentsOfSubpages);
+    },
+    { maxAttempts: 5, baseDelay: 10 }
+  )
+  .step(
+    'save_to_db',
+    ['subpages', 'summaries', 'sentiments'],
+    async (payload) => {
+      // Placeholder function
+      return await saveToDb(
+        payload.subpages,
+        payload.summaries,
+        payload.sentiments
+      );
+    }
+  );

package/prompts/dsl-options.md

@@ -0,0 +1,39 @@
+# Flow DSL with options
+
+The idea is to add 4th argument to the `.step` method which will be an object
+for the step options:
+
+```ts
+{
+  runIf: Json;
+  runUnless: Json;
+  maxAttempts: number;
+  baseDelay: number;
+}
+```
+
+## Full flow example
+
+```ts
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step('verify_status', async (payload) => {
+    // Placeholder function
+    return { status: 'success' }
+  })
+  .step('when_success', ['verify_status'], async (payload) => {
+    // Placeholder function
+    return await scrapeSubpages(payload.run.url, payload.table_of_contents.urls_of_subpages);
+  }, { runIf: { status: 'success' } })
+  .step('when_server_error', ['verify_status'], async (payload) => {
+    // Placeholder function
+    return await generateSummaries(payload.subpages.contentsOfSubpages);
+  }, { runUnless: { status: 'success' } })
+  .step('sentiments', ['subpages'], async (payload) => {
+    // Placeholder function
+    return await analyzeSentiments(payload.subpages.contentsOfSubpages);
+  }, { maxAttempts: 5, baseDelay: 10 })
+  .step('save_to_db', ['subpages', 'summaries', 'sentiments'], async (payload) => {
+    // Placeholder function
+    return await saveToDb(payload.subpages, payload.summaries, payload.sentiments);
+  });
+```

package/prompts/dsl-single-arg.ts

@@ -0,0 +1,51 @@
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step({
+    id: 'verify_status',
+    handler: async (payload) => {
+      // Placeholder function
+      return { status: 'success' };
+    },
+  })
+  .step({
+    id: 'when_success',
+    deps: ['verify_status'],
+    runIf: { verify_status: { status: 'success' } },
+    async handler(payload) {
+      // Placeholder function
+      return await scrapeSubpages(
+        payload.run.url,
+        payload.table_of_contents.urls_of_subpages
+      );
+    }
+  })
+  .step({
+    id: 'when_server_error',
+    deps: ['verify_status'],
+    runUnless: { verify_status: { status: 'success' } },
+    async handler(payload) {
+      // Placeholder function
+      return await generateSummaries(payload.subpages.contentsOfSubpages);
+    }
+  })
+  .step({
+    id: 'sentiments',
+    deps: ['subpages'],
+    async handler(payload) {
+      // Placeholder function
+      return await analyzeSentiments(payload.subpages.contentsOfSubpages);
+    },
+    maxAttempts: 5,
+    baseDelay: 10
+  })
+  .step({
+    id: 'save_to_db',
+    deps: ['subpages', 'summaries', 'sentiments'],
+    async handler(payload) {
+      // Placeholder function
+      return await saveToDb(
+        payload.subpages,
+        payload.summaries,
+        payload.sentiments
+      );
+    },
+  });

package/prompts/dsl-two-arg.ts

@@ -0,0 +1,61 @@
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step(
+    {
+      slug: 'verify_status',
+    },
+    async (payload) => {
+      // Placeholder function
+      return { status: 'success' };
+    }
+  )
+  .step(
+    {
+      slug: 'when_success',
+      dependsOn: ['verify_status'],
+      runIf: { verify_status: { status: 'success' } },
+    },
+    async (payload) => {
+      // Placeholder function
+      return await scrapeSubpages(
+        payload.run.url,
+        payload.table_of_contents.urls_of_subpages
+      );
+    }
+  )
+  .step(
+    {
+      slug: 'when_server_error',
+      dependsOn: ['verify_status'],
+      runUnless: { verify_status: { status: 'success' } },
+    },
+    async (payload) => {
+      // Placeholder function
+      return await generateSummaries(payload.subpages.contentsOfSubpages);
+    }
+  )
+  .step(
+    {
+      slug: 'sentiments',
+      dependsOn: ['subpages'],
+      maxAttempts: 5,
+      baseDelay: 10,
+    },
+    async (payload) => {
+      // Placeholder function
+      return await analyzeSentiments(payload.subpages.contentsOfSubpages);
+    }
+  )
+  .step(
+    {
+      slug: 'save_to_db',
+      dependsOn: ['subpages', 'summaries', 'sentiments'],
+    },
+    async (payload) => {
+      // Placeholder function
+      return await saveToDb(
+        payload.subpages,
+        payload.summaries,
+        payload.sentiments
+      );
+    }
+  );

package/prompts/dsl.md

@@ -0,0 +1,119 @@
+# Flow DSL
+
+Flow DSL is used do define shape of the flow and tie functions to particular steps.
+
+## Full flow example
+
+```ts
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step('table_of_contents', async (payload) => {
+    // Placeholder function
+    return await fetchTableOfContents(payload.run.url);
+  })
+  .step('subpages', ['table_of_contents'], async (payload) => {
+    // Placeholder function
+    return await scrapeSubpages(payload.run.url, payload.table_of_contents.urls_of_subpages);
+  })
+  .step('summaries', ['subpages'], async (payload) => {
+    // Placeholder function
+    return await generateSummaries(payload.subpages.contentsOfSubpages);
+  })
+  .step('sentiments', ['subpages'], async (payload) => {
+    // Placeholder function
+    return await analyzeSentiments(payload.subpages.contentsOfSubpages);
+  })
+  .step('save_to_db', ['subpages', 'summaries', 'sentiments'], async (payload) => {
+    // Placeholder function
+    return await saveToDb(payload.subpages, payload.summaries, payload.sentiments);
+  });
+```
+
+## Explanation
+
+This is Fluent API stype DSL but it is very simple:
+
+1. Users create a flow by initializing a `Flow` object with a mandatory
+   type annotation for the Flow `input` - this is the type of the payload
+   users would start flow with and must be serializable to Json:
+
+```ts
+type Input = {
+  url: string; // url of the website to scrape
+};
+
+const ScrapeWebsiteFlow = new Flow<Input>()
+```
+
+2. Then they define steps by calling `.step(stepSlug: string, depsSlugs: string[], handler: Function)` method.
+   The `depsSlugs` array can be ommited if the step has no dependencies.
+   This kind of steps are named "root steps" and are run first and passed only the flow input payload:
+
+```ts
+const ScrapeWebsiteFlow = new Flow<Input>()
+  .step('table_of_contents', async (payload) => {
+    const { run } = payload;
+    // do something
+    // make sure to return some value so next steps can use it
+    return {
+      urls_of_subpages,
+      title
+    }
+  })
+```
+
+The `payload` object always have a special key `run` which is value passed as flow input -
+every step can access and use it.
+
+What the step handler returns is very important!
+We name it `output` and it will be persisted in the the database 
+and used as `input` for the dependent steps.
+
+It must be serializable to json.
+
+3. Then they define dependent steps by calling `.step(stepSlug: string, depsSlugs: string[], handler: Function)` method,
+   now providing an array of dependencies slugs: `['table_of_contents']`.
+
+```ts
+  .step('subpages', ['table_of_contents'], async (payload) => {
+    const { run, urls_of_subpages } = payload;
+    // do something
+    // make sure to return some value so next steps can use it
+    return {
+      contentsOfSubpages
+    }
+  })
+```
+
+Notice how the `payload` object got a new key `urls_of_subpages` - each dependency
+results (the persisted return value from handler) will get passed to `payload` under the dependency slug key.
+
+```ts
+{
+  run: { url: 'https://example.com' },
+  table_of_contents: { 
+    urls_of_subpages: ['https://example.com/subpage1', 'https://example.com/subpage2'] 
+  }
+}
+```
+
+4. There can be multiple steps in parallel:
+
+```ts
+.step('summaries', ['subpages'], async (payload) => await doSomeStuff())
+.step('sentiments', ['subpages'], async (payload) => await doSomeStuff())
+```
+
+5. Steps can also depend on more than one other step:
+
+```ts
+.step('save_to_db', ['subpages', 'summaries', 'sentiments'], async (payload) => await saveToDb())
+```
+
+6. When run finishes, the `output`s of steps that have no dependents will be combined
+   together and saved as the run's `output`. This object will be built in similar
+   way as the step `input` object, but will lack the `run` key.
+
+7. Type Safety - all the step payloads types are inferred from the combination
+   of Flow input, handler inferred return type and the shape of the graph.
+
+   So users will always know that type is their step input.

package/prompts/fanout_steps.md

@@ -0,0 +1 @@
+.file prompts/architect.md -- we are in monorepo for pgflow - postgres native workflow engine. we are not in the root, we are in pkgs/core - an sql part. sql code lives in supabase/migrations/ and tests live in supabase/tests/. you can check some info about step types and declarative sql in prompts/ folder. your jobs is to implement next step type - 'fanout_tasks' type, which just enqueues multiple tasks for a single step - one task per input array item. this step type must have a json path parameter that will tell which part of input is an array of items. you must change the code so steps with this type are handled differently in all the functions. use task_index to indicate which array item the given task is for. when compoleting task, do not proceed with completing steps etc. unless all the tasks for given step state are completed, then use task_index to order their outputs and use all those as a step output array. try to understand what needs to be done first. modify existing migrations, do not create new ones - this is unrealeased source code and we can change migrations.

package/prompts/json_schemas.md

@@ -0,0 +1,36 @@
+# JSON Schemas
+
+JSON schemas can be inferred from the steps `input` types,
+so it is relatively easy to build a JSON schema for each step input.
+
+The same goes for the JSON Schema for the flow input.
+
+## Schema storage
+
+Schemas should be stored in the `pgflow.flows` and `pgflow.steps` tables.
+
+## Schemas in versioning
+
+To make sure that slight changes in the input/output types of steps
+trigger a new version of the flow, we need to use the inferred schemas
+when generating a version hash of the flow.
+
+## Schemas as validation
+
+We can use schemas to do data validation for step handlers:
+
+1. Task executors can validate the runtime input payloads for handler
+   and their output results against the schema.
+2. Core SQL engine can use `pg_jsonschema` to validate the input values to flows
+   and maybe the input values to steps and fail steps if they don't match.
+
+## Problems
+
+Doing any JSON Schema validation in database is probably not a good idea because
+of performance impact it would have.
+
+Using runtime validation in Task Executors is probably good enough,
+with exception of validating the Flow input - you start flows less often than
+steps and it seems like a good idea to validate the input database-wise.
+
+