@pgflow/dsl 0.0.5 → 0.0.6

package/brainstorming/dsl/dsl-analysis.md

@@ -1,512 +0,0 @@
-# Comprehensive Critique of a Database-Centric, DSL-Driven Flow Approach
-
-Below is a detailed analysis of using a TypeScript-based DSL to declaratively define and manage workflows (DAGs) directly in PostgreSQL (for Supabase or similar). The approach is often referred to as “DSL-in-the-DB.”
-
-We’ll first outline **10 common pitfalls and problems** you might face. Each pitfall has a short code snippet or example that illustrates the issue. Next, we’ll discuss **10 positive use cases** where this approach really shines.
-
----
-
-## 1. Potential Pitfalls and Challenges
-
-### 1.1 Complex Conditional Flows
-
-**Description**  
-When you define flows with many conditional branching (e.g., short-circuiting certain steps if a condition is met), concurrency and runIf conditions can make your definitions complex. Developers may struggle with the logic or misuse the condition checks.
-
-**Example**
-
-```ts
-// "runIf" or "runUnless" might cause confusion if combined incorrectly:
-const SomeComplexFlow = new Flow<{ userIsVIP: boolean }>()
-  .step(
-    {
-      slug: 'special_vip_step',
-      dependsOn: ['check_vip'],
-      runIf: { run: { userIsVIP: true } }, // runs only for VIP
-    },
-    async () => {
-      // ...
-      return { msg: 'Completed VIP step.' };
-    }
-  )
-  .step(
-    {
-      slug: 'normal_step',
-      dependsOn: ['check_vip'],
-      runUnless: { run: { userIsVIP: true } }, // runs only for non-VIP
-    },
-    async () => {
-      // ...
-      return { msg: 'Completed normal step.' };
-    }
-  );
-
-// If a developer forgets to handle unexpected logic (like userIsVIP = undefined),
-// the runIf conditions can fail or skip steps unintentionally.
-```
-
-**Key Issue**
-
-- Hard-to-debug mixing of `runIf` and `runUnless` conditions, potentially leading to partial or skipped steps.
-- In some cases, you might need nested conditional flows, which become verbose.
-
----
-
-### 1.2 Over-Reliance on Database Transactions
-
-**Description**  
-Everything runs within database constructs. Handling long-running tasks entirely inside transactions or relying too heavily on transactional guarantees might lead to performance or locking issues.  
-**Example**
-
-```sql
--- Suppose we have a step that triggers a heavy operation:
-SELECT pgflow.add_step(
-  flow_slug => 'heavy_computation_flow',
-  step_slug => 'ml_training',
-  deps => ARRAY['data_fetch']
-);
-
--- Worker picks up 'ml_training' but it runs for hours:
--- If we rely on immediate DB transaction, we risk timeouts or locks.
-```
-
-**Key Issue**
-
-- When tasks run too long, the worker might exceed the DB or locked row’s timeouts.
-
----
-
-### 1.3 Incorrect or Missing Step Ordering
-
-**Description**  
-The DSL typically enforces topological ordering, but you can still forget dependencies or incorrectly define them, causing cycles or orphan steps.
-
-**Example**
-
-```sql
--- Mistakenly adding a step that points back to an ancestor (cycle):
-SELECT pgflow.add_step('some_flow', 'step_a');
-SELECT pgflow.add_step('some_flow', 'step_b', deps => ARRAY['step_a']);
-SELECT pgflow.add_step('some_flow', 'step_a', deps => ARRAY['step_b']);
--- This redefines a cycle that the system may refuse or produce an error for.
-```
-
-**Key Issue**
-
-- Cycles or misordered “root steps” cause the entire flow to fail at definition time.
-- Breaking big flows into smaller subflows might help, but a large monolithic flow can create confusion.
-
----
-
-### 1.4 Data Bloating in JSON Fields
-
-**Description**  
-Each step’s output is stored in JSON (or JSONB). Large outputs can bloat database storage, degrade performance, or hamper indexing.
-
-**Example**
-
-```ts
-// Step that might store large data:
-.step(
-  { slug: 'fetch_big_report' },
-  async (input) => {
-    // Potentially returns massive JSON
-    return { bigReport: "..." };
-  }
-)
-```
-
-**Key Issue**
-
-- Over time, you accumulate large JSON payloads in the DB.
-- Hard to index or query partial fields from large JSON unless carefully planned (e.g., domain-specific partial storage).
-
----
-
-### 1.5 Versioning Flows in Production
-
-**Description**  
-Making changes or versioning existing flows while runs are in progress can cause confusion. Some “in-flight” runs rely on old definitions; new runs rely on updated definitions.
-
-**Example**
-
-```sql
--- Attempting to rename a step slug in production:
-UPDATE steps
-  SET slug = 'fetch_data_v2'
-  WHERE slug = 'fetch_data_v1'
-    AND flow_slug = 'my_flow';
--- Could break or orphan old runs that reference the old slug.
-```
-
-**Key Issue**
-
-- If you rename or remove steps in the DB definition while older runs still reference them, you can end up with incomplete or stuck runs.
-
----
-
-### 1.6 Mismatch Between TypeScript DSL and SQL State
-
-**Description**  
-The TypeScript DSL might produce a shape that is out of sync with the actual SQL definitions. If you forget a migration step or run local code that doesn’t match the DB schema, you’ll see inconsistent runs.
-
-**Example**
-
-```ts
-// TypeScript code defines "step_C" dependsOn: ["step_B"],
-// but in the DB, "step_C" might not have that dependency due to a missed migration script.
-.step(
-  { slug: 'step_C', dependsOn: ['step_B'] },
-  async (input) => { /* ... */ }
-);
-```
-
-**Key Issue**
-
-- Accidental drift between the DSL and the actual DB.
-- Forces you to maintain either an automated migration step or a manual alignment process.
-
----
-
-### 1.7 Potential Overhead for Simple Flows
-
-**Description**  
-If your flows are simple “do step A, then step B,” the overhead of designing them as DAG definitions, SQL objects, and DSL code might be too heavy.
-
-**Example**
-
-```ts
-// Overkill for a simple 2-step sequence:
-const SimpleFlow = new Flow<{ name: string }>()
-  .step({ slug: 'stepA' }, async (input) => `Hello, ${input.run.name}`)
-  .step({ slug: 'stepB', dependsOn: ['stepA'] }, async (input) =>
-    console.log(input.stepA)
-  );
-
-// Could just do it in a direct function call with fewer abstractions.
-```
-
-**Key Issue**
-
-- Overhead of setting up the DSL, storing definitions, polling tasks, etc., might not justify the complexity for trivial flows.
-
----
-
-### 1.8 Debugging Failed Steps with Partial Data
-
-**Description**  
-When steps fail, partial data might remain. If you rely on the DSL to pass data from step to step, you might not see intermediate debugging logs in your usual console.
-
-**Example**
-
-```ts
-// A step that frequently fails, but only partial data is in DB logs
-.step(
-  { slug: 'fragile_api_call' },
-  async (input) => {
-    // If this fails mid-call, you might not see intermediate states
-    // unless you explicitly log them or store them in partial steps.
-  }
-);
-```
-
-**Key Issue**
-
-- Observability might require a more robust approach than simply storing JSON outputs.
-- Hard to step through (like a debugger) inside a distributed worker scenario.
-
----
-
-### 1.9 Handling Secrets and Sensitive Data
-
-**Description**  
-When outputs or inputs contain sensitive info (API tokens, user data, secrets), storing them as plain JSON in the DB can be risky or require special encryption logic.
-
-**Example**
-
-```ts
-// Suppose input has tokens:
-type Input = { userToken: string; url: string };
-const SecureFlow = new Flow<Input>().step(
-  { slug: 'use_token' },
-  async (payload) => {
-    // We might inadvertently store userToken in final logs or outputs
-    return { usedToken: payload.run.userToken };
-  }
-);
-```
-
-**Key Issue**
-
-- If logs or DB entries are compromised, secrets might leak.
-- PBKDF or encryption at rest might help, but is not always built-in.
-
----
-
-### 1.10 Managing Parallelism vs. Resource Limits
-
-**Description**  
-The DSL can trigger multiple parallel steps. On a small Supabase plan, you might overrun resource or concurrency limits if you scale flows too aggressively.
-
-**Example**
-
-```ts
-// Each root step can spawn 20 parallel sub-steps:
-const BigParallelFlow = new Flow<{ items: number[] }>()
-  .step({ slug: 'split' }, async (payload) => payload.run.items)
-  // ...
-  .step(
-    { slug: 'fan_out_step_19', dependsOn: ['split'] },
-    (async) /* ... */ => {}
-  )
-  .step(
-    { slug: 'fan_out_step_20', dependsOn: ['split'] },
-    (async) /* ... */ => {}
-  );
-```
-
-**Key Issue**
-
-- The database queue might produce 20 parallel tasks, each worker might open heavy connections or saturate your CPU.
-- On limited orchestrations, it can lead to unexpected slowdowns, partial failures, or queue time expansions.
-
----
-
-## 2. Positive and Promising Use Cases
-
-Despite the above pitfalls, there are numerous scenarios where this DSL approach is very powerful.
-
-### 2.1 Clear Separation of Concerns
-
-**Description**  
-You keep the “what to run and in what order” in the database, while your actual logic is in edge functions or TypeScript. This separation clarifies “flow structure” vs. “execution code.”
-
-**Example**
-
-```ts
-// Flow definition is purely describing steps:
-const MyFlow = new Flow<{ userId: string }>()
-  .step({ slug: 'get_user_data' }, async (p) =>
-    fetchUserProfileFromDb(p.run.userId)
-  )
-  .step({ slug: 'process_data', dependsOn: ['get_user_data'] }, async (p) =>
-    doSomeComputation(p.get_user_data)
-  );
-
-// The DB has a stable definition: flow = MyFlow.
-```
-
-**Key Benefit**
-
-- DB queries remain straightforward; step orchestration logic is separate.
-- A single source of truth in the DB for which steps exist and how they connect.
-
----
-
-### 2.2 Automatic Retry and Exact-Once Semantics
-
-**Description**  
-Combining the queue (e.g., pgmq) with step state transitions ensures at-most-once or exactly-once semantics. The DSL automatically handles retries, skipping steps if already completed, etc.
-
-**Example**
-
-```ts
-const RetryFlow = new Flow<{ value: number }>().step(
-  { slug: 'division_step', maxAttempts: 3, baseDelay: 2 },
-  async (p) => {
-    if (p.run.value === 0) throw new Error("Can't divide by zero!");
-    return { result: 100 / p.run.value };
-  }
-);
-```
-
-**Key Benefit**
-
-- Automatic exponential backoff after failures reduces developer overhead.
-- The system handles concurrency, so you only write your logic.
-
----
-
-### 2.3 Parallel Steps for Faster Execution
-
-**Description**  
-DAG shape means you can define steps that run in parallel, drastically speeding up certain pipelines. The DSL approach makes parallel definitions simple.
-
-**Example**
-
-```ts
-// Summarize and sentiment-analyze in parallel:
-const AnalyzeParallel = new Flow<{ text: string }>()
-  .step({ slug: 'sentiment' }, async (p) => analyzeSentiment(p.run.text))
-  .step({ slug: 'summary' }, async (p) => summarizeWithLLM(p.run.text))
-  .step({ slug: 'combine', dependsOn: ['sentiment', 'summary'] }, async (p) => {
-    return {
-      combined: `Summary: ${p.summary}, Sentiment: ${p.sentiment}`,
-    };
-  });
-```
-
-**Key Benefit**
-
-- You define concurrency simply by listing dependencies.
-- No extra code to manage parallel tasks or threads; the worker pool does the heavy lifting.
-
----
-
-### 2.4 Observability with Step-Level State
-
-**Description**  
-Each step’s status and output is tracked in PostgreSQL. This makes it easy to query run history, see which steps failed, and retrieve the partial outputs.
-
-**Example**
-
-```sql
--- Querying the DB table "step_states" for a given run:
-SELECT step_slug, status, output
-FROM step_states
-WHERE run_id = '<some_run_id>';
-```
-
-**Key Benefit**
-
-- Simple SQL queries to see current or historical status.
-- Easy to integrate with dashboards or a BI tool (on Supabase).
-
----
-
-### 2.5 Transactional Consistency
-
-**Description**  
-If a step depends on multiple prior steps, you can be sure the outputs from those steps are fully consistent in the DB. The system handles commits atomically.
-
-**Example**
-
-```sql
--- When edge function calls complete_task, it either commits everything or rolls back:
-SELECT pgflow.complete_task(
-  run_id => 'some_run_id',
-  step_slug => 'fetch_data',
-  output => '{"data":"..."}'
-);
-/*
-   This ensures next steps only see the output if transaction committed.
-   Otherwise they retry tasks.
-*/
-```
-
-**Key Benefit**
-
-- Minimizes risk of partial writes or half-updated states in concurrency scenarios.
-- Straightforward approach to ensuring data integrity.
-
----
-
-### 2.6 Scalable Worker Model
-
-**Description**  
-The worker poll mechanism can scale horizontally: multiple edge function instances can poll the same queue. The flow engine keeps track of tasks in flight using PG concurrency primitives.
-
-**Key Benefit**
-
-- Works well in serverless environments (e.g., multiple Supabase edge functions).
-- Good for “burst” scenarios: you can scale up workers quickly if many tasks come in at once.
-
----
-
-### 2.7 Automatic Type Inference and IntelliSense
-
-**Description**  
-The TypeScript DSL can infer the shape of inputs, outputs, and step dependencies, giving you better auto-completion in your IDE.
-
-**Example**
-
-```ts
-const InferredFlow = new Flow<{ name: string }>()
-  .step({ slug: 'greet' }, (p) => `Hello, ${p.run.name}`)
-  .step({ slug: 'uppercase', dependsOn: ['greet'] }, (p) =>
-    p.greet.toUpperCase()
-  );
-
-// 'p.greet' is inferred as string from the previous step.
-// No manual type definitions for 'uppercase' needed other than the initial flow input.
-```
-
-**Key Benefit**
-
-- Less boilerplate.
-- TypeScript ensures you don’t access data that wasn’t provided by a dependency.
-
----
-
-### 2.8 Flexible “Fan-In” or “Fan-Out”
-
-**Description**  
-Declarative steps let you do fan-out to process multiple items in parallel and then fan-in to combine outputs. This is especially handy for data transformations or chunked tasks.
-
-**Example**
-
-```ts
-// Suppose you have an array of items:
-.step({ slug: 'split_items' }, async (p) => p.run.items)
-.step(
-  { slug: 'process_each_item', dependsOn: ['split_items'], fanOut: true },
-  async (p, chunkIndex) => handleSingleItem(p.split_items[chunkIndex])
-)
-.step(
-  { slug: 'combine_results', dependsOn: ['process_each_item'] },
-  async (p) => aggregateAll(p.process_each_item)
-);
-```
-
-**Key Benefit**
-
-- DSL approach can automatically generate multiple sub-tasks for each item.
-- You get aggregated results at the “fan-in” step.
-
----
-
-### 2.9 Persistent State for Long-Lived Processes
-
-**Description**  
-Workflows can remain “running” for days or weeks, especially if you have delayed steps or wait states. The DB acts as a single source of truth.
-
-**Example**
-
-```sql
--- A step that triggers a follow-up after 7 days:
-SELECT pgflow.add_step(
-  flow_slug => 'user_onboarding',
-  step_slug => 'follow_up_email',
-  deps => ARRAY['initial_welcome'],
-  delay_seconds => 604800
-);
-```
-
-**Key Benefit**
-
-- No need for a separate “cron” to wait for 7 days. The engine can hold the step until the delay passes.
-- You can pick up the flow run exactly where it left off after downtime/deployments.
-
----
-
-### 2.10 Single Platform (Supabase + Postgres + Edge Functions)
-
-**Description**  
-You don’t need Kubernetes, external queues, or heavy infra. You run SQL plus minimal worker code in Edge Functions. This lowers the operational overhead.
-
-**Key Benefit**
-
-- Ideal for small/medium apps that want a serverless or single cloud.
-- Easy to manage in the Supabase ecosystem (auth, db, functions).
-
----
-
-## Concluding Thoughts
-
-Using a database-centric DSL to manage flows has powerful advantages—particularly the robust state tracking, type inference, and parallel step orchestration. However, it also introduces potential complexities:
-
-- **Workflow versioning** can be tough.
-- **Data bloat** and **sensitive data** might need special handling.
-- **Excessive overhead** for simple use cases.
-
-For teams building multi-step, parallelizable, or long-running processes in a single Postgres-based environment (e.g., Supabase), this approach can unify data and orchestration tracking under one roof. It’s a compelling solution, as long as you plan carefully around potential pitfalls such as concurrency, conditional branching complexity, and output storage.

package/brainstorming/dsl/dsl-critique.md

@@ -1,41 +0,0 @@
-# Critique of Current Flow DSL Implementation
-
-Below are several key points where the current Flow DSL could be improved or clarified. These focus on non-trivial issues that aren’t already covered by TypeScript’s type system or foreign key constraints enforcing topological ordering.
-
-## 1. Step Name Collisions
-Currently, when adding steps using the `step()` method, there is no mechanism for detecting or preventing duplicate step slugs within the same Flow instance. Accidentally reusing an existing slug would silently overwrite the corresponding `stepDefinitions` entry, leading to unexpected behavior.
-
-### Possible Improvements
-- Throw an error if a step is defined more than once with the same slug.
-- Provide a helper or check to ensure slugs remain unique in a given Flow.
-
-## 2. Cross-Flow Step References
-There is no hard check to ensure that a `dependsOn` array only references steps within the same Flow. If a user mistakenly references a slug that belongs to a different Flow or that doesn’t exist in the current Flow, they’d only discover the error at execution time (e.g., missing output in a payload).
-
-### Possible Improvements
-- Validate each `dependsOn` slug to ensure it exists in the current `stepDefinitions`.
-- Fail early (i.e., throw) at call time when an invalid dependency is specified.
-
-## 3. Missing Flow-Level Default Fallback
-Although each step can define overrides for `maxAttempts`, `baseDelay`, and `timeout`, there is no fallback logic at runtime if a step-level option is undefined. For instance, if a user sets a `baseDelay` at the flow level only, a step requiring that delay but not providing its own override should inherit the flow’s value rather than defaulting to `undefined`.
-
-### Possible Improvements
-- When building `stepOptionsStore`, merge step-level overrides with flow-level defaults.
-- Expose a utility method (e.g., `getEffectiveOptions(stepSlug)`) that returns the merged set of options for each step.
-
-## 4. Runtime Validation of Handler Output Shape
-Since handlers return arbitrary JSON, there is little reassurance at runtime if a handler’s actual returned shape mismatches the inferred type. The TypeScript compiler can help, but cases of dynamic data or incorrectly typed library calls could pass the compiler but break at runtime.
-
-### Possible Improvements
-- Provide an optional debug or development mode that checks a handler’s returned data against the inferred type structure (e.g., via JSON schema or similar approach).
-
-## 5. Handling of Late-Added Steps
-The current structure re-creates a new `Flow` each time `.step()` is called. If a user tries to add new steps after certain operations or references have been made (such as storing an existing Flow instance for execution), it may cause confusion or partial definitions. The final shape of the Flow is only fully determined after all `.step()` calls, but nothing prevents referencing incomplete or out-of-order definitions in the meantime.
-
-### Possible Improvements
-- Enforce a “building phase” vs. “execution phase” distinction—only allow `.step()` declarations before you attempt to retrieve or execute the Flow definitions.
-- Consider a final “freeze” step to lock the DSL so no further definitions can be added.
-
----
-
-By addressing these points, the Flow DSL will be more robust, more fault-tolerant, and less prone to subtle errors that can arise in real-world usage.