@pgflow/dsl 0.0.5 → 0.0.6

package/brainstorming/fanouts/fanout-subflows-flattened-vs-subruns.md

@@ -1,213 +0,0 @@
-# Exploring "Fanout Subflows" in a Workflow System
-
-This document explores the notion of “fanout” in a workflow DSL—where an array of items at runtime spawns multiple parallel subflows (or steps), merging their outputs when each is complete. We will examine:
-
-1. **Practical Utility**: Is fanout really needed or overkill?  
-2. **Flattened vs. Sub-Run Implementation**: Can we feasibly do it if we flatten all steps, or do we need genuine sub-runs?  
-3. **Generic Implementations**: Is there a “basic” mechanism that could unify both fanout tasks and fanout subflows?  
-4. **Implications of Using Sub-Runs**: If we pick sub-runs, how does that affect the entire subflow abstraction, and can the `.subflow()` idea still work? What about returning a `SubflowObject` in place of a callback?
-
-Throughout, keep in mind that the array length for the fanout is **unknown at compile time** and only becomes clear once the flow is actually running.
-
----
-
-## 1. Is a Fanout Feature Really Useful?
-
-**Fanout** means taking an array (of N items) and producing N parallel tasks or subflows. Each parallel execution handles one array element, and when all parallel executions finish, their results merge back into a single array.
-
-1. **Massively Parallel Processing**  
-   - Whenever you have a list of items—say user IDs, URLs to scrape, or documents to process—fanout is an efficient way to parallelize.
-   - This pattern is quite common: ingest multiple items, run the same logic on each item, then aggregate the outputs.
-
-2. **Order-Sensitive Merging**  
-   - A typical requirement is that the final aggregated array lines up with the original input order.
-   - This is a non-trivial detail: if you run items in parallel, you must ensure results remain aligned with the input indices.
-
-3. **Use Cases**  
-   - Data pipelines (e.g., transform a batch of logs).  
-   - Multi-tenant or multi-customer flows.  
-   - Bulk computations (like AI inference or image processing) that must eventually return combined results.
-
-4. **Conclusion**  
-   - Fanout is extremely common in workflow systems, so having built-in support is more than a nicety; it’s borderline essential for serious batch parallelism.  
-
-**Hence, yes, it’s quite useful and definitely worth adding**—provided the implementation overhead isn’t overwhelming.
-
----
-
-## 2. Feasibility of Implementing Fanout in a Flattened Approach
-
-The **flattened** approach means your workflow engine eventually sees a single “flat” DAG of steps. If subflows are also flattened, we might imagine a scenario like:
-
-- At compile time, the DSL transforms each subflow’s steps into part of the parent flow, prefixing their slugs to avoid collisions.
-- For a fanout, you might need “N copies” of the subflow’s steps, one copy per array element.
-
-### 2.1 The Key Complication
-
-If the array length is unknown at compile time, flattening is not straightforward:
-
-- You cannot create a fixed number of steps at design-time, because you don’t know how many items will appear.  
-- You’d have to dynamically insert steps into the DAG at runtime—meaning the “compiled” flow is incomplete until you see the array.  
-- This dynamic insertion is not trivial: it could require altering the database schema or adding new step rows “on the fly,” which changes your flow’s shape in ways the engine may not expect.
-
-### 2.2 Potential Workarounds
-
-1. **Maximum Bound**  
-   - You could artificially set a maximum array length (say 100 items) and flatten at compile time, ignoring or skipping the extra steps if you have fewer items.  
-   - This is messy—wasteful for smaller arrays and possibly insufficient for bigger arrays.
-
-2. **Dynamic Step Creation**  
-   - Your engine might “clone” or “spawn” steps at runtime.  
-   - This means the engine must be built to handle newly inserted steps mid-run. That’s a lot of complexity.
-
-3. **Single Step with Internal Logic**  
-   - Instead of representing each item with a distinct DB record, you might rely on a single “fanout” step that manually processes each item.  
-   - But then you lose the built-in concurrency or separate step status tracking for each item. You also lose the ability to fail or retry them individually in a clean way.
-
-### 2.3 Conclusion on Flattening
-
-Flattening a “true fanout subflow” (with unknown N) can be **very challenging**. If your engine is not already designed for dynamic DAG expansion at runtime, the implementation overhead can be huge.  
-
----
-
-## 3. A “Basic Mechanism” for Both Fanout Tasks and Subflows
-
-An ideal solution might unify both:
-
-- **Fanout tasks**: e.g., run the same function on each array element.  
-- **Fanout subflows**: e.g., run the same subflow on each array element.
-
-### 3.1 High-Level Idea
-
-**Single “Fanout Step or Subflow” with Pluggable Logic**  
-- You define some DSL construct, like `flow.fanout(sourceArraySelector, subflowDefinition)`.  
-- At runtime, the engine sees that this step is special: it spawns multiple parallel runs of the “subflowDefinition,” each with one item from the array.  
-- The engine merges the results into an array output that is aligned to the original array indices.
-
-### 3.2 Implementation Sketch
-
-1. **Fanout Step in DB**  
-   - A step type: `step_type='fanout'`. This step references the subflow or the single-step logic that must be repeated.  
-   - The engine knows how to “expand” or “execute” that fanout step by launching multiple “child tasks or sub-subflow runs” behind the scenes.
-
-2. **Common Core for Both Single-Task and Subflow**  
-   - The engine’s internal logic for fanout doesn’t care if it’s a one-step subflow or a multi-step subflow. The key is “Take an array, apply something repeatedly, gather results.”
-
-3. **Challenges**  
-   - The engine must track partial completions, partial failures, and how to handle indexing.  
-   - If you want fully independent subflow concurrency, you still need a robust concurrency approach.  
-
-With a carefully-designed “fanout step,” both single-step logic and multi-step subflows can be handled uniformly.
-
----
-
-## 4. Going with Sub-Runs: Impact and .subflow() Abstraction
-
-Because flattening an unknown-size fanout is so complex, many workflow engines prefer **true sub-runs**: each array item spawns a child run that references the same subflow “definition,” just with different input. Then the parent flow:
-
-1. Waits until all child runs complete.  
-2. Assembles their outputs in order.
-
-### 4.1 Pros of Sub-Runs
-
-- **No Need for Dynamically Expanding the DAG**  
-  Each subflow is just a new run record in your DB. The “parent run” references it, but you are not injecting new steps into the parent’s DAG.  
-- **Engine Reuse**  
-  Each sub-run is scheduled and executed exactly like a normal run. No specialized logic to create “extra steps” on the parent side.  
-- **Scalability**  
-  Potentially you can spin up thousands of sub-runs in parallel if your system allows.
-
-### 4.2 Impact on Subflow/Branching Abstraction
-
-If everything is a separate run:
-
-- **.subflow()** Implementation:  
-  - Instead of flattening subflow definitions right away, calling `.subflow()` at runtime triggers a new run of a subflow ID.  
-  - That subflow ID is a separate Flow definition in your DB (i.e., a fully versioned flow).  
-
-- **Passing a `SubflowObject` Instead of a Callback**  
-  - In a flattened approach, `.subflow((sub) => { ... })` typically merges or “inline compiles” the sub-steps.  
-  - With sub-runs, you might instead pass or retrieve a `SubflowObject` that points to a stand-alone flow. The DSL might look like `flow.fanout( items, SubflowObject )`, meaning: spawn the subflow described by `SubflowObject` for each item.  
-  - The parent’s DSL code becomes:  
-    ```ts
-    flow.fanout(
-      (input) => input.run.items,
-      SubflowObject("mySubflow_v2") // or something akin to that
-    );
-    ```
-  - You can still preserve a DSL style that calls `.subflow()` and returns that `SubflowObject`, but ultimately it’s referencing a separate flow ID. The difference is it won’t flatten—rather, it spawns sub-runs with that flow ID and merges results when done.
-
-### 4.3 Downsides
-
-- **Separate Metaruns**  
-  - You now must query multiple run entries to see the overall state.  
-  - The UI must reflect that 20 child runs exist for the parent.  
-
-- **Infrastructure Complexity**  
-  - If your workflow system was simpler with only single-run DAGs, you now add an entire dimension of parent–child run orchestration.  
-
----
-
-### Can We Still Use `.subflow()` for Implementation?
-
-Yes—**.subflow()** can still exist in the DSL. But the meaning changes from “directly inline these steps into my parent flow” to “store a reference to a distinct flow or subflow object, and at runtime, spawn runs for it.”  
-
-1. **At Code Definition**  
-   ```ts
-   const MySubflow = defineFlow("mySubflow", (sub) => {
-     // sub steps
-   });
-
-   const ParentFlow = defineFlow("parentFlow", (flow) => {
-     flow.fanout(
-       (payload) => payload.items,
-       MySubflow
-     );
-   });
-   ```
-   - The engine sees that `MySubflow` is a separate flow definition.  
-   - The parent flow references it for a fanout step.  
-
-2. **At Runtime**  
-   - The parent run is created.  
-   - When we get to the fanout step, we read the array length from parent’s data.  
-   - For each item, we spawn a child run referencing “mySubflow.”  
-   - We collect outputs once all child runs complete.
-
-**That approach elegantly supports unknown array lengths,** because each child's subflow process is just a new run that can be started dynamically.
-
----
-
-## Summary of Key Points
-
-1. **Yes, Fanout is Often Needed**  
-   - It’s a common scenario: parallelize processing for an arbitrary-length array.
-
-2. **Flattened Approach is Possible… but Hard**  
-   - If array length is not known until runtime, you’d have to dynamically add steps. Implementing that is not trivial.
-
-3. **A Basic “Fanout Step”**  
-   - One path is to implement a special step type in your engine that orchestrates sub-tasks or subflows in parallel and merges their outputs.  
-   - This “fanout step” can handle both single-step logic (just a function) or multi-step logic (a subflow).
-
-4. **Sub-Runs Are Often Simpler at Scale**  
-   - Instead of flattening, spawn child runs for each array element.  
-   - The DSL can still have `.subflow()`, but under the hood, it references a standalone subflow definition, invoked repeatedly.
-
-5. **Passing a `SubflowObject` vs. Callback**  
-   - In a flattening DSL, `.subflow(callback)` merges your sub-steps into the main flow.  
-   - In a sub-run DSL, `.subflow()` likely returns a “flow reference” or `SubflowObject`. Then fanout or the parent flow just calls the subflow by ID.  
-
----
-
-## Conclusion
-
-Implementing array-based “fanout” with unknown sizing is a powerful feature but can be **extremely difficult** if you rely on a purely flattened DAG. Sub-runs (where each array element gets its own run) can drastically simplify concurrency and dynamic expansion—at the cost of extra run objects in your database.
-
-If your prime concern is the complexity of “expanding” the DAG at runtime, **sub-runs** are the more straightforward path. You can still provide a user-friendly DSL like `.fanout([...items], MySubflow)` which behind the scenes launches multiple runs and merges outputs in order. Your `.subflow()` concept can remain intact as a convenient way to define or reference re-usable subflows, but it will revolve around a separate flow entity rather than inlining steps. 
-
-In short:  
-- **Yes, fanout is important.**  
-- **Flattening with unknown array sizes can be done but is painful.**  
-- **A “unified fanout approach”** is possible if you define a dedicated “fanout step” concept that can handle both single-step tasks and multi-step subflows.  
-- **Sub-runs** are often the more **scalable** and **developer-friendly** solution, preserving a neat `.subflow()` abstraction without needing to do dynamic DAG expansions at runtime.

package/brainstorming/fanouts/fanouts-task-index.md

@@ -1,150 +0,0 @@
-# Revisiting the Fanout Challenge: Dynamic Number of Items and Subflows
-
-> **Context**  
-> In a workflow engine, we often have a _fanout_ scenario: a step receives an array of unknown length at runtime and must process each element in parallel. The engine should keep track of each item’s partial failure/success. How do we represent these **many parallel tasks** in the database while also allowing partial failures, clean skipping logic, etc.?
-
-Below we explore:
-
-1. **One-Step-With-Multiple-Tasks** using a `task_index` strategy.  
-2. Why that’s problematic for partial failure or skipping.  
-3. **Best alternative** ideas that balance ease of implementation, schema simplicity, flexibility, and readability.
-
----
-
-## 1. One-Step-with-Multiple-Tasks (the `task_index` approach)
-
-### 1.1 Rationale
-A direct way to handle fanout is to say:
-- We store a single `step_state` row in `step_states`.
-- We spawn multiple parallel “tasks” for that step, each with a unique `task_index` (0, 1, 2, …).
-- Each `task` processes one item from the input array.
-
-This means:
-
-- The “fanout” is visible at the **task** level, not the **step** level.  
-- All tasks share the same step definition and the same step state row (`step_states`), but we differentiate them with `task_index`.
-
-### 1.2 Pros
-
-1. **Minimal Schema Changes**  
-   You don’t need extra columns or special “fanout” logic in `step_states`. The only difference is that you spawn multiple tasks (`fanout_count = array_length`) for that single step.
-2. **Simplicity**  
-   At first glance, it seems simpler in code: same step, many tasks.
-
-### 1.3 Cons & Caveats
-
-1. **Partial Failures Are Hard**  
-   If any single task fails, the entire step is set to `failed`. You can’t have a scenario where _item #2 fails_ but _item #1 and item #3 succeed_ with that step eventually “completing.”  
-   - The step-level status is a single field: `'created'`, `'started'`, `'completed'`, `'failed'`, or `'skipped'`.  
-   - If you truly need a partial failure, you can’t represent it with a single step-level status.
-
-2. **Skipping Individual Items**  
-   “Skipping” normally marks the step as `'skipped'`. If you wanted to skip certain items while processing others, the one-step approach doesn’t handle that gracefully. You either skip the entire step or not at all.
-
-3. **Inconsistent Aggregation**  
-   When all tasks finish, you’re forced to unify outputs at the task level. The engine sees only one `step_state.output` for the entire step. Handling an array of partial results or partial failures is clumsy.
-
-4. **Limited Future Extensibility**  
-   If you want to add sub-steps that each item must run, or if item #2 needs a different path, you can’t nest further logic under a single step row. You would be forced to do contrived hacks or spawn new pseudo-steps.
-
---- 
-
-## 2. Why Partial Success or Skipping Matters
-
-Real-world scenarios often involve:
-
-- **One item is invalid** → skip or fail that item alone, but continue others.  
-- **Condition-based skipping** analysis (e.g., `runIf: { isActive: true }`) that might apply differently to each array element.  
-- **Chained subflows** for each element (item-based sub-processing).
-
-If you rely on a single step row with a single status, you lose the ability to differentiate item-level outcomes. In practice, an entire run might get stuck at “failed” even if only 1% of the items encountered a problem.
-
----
-
-## 3. A Better Alternative: “Fanout Subflow” or “Multi-Step Fanout”
-
-A more flexible (and only slightly more complex) approach is to create **multiple distinct step-level records** for each item **or** wrap them into subflows dynamically. That is:
-
-1. **At runtime**, when the engine sees an array, it spawns a “fanout group” of steps (or subflows).  
-2. Each item receives its own “step_state” (or subflow instance).  
-3. Each one can independently succeed, fail, or skip.  
-4. An optional aggregator step can unify the outputs afterward.
-
-### 3.1 Minimal Implementation Sketch
-
-1. **Introduce a “fanout_config”** or a “step_type = 'fanout'” column in the `steps` table.  
-   - This signals that the step can produce multiple child step entries at runtime.  
-2. **When the engine sees** that a `fanout` step is ready:
-   1. Retrieve the array (e.g. `payload.items`)  
-   2. Dynamically insert N new `step_states` (or “subflow” entries), one per item.  
-   3. Mark each child step as `'created'`, then `'started'` if dependencies are satisfied.  
-   4. For partial failures, only that child’s row is `'failed'`. The rest can complete.
-
-3. **Aggregate** the child outputs in a single aggregator step, or store them as an array in the parent. The aggregator is automatically triggered when all children reach a final state.
-
-### 3.2 Pros
-1. **Accurate Partial Failure**  
-   Each item can fail, skip, or succeed. These statuses are reflected individually in `step_states`.
-2. **Clean Skipping**  
-   If some items are filtered out, they can be set to `'skipped'`. Others proceed normally.
-3. **Extensible to Nested Logic**  
-   Each “item step” can run further sub-steps or even be a subflow if needed, which is impossible with a single step row.
-4. **Future Condition Support**  
-   You can combine fanout with `runIf` or `runUnless` at the item level if desired (e.g., skip certain items automatically).
-
-### 3.3 Cons
-1. **Slightly More Schema or Engine Logic**  
-   You must store “fanout” info in the `steps` table or store the child steps as a new row pointing to the parent. 
-2. **Potential Table Growth**  
-   For large arrays, you get many new step-level rows. But that’s precisely what you need if you want item-level tracking.
-
----
-
-## 4. The “Subflow for Each Item” Variation
-
-An even more powerful approach is to treat each item as its own subflow instance:
-
-1. **Subflow Root Step**: Mark the parent step as `step_type = 'subflow'` **plus** `fanout = true`.  
-2. **At runtime**, if the array has length `m`, spawn `m` subflows. Each subflow is flattened into your existing `steps` and `step_states` with an index or a new slug prefix.  
-3. Each subflow can contain multiple steps, can fail or skip individually.  
-4. A final aggregator collects all subflow outputs into an array or a structured object.
-
-**Pro**: Perfect if each item requires multiple steps of its own processing.  
-**Con**: More table rows, more overhead. But you gain full power with minimal conceptual duplication.
-
----
-
-## 5. Conclusion: Recommended Path for Balanced Implementation
-
-Below is a proposed approach that balances:
-
-- **Ease of Implementation**  
-- **Future Flexibility**  
-- **Minimal Schema Changes**  
-- **Clear Partial Failure Model**  
-
-1. **Introduce a “fanout” concept** (could be a `boolean` or an object, e.g. `{ path: 'someArrayPath' }`) in your step definition.  
-2. **Extend the database** to handle multiple “child step_states” or subflows for each item. Minimal changes might be:  
-   - A `parent_step_id` field in `step_states` or a small “fanout_metadata” structure.  
-   - Or store subflows in the same `steps` table with an additional `index` if needed.  
-3. **Each child** has its own row in `step_states`, allowing **independent** `'failed'`, `'skipped'`, `'completed'`.  
-4. **An aggregator** or the parent step can unify the final outcome (collate arrays, detect partial fails, etc.).
-
-**Why this is best**:
-- **Flexibility**: You can add more steps for each item (like sub-steps) later without forced kludges.  
-- **Readability**: The DB statuses match intuitive runtime reality (each item is its own “mini-run” or “mini-step”).  
-- **Simplicity of Conditions**: If you want to skip certain items entirely, you can do so on each item’s row. No complex shared step state needed.  
-- **Minimal Overhead**: You only add extra rows (or subflow expansions) when the array is discovered, which is exactly when you need them.
-
----
-
-### Final Takeaway
-
-Although using a single step with multiple tasks (identified by `task_index`) seems tempting at first, it quickly becomes unwieldy for partial failures and item-level skipping. A more robust solution is to model each item’s work as its own step or subflow record. This approach:
-
-- Preserves partial success/failure.  
-- Integrates cleanly with skipping logic.  
-- Ensures future fanout + multi-step item processing is possible.  
-- Remains relatively simple in terms of schema changes (one or two extra columns plus some runtime logic to spawn item-level states).
-
-This is the path that will keep a growing fanout and subflow system both **powerful** and **maintainable** as your workflow engine evolves.

package/brainstorming/fanouts/fanouts-with-conditions-and-subflows.md

@@ -1,239 +0,0 @@
-# Analyzing the Impact of `runIf` / `runElse` and Subflows with Potential Fanout
-
-This document explores how the upcoming _conditional logic_ (`runIf` / `runUnless`) and _subflows/branching_—specifically with flattening but keeping a `parent_subflow_slug` column and `step_type = subflow`—can set the stage for a **fanout** mechanism. “Fanout” in this context means that one step or subflow would produce an array of items, and each item would spawn its own parallel step or subflow execution.
-
-We will revisit the previously discussed abstractions (flattening subflows vs. separate runs vs. a “middle ground” approach) and analyze how adding a fanout concept interacts with conditions (`runIf` / `runUnless`) in each case. The goal is to strike a balance between **MVP simplicity** and future extensibility.
-
----
-
-## 1. Introduction and Background
-
-### 1.1 Conditional Execution with `runIf` & `runUnless`
-In the near-future implementation, each step can optionally be guarded by conditions:
-
-- **`runIf`**: The step runs if this condition is **true** given the step’s input.
-- **`runUnless`**: The step runs if this condition is **false** given the step’s input.
-
-Any step for which these conditions fail is marked **`skipped`**. Transitive dependencies down the line also get skipped.
-
-### 1.2 Subflows / Branching with “Flattened Steps” & `parent_subflow_slug`
-We plan to store subflows in the same `steps` table as a “flattened” structure but with:
-
-- A `step_type` column (e.g. `subflow` vs. `normal`).  
-- A `parent_subflow_slug` so we can group subflow steps.
-- The possibility of referencing subflow-wide configuration in a “subflow step.”
-
-This approach acts as a middle ground between:
-
-1. *Fully flattening* subflow steps (just name them differently).  
-2. *Fully separate runs* for each subflow (with separate run IDs).
-
-With the “flattened-with-parent” approach, everything remains within the same run ID, but we can identify subflows as distinct groups.
-
-### 1.3 The New Fanout Concept
-A **fanout** instructs the engine to spawn multiple tasks or sub-subflows in parallel when the input is an array (or some multi-value structure). For a normal step, setting a flag like `fanout: true` (or a more structured property like `fanout: { path: "someArrayField" }`) could cause creation of N parallel tasks, one per array element. For a subflow, this approach would run the entire subflow once per array item, also in parallel. Either way, we want the step DSL to handle:
-
-- `runIf` / `runUnless` as usual.  
-- `fanout` as an **additional** or **alternative** property.
-
-The idea is to keep the design MVP-friendly: minimal friction, consistent with how we already define conditions or branching.
-
----
-
-## 2. Revisiting Flattening vs. Keeping a `parent_subflow_slug`
-### 2.1 Flatten-Only Approach
-If we were to flatten subflows entirely without references to a parent subflow entity, implementing fanout might require:
-
-- Marking each expanded step with a unique slug suffix per array item.
-- Dealing with a large blow-up in the steps table for big arrays.
-
-**Pros:**
-- Very direct: one steps table, everything is expanded in a single dimension.
-- Minimal changes to dependency resolution.
-
-**Cons:**
-- Harder to see subflow groupings in the DB or UI.
-- Potentially large expansions for big fanouts.
-
-### 2.2 Using `parent_subflow_slug` (and `step_type = subflow`)
-If each subflow is recorded as a special step row (e.g. `step_type = subflow`), then child steps are associated with the subflow’s slug. This approach is especially helpful for:
-
-- **Visibility**: We can quickly see which steps belong to subflow “X” or “Y.”
-- **Configuration**: Subflow-wide overrides (like `maxAttempts`).
-- **Dependency Handling**: The parent flow can treat the subflow as a single “virtual step,” while the engine internally expands it.
-
-**Where Fanout Fits**  
-With subflows, if we set `fanout: true` on that subflow root, the engine would spawn one parallel subflow instance per array element. Each instance has the same internal steps. This remains a “flattened” approach under the hood, but the system can group them by subflow ID or `parent_subflow_slug` plus an index for each array item.
-
----
-
-## 3. Potential Implementation of Fanout + Conditions
-
-### 3.1 Adding a `fanout` Property in Parallel to `runIf` / `runUnless`
-We could treat **fanout** as a top-level property on the step definition, e.g.:
-
-```ts
-flow.step(
-  {
-    slug: "ProcessItems",
-    dependsOn: ["FetchArray"],
-    runIf: {...},
-    runUnless: {...},
-    fanout: {
-      path: "items",      // or a boolean if there's only one array
-      concurrency?: 5     // optional concurrency limit
-    },
-  },
-  handlerFn
-);
-```
-
-**Behavior**:
-1. **Condition Check** (`runIf`, `runUnless`) – If the conditions fail, the entire step is `skipped`, no tasks are spawned.  
-2. If not skipped and `fanout` is set – The engine looks up the array at `payload.items` and spawns one “task” (or subflow instance) per element.  
-3. Each parallel instance has the item data embedded in its payload.
-
-### 3.2 Subflows with Fanout
-When “fanout: true” is specified on a subflow root step:
-
-1. The subflow root step is repeated for each array element.  
-2. All child steps of that subflow are repeated accordingly.  
-3. The final aggregated output is typically an array of subflow outputs.
-
-**Challenges**:
-- **Skipping**: If `runIf` fails on the subflow root, the entire subflow is skipped.  
-- **Mixed Condition**: Potentially each item in the array could have a different condition outcome. If the condition depends on that item, you might skip some items but not others. 
-- **Aggregating Results**: The engine needs a consistent way to merge all partial or skipped results back.  
-
----
-
-## 4. Challenges Combining Conditions + Fanout
-
-1. **Per-Item Condition Checking**  
-   For a fanout step, we might want to do `runIf` at the item level. For example, only spawn tasks for items that meet a certain criterion. This might require a “filter” concept or skipping logic per item.
-
-2. **Complexity in UI**  
-   If a subflow fans out to 1,000 items, we don’t want to see 1,000 sub-step expansions in a naive interface. We may need summarizations or chunking.
-
-3. **Transitive Dependencies**  
-   If a subflow or step fans out, its downstream steps must wait for all parallel tasks or subflows to complete (unless we have more advanced partial-latching conditions). A “runIf” might skip the entire set or skip partial subsets.
-
-4. **Performance**  
-   Large fanouts can generate many tasks. We need to ensure our scheduling and skipping logic in the database is efficient.
-
-Despite these challenges, the core conceptual structure stays the same: we store subflows with a `parent_subflow_slug` or `step_type`, use conditions to skip steps, and add a new concept for spawning parallel tasks when the input is an array.
-
----
-
-## 5. Emphasizing MVP Readiness Over Complexity
-
-A recommended MVP approach to keep things simple:
-
-1. **Allow a Boolean `fanout` or a basic `fanout: { path: string }`.**  
-   - If `fanout: true`, we assume the entire input to this step is already an array.  
-   - If `fanout: { path }`, we look up `payload[path]` at runtime.
-
-2. **Apply `runIf` / `runUnless` at the step level**  
-   - If conditions fail, skip the entire fanout step.  
-   - In the first iteration, do not provide a built-in mechanism for skipping _some_ items but running others. That can be a future enhancement.
-
-3. **In subflows**, if `fanout: true` is on the subflow’s root, fan out the entire subflow.  
-   - One subflow instance per array element.  
-   - All sub-steps reference the per-item context.  
-   - This keeps each item neatly grouped in subflow “batches,” and the parent sees it as a single entity (the root step or subflow aggregator).
-
-4. **Keep the schema additions minimal.**  
-   - Possibly store a `fanout_config jsonb` next to `run_if_condition` / `run_unless_condition`.  
-   - Use existing “skipped” statuses for steps or tasks.
-
-This incremental approach yields some immediate wins:
-
-- Parallel item processing out of the box.  
-- Condition-based skipping.  
-- Subflows remain a single “block,” which can also be fanned out if needed.
-
----
-
-## 6. Illustrating Fanout + Conditions in Each Subflow Abstraction
-
-### 6.1 Flattened Only
-- Steps for each item plus `_index` appended to the step slug.  
-- `runIf` / `runUnless` applies to the entire set; either everything gets spawned or everything is skipped.  
-- Some difficulty grouping & referencing subflow blocks if the array is large.
-
-### 6.2 Flattened With `parent_subflow_slug`
-- Mark the root subflow step with `step_type = subflow`.  
-- If `fanout = true` on that subflow step, it spawns N subflow “instances.”  
-- For each instance (i.e., each array item), the steps all share a consistent subflow group ID plus an index.  
-- The engine aggregates them at the subflow root or final subflow aggregator step.
-
-### 6.3 Full Separate Runs for Each Item
-- You’d create a new run for each array item.  
-- Overkill for an MVP. This approach quickly becomes complex, though it’s highly decoupled.
-
----
-
-## 7. Creative Ideas for Simplicity
-
-1. **Single “Fanout Step” Type**  
-   Instead of layering `fanout: true`, define `step_type: 'fanout'`. This might reduce confusion by clarifying that the step always has multiple tasks.
-
-2. **`runIf` as an Inline Filter**  
-   Instead of skipping the entire step, allow a filter expression to remove some items from the array automatically. (“If item.status != 'active', skip that item.”)
-
-3. **Hybrid Condition + Fanout Logging**  
-   For debugging, store “item-based skip logs” to show which items were processed or skipped.
-
-4. **Parallelism Limits**  
-   A simple concurrency or “max parallel” property could avoid large floods of tasks in enormous fanouts.
-
-5. **Subflow Post-Process**  
-   Let the subflow aggregator define a single “merge” function that processes all item outputs, especially for conditions that skip a subset of items.
-
----
-
-## 8. **Unexpected Alternative Solutions**  
-Below are five novel ideas that combine conditions, subflows, and fanouts in ways not previously explored in depth:
-
-1. **Conditional Fanout at the Item Level**  
-   Instead of skipping the entire step if `runIf` is false, do a partial item-level condition. For example, feed the array into a small function that returns only the items that pass `runIf`. Then only those items spawn tasks. This is a “built-in filter step.”
-
-2. **Multi-layered Subflows with Dynamic Fanout**  
-   A subflow might itself produce an array for another deeper subflow fanout. This vertical layering allows a “fan-out, gather, fan-out again” approach. At each subflow boundary, you can apply `runIf` or `runUnless` to skip entire sub-layers.
-
-3. **Fanout as a Separate “Split” Step**  
-   Instead of toggling a property on a normal step, have a dedicated “split” step that transforms a single object into multiple “emit” events. Each “emit” becomes a parallel path. Then, an optional “join” step merges them back. This more explicitly models fanout/fanin concepts.
-
-4. **Conditional Subflow Branch**  
-   Combine “branching logic” with fanout by letting a subflow pick which internal steps to spawn based on item-level conditions. The subflow might skip half its steps or sub-branches per item. This is more of a “decision tree” built within the subflow.
-
-5. **DSL-Level Macros**  
-   Introduce a DSL macro that auto-generates subflows for each item, each with its own runIf/unless. The user writes a single step, and the macro expands it behind the scenes. In effect, you write a simple block, and it compiles down to a set of parallel runIf-laden steps or subflows in concurrency form.
-
----
-
-## 9. Conclusion and Final Recommendation
-
-Summarizing the key points:
-
-1. **Conditions + Fanout Fit Well Together**  
-   - Mark a step or subflow with `runIf` / `runUnless` to decide if it should execute at all.  
-   - If not skipped, a `fanout` property spawns parallel executions for each array item.
-
-2. **Flattened Approach with a “Subflow Step”**  
-   - Recommend continuing with the plan to store subflows in the same run, but label them with `step_type = subflow` and `parent_subflow_slug`.  
-   - This approach maintains a single run while still letting the DB and UI group subflow steps.
-
-3. **MVP-Ready, Minimal Implementation**  
-   - Keep “fanout” either a simple boolean or a small config like `fanout: { path: "myArray" }`.  
-   - Handle skipping at the top-level step first (all-or-nothing). Future expansions can filter per item.
-
-4. **Adopt Simple Aggregation**  
-   - When a fanout step completes, produce an array of outputs. For subflows, gather final step outputs from each parallel instance.  
-   - The aggregator can be a single “virtual step” or the subflow root that marks when all parallel tasks/subflows finish.
-
-5. **Balance Complexity Against Clarity**  
-   - Provide enough detail to handle real-world array processing in parallel.  
-   - Avoid partial-skip complexities in an initial MVP. That can be introduced later as a “filtered fanout” or “conditional item-level skip.”
-
-By following this path, you keep the schema changes small, the runtime logic consistent with existing `skipped` status, and lay the foundation for robust parallelization scenarios. The net result is a flexible, developer-friendly approach to conditions, subflows, and fanout—without incurring unnecessary complexity upfront.

package/brainstorming/subflows/branching.ts.md

@@ -1,38 +0,0 @@
-new Flow<string>({ slug: 'analyzeWebsite' })
-  .step({ slug: 'website' }, async ({ run }) => await fetchData(run.url))
-  .branch(
-    {
-      slug: 'ifSuccess',
-      dependsOn: ['fetchData'],
-      runIf: { website: { status: 200 } },
-    },
-    (flow) =>
-      flow
-        .step({ slug: 'sentiment' }, async ({ run, fetchData }) =>
-          saveData(run.url, fetchData.body)
-        )
-        .step({ slug: 'summary' }, async ({ run, fetchData }) =>
-          saveData(run.url, fetchData.body)
-        )
-        .step(
-          { slug: 'saveToDb', dependsOn: ['summary', 'sentiment'] },
-          async (payload) =>
-            await slackHandler({
-              url: payload.run.url,
-              summary: payload.summary,
-              sentiment: payload.sentiment,
-            })
-        )
-        .step(
-          { slug: 'sendSlackMessage', dependsOn: ['summary'] },
-          async ({ saveToDb }) => await slackHandler({ message: '' })
-        )
-  )
-  .branch(
-    {
-      slug: 'ifFailure',
-      dependsOn: ['fetch'],
-      runUnless: { website: { status: 200 } },
-    },
-    (flow) => flow.step({ slug: 'notifySentry' })
-  );