@pgflow/core 0.0.0-array-map-steps-cd94242a-20251008042921 → 0.0.0-condition-4354fcb6-20260108134756

package/README.md

@@ -5,10 +5,6 @@ PostgreSQL-native workflow engine for defining, managing, and tracking DAG-based
 > [!NOTE]
 > This project and all its components are licensed under [Apache 2.0](./LICENSE) license.
 
-> [!WARNING]
-> This project uses [Atlas](https://atlasgo.io/docs) to manage the schemas and migrations.
-> See [ATLAS.md](ATLAS.md) for more details.
-
 ## Table of Contents
 
 - [Overview](#overview)
@@ -47,6 +43,16 @@ This package focuses on:
 
 The actual execution of workflow tasks is handled by the [Edge Worker](../edge-worker/README.md), which calls back to the SQL Core to acknowledge task completion or failure.
 
+## Requirements
+
+> [!IMPORTANT] > **pgmq Version Requirement** (since v0.8.0)
+>
+> pgflow v0.8.0 and later requires **pgmq 1.5.0 or higher**. This version of pgflow will NOT work with pgmq 1.4.x or earlier.
+>
+> - **Supabase Cloud**: Recent versions include pgmq 1.5.0+ by default
+> - **Self-hosted**: You must upgrade pgmq to version 1.5.0+ before upgrading pgflow
+> - **Version Check**: Run `SELECT extversion FROM pg_extension WHERE extname = 'pgmq';` to verify your pgmq version
+
 ## Key Features
 
 - **Declarative Workflows**: Define flows and steps via SQL tables
@@ -140,6 +146,7 @@ SELECT pgflow.add_step(
 #### Root Map vs Dependent Map
 
 **Root Map Steps** process the flow's input array directly:
+
 ```sql
 -- Root map: no dependencies, processes flow input
 SELECT pgflow.add_step(
@@ -156,6 +163,7 @@ SELECT pgflow.start_flow(
 ```
 
 **Dependent Map Steps** process another step's array output:
+
 ```sql
 -- Dependent map: processes the array from 'fetch_items'
 SELECT pgflow.add_step(
@@ -169,6 +177,7 @@ SELECT pgflow.add_step(
 #### Edge Cases and Special Behaviors
 
 1. **Empty Array Cascade**: When a map step receives an empty array (`[]`):
+
    - The SQL core completes it immediately without creating tasks
    - The completed map step outputs an empty array
    - Any dependent map steps also receive empty arrays and complete immediately
@@ -184,12 +193,14 @@ SELECT pgflow.add_step(
 #### Implementation Details
 
 Map steps utilize several database fields for state management:
+
 - `initial_tasks`: Number of tasks to create (NULL until array size is known)
 - `remaining_tasks`: Tracks incomplete tasks for the step
 - `task_index`: Identifies which array element each task processes
 - `step_type`: Column value 'map' triggers map behavior
 
 The aggregation process ensures:
+
 - **Order Preservation**: Task outputs maintain array element ordering
 - **NULL Handling**: NULL outputs are included in the aggregated array
 - **Atomicity**: Aggregation occurs within the same transaction as task completion
@@ -262,8 +273,9 @@ When a workflow starts:
 The Edge Worker uses a two-phase approach to retrieve and start tasks:
 
 **Phase 1 - Reserve Messages:**
+
 ```sql
-SELECT * FROM pgflow.read_with_poll(
+SELECT * FROM pgmq.read_with_poll(
   queue_name => 'analyze_website',
   vt => 60, -- visibility timeout in seconds
   qty => 5  -- maximum number of messages to fetch
@@ -271,6 +283,7 @@ SELECT * FROM pgflow.read_with_poll(
 ```
 
 **Phase 2 - Start Tasks:**
+
 ```sql
 SELECT * FROM pgflow.start_tasks(
   flow_slug => 'analyze_website',
@@ -379,6 +392,7 @@ Timeouts are enforced by setting the message visibility timeout to the step's ti
 The SQL Core is the DAG orchestration engine that handles dependency resolution, step state management, and task spawning. However, workflows are defined using the TypeScript Flow DSL, which compiles user intent into the SQL primitives that populate the definition tables (`flows`, `steps`, `deps`).
 
 See the [@pgflow/dsl package](../dsl/README.md) for complete documentation on:
+
 - Expressing workflows with type-safe method chaining
 - Step types (`.step()`, `.array()`, `.map()`)
 - Compilation to SQL migrations
@@ -441,6 +455,7 @@ Map step tasks receive a fundamentally different input structure than single ste
 ```
 
 This means:
+
 - Map handlers process individual elements in isolation
 - Map handlers cannot access the original flow input (`run`)
 - Map handlers cannot access other dependencies
@@ -456,8 +471,10 @@ When a step depends on a map step, it receives the aggregated array output:
 
 // A step depending on 'process_users' receives:
 {
-  "run": { /* original flow input */ },
-  "process_users": [{"name": "Alice"}, {"name": "Bob"}]  // Full array
+  "run": {
+    /* original flow input */
+  },
+  "process_users": [{ "name": "Alice" }, { "name": "Bob" }] // Full array
 }
 ```
 

package/dist/CHANGELOG.md

@@ -1,29 +1,203 @@
 # @pgflow/core
 
-## 0.0.0-array-map-steps-cd94242a-20251008042921
+## 0.0.0-condition-4354fcb6-20260108134756
+
+### Patch Changes
+
+- c25ab9f: Add whenFailed option for error handling after retries exhausted (fail, skip, skip-cascade)
+- 74202cd: Add skip infrastructure schema for conditional execution - new columns (condition_pattern, when_unmet, when_failed, skip_reason, skipped_at), 'skipped' status, and cascade_skip_steps function
+- Updated dependencies [c25ab9f]
+  - @pgflow/dsl@0.0.0-condition-4354fcb6-20260108134756
+
+## 0.13.1
+
+### Patch Changes
+
+- 199fbe1: Note: Version 0.13.0 was yanked due to broken local environment detection. This release (0.13.1) includes both the fix and the features from 0.13.0.
+
+  - Fix local environment detection to use SUPABASE_URL instead of API keys
+  - Add step output storage optimization for 2x faster Map chains (outputs now stored in step_states.output instead of aggregated on-demand)
+  - @pgflow/dsl@0.13.1
+
+## 0.13.0
+
+### Minor Changes
+
+- 05738ed: Performance: Store step outputs atomically for 2x faster downstream task startup
+
+  Step outputs are now stored in step_states.output when steps complete, eliminating expensive aggregation queries. Benchmarks show 2.17x improvement for Map->Map chains. Includes data migration to backfill existing completed steps.
+
+### Patch Changes
+
+- @pgflow/dsl@0.13.0
+
+## 0.12.0
+
+### Minor Changes
+
+- 37402eb: BREAKING: Asymmetric handler signatures - remove `run` key from step inputs
+
+  - Root steps: `(flowInput, ctx) => ...` - flow input directly as first param
+  - Dependent steps: `(deps, ctx) => ...` - only dependency outputs as first param
+  - Access flow input in dependent steps via `await ctx.flowInput` (async/lazy-loaded)
+  - Lazy loading prevents data duplication for map steps processing large arrays
+  - Enables functional composition and simplifies types for future subflows
+
+### Patch Changes
+
+- Updated dependencies [37402eb]
+- Updated dependencies [5dc5cfc]
+  - @pgflow/dsl@0.12.0
+
+## 0.11.0
+
+### Minor Changes
+
+- 0cb5500: New compilation config with allowDataLoss option for rapid iteration platforms. Breaking: ensureCompiledOnStartup removed in favor of compilation option.
+
+### Patch Changes
+
+- @pgflow/dsl@0.11.0
+
+## 0.10.0
+
+### Minor Changes
+
+- 90276ce: Add automatic worker restart via `ensure_workers()` cron job that keeps edge functions running. Add `worker_functions` table for tracking registered edge functions and their health status. Add `stopped_at` column to workers table for graceful shutdown detection. Integrate `trackWorkerFunction` and `markWorkerStopped` into edge worker lifecycle for automatic registration and shutdown signaling.
+
+### Patch Changes
+
+- 0b84bb0: Add automatic flow compilation at worker startup. Workers now call ensure_flow_compiled to verify flows are up-to-date. In development, mismatched flows are recompiled automatically. In production, mismatches cause errors. Use ensureCompiledOnStartup: false to opt-out.
+- Updated dependencies [0b84bb0]
+  - @pgflow/dsl@0.10.0
+
+## 0.9.1
+
+### Patch Changes
+
+- Updated dependencies [992a86b]
+  - @pgflow/dsl@0.9.1
+
+## 0.9.0
+
+### Patch Changes
+
+- @pgflow/dsl@0.9.0
+
+## 0.8.1
+
+### Patch Changes
+
+- f1d3c32: Fix incorrect Supabase CLI version requirement from 2.34.3 to 2.50.3. CLI 2.50.3 is the first version to include pgmq 1.5.0+, which is required for pgflow 0.8.0+.
+  - @pgflow/dsl@0.8.1
+
+## 0.8.0
+
+### Minor Changes
+
+- 7380237: BREAKING CHANGE: pgflow 0.8.0 requires pgmq 1.5.0+, PostgreSQL 17, and Supabase CLI 2.50.3+
+
+  This version modernizes infrastructure dependencies and will NOT work with pgmq 1.4.x or earlier. The migration includes a compatibility check that aborts with a clear error message if requirements are not met.
+
+  **Requirements:**
+
+  - pgmq 1.5.0 or higher (previously supported 1.4.x)
+  - PostgreSQL 17 (from 15)
+  - Supabase CLI 2.50.3 or higher (includes pgmq 1.5.0+)
+
+  **For Supabase users:** Upgrade your Supabase CLI to 2.50.3+ which includes pgmq 1.5.0 by default.
+
+  **For self-hosted users:** Upgrade pgmq to 1.5.0+ and PostgreSQL to 17 before upgrading pgflow.
+
+  **If you cannot upgrade immediately:** Stay on pgflow 0.7.x until your infrastructure is ready. The migration safety check ensures you cannot accidentally upgrade to an incompatible version.
+
+### Patch Changes
+
+- @pgflow/dsl@0.8.0
+
+## 0.7.3
+
+### Patch Changes
+
+- @pgflow/dsl@0.7.3
+
+## 0.7.2
+
+### Patch Changes
+
+- c22a1e5: Fix missing realtime broadcasts for step:started and step:completed events
+
+  **Critical bug fix:** Clients were not receiving `step:started` events when steps transitioned to Started status, and `step:completed` events for empty map steps and cascade completions were also missing.
+
+  **Root cause:** PostgreSQL query optimizer was eliminating CTEs containing `realtime.send()` calls because they were not referenced by subsequent operations or the final RETURN statement.
+
+  **Solution:** Moved `realtime.send()` calls directly into RETURNING clauses of UPDATE statements, ensuring they execute atomically with state changes and cannot be optimized away.
+
+  **Changes:**
+
+  - `start_ready_steps()`: Broadcasts step:started and step:completed events in RETURNING clauses
+  - `cascade_complete_taskless_steps()`: Broadcasts step:completed events atomically with cascade completion
+  - `complete_task()`: Added PERFORM statements for run:failed and step:failed broadcasts
+  - Client: Added `applySnapshot()` methods to FlowRun and FlowStep for proper initial state hydration without event emission
+  - @pgflow/dsl@0.7.2
+
+## 0.7.1
+
+### Patch Changes
+
+- a71b371: Fix installation failures on new Supabase projects by removing pgmq version pin.
+
+  Supabase upgraded to pgmq 1.5.1 in Postgres 17.6.1.016+ (https://github.com/supabase/postgres/pull/1668), but pgflow was pinned to 1.4.4, causing "extension has no installation script" errors on fresh instances.
+
+  Only affects new projects - existing installations are unaffected and require no action.
+
+  Thanks to @kallebysantos for reporting this issue!
+
+  - @pgflow/dsl@0.7.1
+
+## 0.7.0
 
 ### Minor Changes
 
 - 524db03: Add map step type infrastructure in SQL core
 
-  ## 🚨🚨🚨 CRITICAL MIGRATION WARNING 🚨🚨🚨
+  ⚠️ **This migration includes automatic data migration**
+
+  The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions.
 
-  **THIS MIGRATION REQUIRES MANUAL DATA UPDATE BEFORE DEPLOYMENT!**
+  💡 **Recommended: Verify before deploying to production**
 
-  The migration adds a new constraint `remaining_tasks_state_consistency` that will **FAIL ON EXISTING DATA** if not handled properly.
+  If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**:
 
-  ### Required Data Migration:
+  1. Open Supabase Studio → SQL Editor
+  2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql`
+  3. Execute against your production database (not local dev!)
+  4. Review results
 
-  Before applying this migration to any environment with existing data, you MUST include:
+  **Expected output for successful migration:**
 
-  ```sql
-  -- CRITICAL: Update existing step_states to satisfy new constraint
-  UPDATE pgflow.step_states
-  SET remaining_tasks = NULL
-  WHERE status = 'created';
   ```
+  type                       | identifier                | details
+  ---------------------------|---------------------------|------------------------------------------
+  DATA_BACKFILL_STARTED      | run=def67890 step=process | initial_tasks will be set to 1 (...)
+  DATA_BACKFILL_COMPLETED    | Found 100 completed steps | initial_tasks will be set to 1 (...)
+  INFO_SUMMARY               | total_step_states=114     | created=0 started=1 completed=113 failed=0
+  ```
+
+  **Interpretation:**
+
+  - ✅ Only `DATA_BACKFILL_*` and `INFO_SUMMARY` rows? **Safe to migrate**
+  - ⚠️ These are expected data migrations handled automatically by the migration
+  - 🆘 Unexpected rows or errors? Copy output and share on Discord for help
+
+  📝 **Note:** This check identifies data that needs migration but does not modify anything. Only useful for production databases with existing runs.
+
+  **Automatic data updates:**
+
+  - Sets `initial_tasks = 1` for all existing steps (correct for pre-map-step schema)
+  - Sets `remaining_tasks = NULL` for 'created' status steps (new semantics)
 
-  **Without this update, the migration WILL FAIL in production!** The new constraint requires that `remaining_tasks` can only be set when `status != 'created'`.
+  No manual intervention required.
 
   ***
 
@@ -66,7 +240,7 @@
 
 - Updated dependencies [524db03]
 - Updated dependencies [524db03]
-  - @pgflow/dsl@0.0.0-array-map-steps-cd94242a-20251008042921
+  - @pgflow/dsl@0.7.0
 
 ## 0.6.1
 

package/dist/PgflowSqlClient.js

@@ -9,7 +9,7 @@ export class PgflowSqlClient {
     async readMessages(queueName, visibilityTimeout, batchSize, maxPollSeconds = 5, pollIntervalMs = 200) {
         return await this.sql `
       SELECT *
-      FROM pgflow.read_with_poll(
+      FROM pgmq.read_with_poll(
         queue_name => ${queueName},
         vt => ${visibilityTimeout},
         qty => ${batchSize},

package/dist/README.md

@@ -5,10 +5,6 @@ PostgreSQL-native workflow engine for defining, managing, and tracking DAG-based
 > [!NOTE]
 > This project and all its components are licensed under [Apache 2.0](./LICENSE) license.
 
-> [!WARNING]
-> This project uses [Atlas](https://atlasgo.io/docs) to manage the schemas and migrations.
-> See [ATLAS.md](ATLAS.md) for more details.
-
 ## Table of Contents
 
 - [Overview](#overview)
@@ -47,6 +43,16 @@ This package focuses on:
 
 The actual execution of workflow tasks is handled by the [Edge Worker](../edge-worker/README.md), which calls back to the SQL Core to acknowledge task completion or failure.
 
+## Requirements
+
+> [!IMPORTANT] > **pgmq Version Requirement** (since v0.8.0)
+>
+> pgflow v0.8.0 and later requires **pgmq 1.5.0 or higher**. This version of pgflow will NOT work with pgmq 1.4.x or earlier.
+>
+> - **Supabase Cloud**: Recent versions include pgmq 1.5.0+ by default
+> - **Self-hosted**: You must upgrade pgmq to version 1.5.0+ before upgrading pgflow
+> - **Version Check**: Run `SELECT extversion FROM pg_extension WHERE extname = 'pgmq';` to verify your pgmq version
+
 ## Key Features
 
 - **Declarative Workflows**: Define flows and steps via SQL tables
@@ -140,6 +146,7 @@ SELECT pgflow.add_step(
 #### Root Map vs Dependent Map
 
 **Root Map Steps** process the flow's input array directly:
+
 ```sql
 -- Root map: no dependencies, processes flow input
 SELECT pgflow.add_step(
@@ -156,6 +163,7 @@ SELECT pgflow.start_flow(
 ```
 
 **Dependent Map Steps** process another step's array output:
+
 ```sql
 -- Dependent map: processes the array from 'fetch_items'
 SELECT pgflow.add_step(
@@ -169,6 +177,7 @@ SELECT pgflow.add_step(
 #### Edge Cases and Special Behaviors
 
 1. **Empty Array Cascade**: When a map step receives an empty array (`[]`):
+
    - The SQL core completes it immediately without creating tasks
    - The completed map step outputs an empty array
    - Any dependent map steps also receive empty arrays and complete immediately
@@ -184,12 +193,14 @@ SELECT pgflow.add_step(
 #### Implementation Details
 
 Map steps utilize several database fields for state management:
+
 - `initial_tasks`: Number of tasks to create (NULL until array size is known)
 - `remaining_tasks`: Tracks incomplete tasks for the step
 - `task_index`: Identifies which array element each task processes
 - `step_type`: Column value 'map' triggers map behavior
 
 The aggregation process ensures:
+
 - **Order Preservation**: Task outputs maintain array element ordering
 - **NULL Handling**: NULL outputs are included in the aggregated array
 - **Atomicity**: Aggregation occurs within the same transaction as task completion
@@ -262,8 +273,9 @@ When a workflow starts:
 The Edge Worker uses a two-phase approach to retrieve and start tasks:
 
 **Phase 1 - Reserve Messages:**
+
 ```sql
-SELECT * FROM pgflow.read_with_poll(
+SELECT * FROM pgmq.read_with_poll(
   queue_name => 'analyze_website',
   vt => 60, -- visibility timeout in seconds
   qty => 5  -- maximum number of messages to fetch
@@ -271,6 +283,7 @@ SELECT * FROM pgflow.read_with_poll(
 ```
 
 **Phase 2 - Start Tasks:**
+
 ```sql
 SELECT * FROM pgflow.start_tasks(
   flow_slug => 'analyze_website',
@@ -379,6 +392,7 @@ Timeouts are enforced by setting the message visibility timeout to the step's ti
 The SQL Core is the DAG orchestration engine that handles dependency resolution, step state management, and task spawning. However, workflows are defined using the TypeScript Flow DSL, which compiles user intent into the SQL primitives that populate the definition tables (`flows`, `steps`, `deps`).
 
 See the [@pgflow/dsl package](../dsl/README.md) for complete documentation on:
+
 - Expressing workflows with type-safe method chaining
 - Step types (`.step()`, `.array()`, `.map()`)
 - Compilation to SQL migrations
@@ -441,6 +455,7 @@ Map step tasks receive a fundamentally different input structure than single ste
 ```
 
 This means:
+
 - Map handlers process individual elements in isolation
 - Map handlers cannot access the original flow input (`run`)
 - Map handlers cannot access other dependencies
@@ -456,8 +471,10 @@ When a step depends on a map step, it receives the aggregated array output:
 
 // A step depending on 'process_users' receives:
 {
-  "run": { /* original flow input */ },
-  "process_users": [{"name": "Alice"}, {"name": "Bob"}]  // Full array
+  "run": {
+    /* original flow input */
+  },
+  "process_users": [{ "name": "Alice" }, { "name": "Bob" }] // Full array
 }
 ```