@pgflow/core 0.0.0-array-map-steps-cd94242a-20251008042921 → 0.0.0-control-plane-a947cb71-20251121164755

package/README.md

@@ -47,6 +47,17 @@ This package focuses on:
 
 The actual execution of workflow tasks is handled by the [Edge Worker](../edge-worker/README.md), which calls back to the SQL Core to acknowledge task completion or failure.
 
+## Requirements
+
+> [!IMPORTANT]
+> **pgmq Version Requirement** (since v0.8.0)
+>
+> pgflow v0.8.0 and later requires **pgmq 1.5.0 or higher**. This version of pgflow will NOT work with pgmq 1.4.x or earlier.
+>
+> - **Supabase Cloud**: Recent versions include pgmq 1.5.0+ by default
+> - **Self-hosted**: You must upgrade pgmq to version 1.5.0+ before upgrading pgflow
+> - **Version Check**: Run `SELECT extversion FROM pg_extension WHERE extname = 'pgmq';` to verify your pgmq version
+
 ## Key Features
 
 - **Declarative Workflows**: Define flows and steps via SQL tables
@@ -140,6 +151,7 @@ SELECT pgflow.add_step(
 #### Root Map vs Dependent Map
 
 **Root Map Steps** process the flow's input array directly:
+
 ```sql
 -- Root map: no dependencies, processes flow input
 SELECT pgflow.add_step(
@@ -156,6 +168,7 @@ SELECT pgflow.start_flow(
 ```
 
 **Dependent Map Steps** process another step's array output:
+
 ```sql
 -- Dependent map: processes the array from 'fetch_items'
 SELECT pgflow.add_step(
@@ -169,6 +182,7 @@ SELECT pgflow.add_step(
 #### Edge Cases and Special Behaviors
 
 1. **Empty Array Cascade**: When a map step receives an empty array (`[]`):
+
    - The SQL core completes it immediately without creating tasks
    - The completed map step outputs an empty array
    - Any dependent map steps also receive empty arrays and complete immediately
@@ -184,12 +198,14 @@ SELECT pgflow.add_step(
 #### Implementation Details
 
 Map steps utilize several database fields for state management:
+
 - `initial_tasks`: Number of tasks to create (NULL until array size is known)
 - `remaining_tasks`: Tracks incomplete tasks for the step
 - `task_index`: Identifies which array element each task processes
 - `step_type`: Column value 'map' triggers map behavior
 
 The aggregation process ensures:
+
 - **Order Preservation**: Task outputs maintain array element ordering
 - **NULL Handling**: NULL outputs are included in the aggregated array
 - **Atomicity**: Aggregation occurs within the same transaction as task completion
@@ -262,8 +278,9 @@ When a workflow starts:
 The Edge Worker uses a two-phase approach to retrieve and start tasks:
 
 **Phase 1 - Reserve Messages:**
+
 ```sql
-SELECT * FROM pgflow.read_with_poll(
+SELECT * FROM pgmq.read_with_poll(
   queue_name => 'analyze_website',
   vt => 60, -- visibility timeout in seconds
   qty => 5  -- maximum number of messages to fetch
@@ -271,6 +288,7 @@ SELECT * FROM pgflow.read_with_poll(
 ```
 
 **Phase 2 - Start Tasks:**
+
 ```sql
 SELECT * FROM pgflow.start_tasks(
   flow_slug => 'analyze_website',
@@ -379,6 +397,7 @@ Timeouts are enforced by setting the message visibility timeout to the step's ti
 The SQL Core is the DAG orchestration engine that handles dependency resolution, step state management, and task spawning. However, workflows are defined using the TypeScript Flow DSL, which compiles user intent into the SQL primitives that populate the definition tables (`flows`, `steps`, `deps`).
 
 See the [@pgflow/dsl package](../dsl/README.md) for complete documentation on:
+
 - Expressing workflows with type-safe method chaining
 - Step types (`.step()`, `.array()`, `.map()`)
 - Compilation to SQL migrations
@@ -441,6 +460,7 @@ Map step tasks receive a fundamentally different input structure than single ste
 ```
 
 This means:
+
 - Map handlers process individual elements in isolation
 - Map handlers cannot access the original flow input (`run`)
 - Map handlers cannot access other dependencies
@@ -456,8 +476,10 @@ When a step depends on a map step, it receives the aggregated array output:
 
 // A step depending on 'process_users' receives:
 {
-  "run": { /* original flow input */ },
-  "process_users": [{"name": "Alice"}, {"name": "Bob"}]  // Full array
+  "run": {
+    /* original flow input */
+  },
+  "process_users": [{ "name": "Alice" }, { "name": "Bob" }] // Full array
 }
 ```
 

package/dist/ATLAS.md

@@ -16,7 +16,7 @@ The database must be empty, but contain everything needed for the schemas to app
 We need a configured [PGMQ](https://github.com/tembo-io/pgmq) extension, which Atlas does not support
 in their dev images.
 
-That's why this setup relies on a custom built image `jumski/postgres-15-pgmq:latest`.
+That's why this setup relies on a custom built image `jumski/postgres-17-pgmq:latest`.
 
 Inspect `Dockerfile.atlas` to see how it is built.
 

package/dist/CHANGELOG.md

@@ -1,29 +1,118 @@
 # @pgflow/core
 
-## 0.0.0-array-map-steps-cd94242a-20251008042921
+## 0.0.0-control-plane-a947cb71-20251121164755
+
+### Patch Changes
+
+- @pgflow/dsl@0.0.0-control-plane-a947cb71-20251121164755
+
+## 0.8.0
+
+### Minor Changes
+
+- 7380237: BREAKING CHANGE: pgflow 0.8.0 requires pgmq 1.5.0+, PostgreSQL 17, and Supabase CLI 2.34.3+
+
+  This version modernizes infrastructure dependencies and will NOT work with pgmq 1.4.x or earlier. The migration includes a compatibility check that aborts with a clear error message if requirements are not met.
+
+  **Requirements:**
+
+  - pgmq 1.5.0 or higher (previously supported 1.4.x)
+  - PostgreSQL 17 (from 15)
+  - Supabase CLI 2.34.3 or higher (includes pgmq 1.5.0+)
+
+  **For Supabase users:** Upgrade your Supabase CLI to 2.34.3+ which includes pgmq 1.5.0 by default.
+
+  **For self-hosted users:** Upgrade pgmq to 1.5.0+ and PostgreSQL to 17 before upgrading pgflow.
+
+  **If you cannot upgrade immediately:** Stay on pgflow 0.7.x until your infrastructure is ready. The migration safety check ensures you cannot accidentally upgrade to an incompatible version.
+
+### Patch Changes
+
+- @pgflow/dsl@0.8.0
+
+## 0.7.3
+
+### Patch Changes
+
+- @pgflow/dsl@0.7.3
+
+## 0.7.2
+
+### Patch Changes
+
+- c22a1e5: Fix missing realtime broadcasts for step:started and step:completed events
+
+  **Critical bug fix:** Clients were not receiving `step:started` events when steps transitioned to Started status, and `step:completed` events for empty map steps and cascade completions were also missing.
+
+  **Root cause:** PostgreSQL query optimizer was eliminating CTEs containing `realtime.send()` calls because they were not referenced by subsequent operations or the final RETURN statement.
+
+  **Solution:** Moved `realtime.send()` calls directly into RETURNING clauses of UPDATE statements, ensuring they execute atomically with state changes and cannot be optimized away.
+
+  **Changes:**
+
+  - `start_ready_steps()`: Broadcasts step:started and step:completed events in RETURNING clauses
+  - `cascade_complete_taskless_steps()`: Broadcasts step:completed events atomically with cascade completion
+  - `complete_task()`: Added PERFORM statements for run:failed and step:failed broadcasts
+  - Client: Added `applySnapshot()` methods to FlowRun and FlowStep for proper initial state hydration without event emission
+  - @pgflow/dsl@0.7.2
+
+## 0.7.1
+
+### Patch Changes
+
+- a71b371: Fix installation failures on new Supabase projects by removing pgmq version pin.
+
+  Supabase upgraded to pgmq 1.5.1 in Postgres 17.6.1.016+ (https://github.com/supabase/postgres/pull/1668), but pgflow was pinned to 1.4.4, causing "extension has no installation script" errors on fresh instances.
+
+  Only affects new projects - existing installations are unaffected and require no action.
+
+  Thanks to @kallebysantos for reporting this issue!
+
+  - @pgflow/dsl@0.7.1
+
+## 0.7.0
 
 ### Minor Changes
 
 - 524db03: Add map step type infrastructure in SQL core
 
-  ## 🚨🚨🚨 CRITICAL MIGRATION WARNING 🚨🚨🚨
+  ⚠️ **This migration includes automatic data migration**
+
+  The migration will automatically update existing `step_states` rows to satisfy new constraints. This should complete without issues due to strict check constraints enforced in previous versions.
 
-  **THIS MIGRATION REQUIRES MANUAL DATA UPDATE BEFORE DEPLOYMENT!**
+  💡 **Recommended: Verify before deploying to production**
 
-  The migration adds a new constraint `remaining_tasks_state_consistency` that will **FAIL ON EXISTING DATA** if not handled properly.
+  If you have existing production data and want to verify the migration will succeed cleanly, run this **read-only check query** (does not modify data) in **Supabase Studio** against your **production database**:
 
-  ### Required Data Migration:
+  1. Open Supabase Studio → SQL Editor
+  2. Copy contents of `pkgs/core/queries/PRE_MIGRATION_CHECK_20251006073122.sql`
+  3. Execute against your production database (not local dev!)
+  4. Review results
 
-  Before applying this migration to any environment with existing data, you MUST include:
+  **Expected output for successful migration:**
 
-  ```sql
-  -- CRITICAL: Update existing step_states to satisfy new constraint
-  UPDATE pgflow.step_states
-  SET remaining_tasks = NULL
-  WHERE status = 'created';
   ```
+  type                       | identifier                | details
+  ---------------------------|---------------------------|------------------------------------------
+  DATA_BACKFILL_STARTED      | run=def67890 step=process | initial_tasks will be set to 1 (...)
+  DATA_BACKFILL_COMPLETED    | Found 100 completed steps | initial_tasks will be set to 1 (...)
+  INFO_SUMMARY               | total_step_states=114     | created=0 started=1 completed=113 failed=0
+  ```
+
+  **Interpretation:**
+
+  - ✅ Only `DATA_BACKFILL_*` and `INFO_SUMMARY` rows? **Safe to migrate**
+  - ⚠️ These are expected data migrations handled automatically by the migration
+  - 🆘 Unexpected rows or errors? Copy output and share on Discord for help
+
+  📝 **Note:** This check identifies data that needs migration but does not modify anything. Only useful for production databases with existing runs.
+
+  **Automatic data updates:**
+
+  - Sets `initial_tasks = 1` for all existing steps (correct for pre-map-step schema)
+  - Sets `remaining_tasks = NULL` for 'created' status steps (new semantics)
 
-  **Without this update, the migration WILL FAIL in production!** The new constraint requires that `remaining_tasks` can only be set when `status != 'created'`.
+  No manual intervention required.
 
   ***
 
@@ -66,7 +155,7 @@
 
 - Updated dependencies [524db03]
 - Updated dependencies [524db03]
-  - @pgflow/dsl@0.0.0-array-map-steps-cd94242a-20251008042921
+  - @pgflow/dsl@0.7.0
 
 ## 0.6.1
 

package/dist/PgflowSqlClient.js

@@ -9,7 +9,7 @@ export class PgflowSqlClient {
     async readMessages(queueName, visibilityTimeout, batchSize, maxPollSeconds = 5, pollIntervalMs = 200) {
         return await this.sql `
       SELECT *
-      FROM pgflow.read_with_poll(
+      FROM pgmq.read_with_poll(
         queue_name => ${queueName},
         vt => ${visibilityTimeout},
         qty => ${batchSize},

package/dist/README.md

@@ -47,6 +47,17 @@ This package focuses on:
 
 The actual execution of workflow tasks is handled by the [Edge Worker](../edge-worker/README.md), which calls back to the SQL Core to acknowledge task completion or failure.
 
+## Requirements
+
+> [!IMPORTANT]
+> **pgmq Version Requirement** (since v0.8.0)
+>
+> pgflow v0.8.0 and later requires **pgmq 1.5.0 or higher**. This version of pgflow will NOT work with pgmq 1.4.x or earlier.
+>
+> - **Supabase Cloud**: Recent versions include pgmq 1.5.0+ by default
+> - **Self-hosted**: You must upgrade pgmq to version 1.5.0+ before upgrading pgflow
+> - **Version Check**: Run `SELECT extversion FROM pg_extension WHERE extname = 'pgmq';` to verify your pgmq version
+
 ## Key Features
 
 - **Declarative Workflows**: Define flows and steps via SQL tables
@@ -140,6 +151,7 @@ SELECT pgflow.add_step(
 #### Root Map vs Dependent Map
 
 **Root Map Steps** process the flow's input array directly:
+
 ```sql
 -- Root map: no dependencies, processes flow input
 SELECT pgflow.add_step(
@@ -156,6 +168,7 @@ SELECT pgflow.start_flow(
 ```
 
 **Dependent Map Steps** process another step's array output:
+
 ```sql
 -- Dependent map: processes the array from 'fetch_items'
 SELECT pgflow.add_step(
@@ -169,6 +182,7 @@ SELECT pgflow.add_step(
 #### Edge Cases and Special Behaviors
 
 1. **Empty Array Cascade**: When a map step receives an empty array (`[]`):
+
    - The SQL core completes it immediately without creating tasks
    - The completed map step outputs an empty array
    - Any dependent map steps also receive empty arrays and complete immediately
@@ -184,12 +198,14 @@ SELECT pgflow.add_step(
 #### Implementation Details
 
 Map steps utilize several database fields for state management:
+
 - `initial_tasks`: Number of tasks to create (NULL until array size is known)
 - `remaining_tasks`: Tracks incomplete tasks for the step
 - `task_index`: Identifies which array element each task processes
 - `step_type`: Column value 'map' triggers map behavior
 
 The aggregation process ensures:
+
 - **Order Preservation**: Task outputs maintain array element ordering
 - **NULL Handling**: NULL outputs are included in the aggregated array
 - **Atomicity**: Aggregation occurs within the same transaction as task completion
@@ -262,8 +278,9 @@ When a workflow starts:
 The Edge Worker uses a two-phase approach to retrieve and start tasks:
 
 **Phase 1 - Reserve Messages:**
+
 ```sql
-SELECT * FROM pgflow.read_with_poll(
+SELECT * FROM pgmq.read_with_poll(
   queue_name => 'analyze_website',
   vt => 60, -- visibility timeout in seconds
   qty => 5  -- maximum number of messages to fetch
@@ -271,6 +288,7 @@ SELECT * FROM pgflow.read_with_poll(
 ```
 
 **Phase 2 - Start Tasks:**
+
 ```sql
 SELECT * FROM pgflow.start_tasks(
   flow_slug => 'analyze_website',
@@ -379,6 +397,7 @@ Timeouts are enforced by setting the message visibility timeout to the step's ti
 The SQL Core is the DAG orchestration engine that handles dependency resolution, step state management, and task spawning. However, workflows are defined using the TypeScript Flow DSL, which compiles user intent into the SQL primitives that populate the definition tables (`flows`, `steps`, `deps`).
 
 See the [@pgflow/dsl package](../dsl/README.md) for complete documentation on:
+
 - Expressing workflows with type-safe method chaining
 - Step types (`.step()`, `.array()`, `.map()`)
 - Compilation to SQL migrations
@@ -441,6 +460,7 @@ Map step tasks receive a fundamentally different input structure than single ste
 ```
 
 This means:
+
 - Map handlers process individual elements in isolation
 - Map handlers cannot access the original flow input (`run`)
 - Map handlers cannot access other dependencies
@@ -456,8 +476,10 @@ When a step depends on a map step, it receives the aggregated array output:
 
 // A step depending on 'process_users' receives:
 {
-  "run": { /* original flow input */ },
-  "process_users": [{"name": "Alice"}, {"name": "Bob"}]  // Full array
+  "run": {
+    /* original flow input */
+  },
+  "process_users": [{ "name": "Alice" }, { "name": "Bob" }] // Full array
 }
 ```