@pgflow/dsl 0.0.5 → 0.0.6

package/brainstorming/versioning/breaking-and-non-breaking-flow-changes.md

@@ -1,259 +0,0 @@
-# Breaking vs. Non-Breaking Flow Changes: Implementation Guidelines
-
-When working with the pgflow orchestration framework, understanding which changes to flows require a new `flow_slug` and which ones can be safely updated in-place is crucial for maintainability and reliability. This document provides comprehensive guidance on making this distinction and implementing changes correctly.
-
-## Core Principles of Flow Versioning
-
-1. **Flows are immutable once registered**: The graph structure and dependencies should not change for an existing flow_slug.
-2. **Message compatibility**: When updating flows, consider whether existing in-flight messages would be compatible with the new flow structure.
-3. **Graph shape integrity**: Changes that alter the graph's topology require a new `flow_slug`.
-
-## Types of Changes
-
-### Breaking Changes (Require New flow_slug)
-
-Breaking changes affect the shape of the graph or the data flow between steps, making them incompatible with messages already in the queue for previous versions.
-
-| Change Type | Example | Why Breaking |
-|-------------|---------|--------------|
-| Adding a step | Adding a new validation step | Changes graph topology |
-| Removing a step | Removing a data processing step | Breaks existing dependencies |
-| Changing step dependencies | Adding/removing a dependency between steps | Alters execution order and data flow |
-| Changing step slug | Renaming `process` to `process_data` | Affects message routing |
-| Changing step return type | Changing return from `string` to `{ value: string }` | Downstream steps expect different format |
-| Changing input payload type | Adding required fields | In-flight messages lack newly required fields |
-| Reordering steps (with different dependencies) | Changing execution sequence | Alters expected data flow |
-
-### Non-Breaking Changes (Compatible with existing flow_slug)
-
-Non-breaking changes don't alter the graph's structure or affect message compatibility, making them safe to update in-place.
-
-| Change Type | Example | Why Non-Breaking |
-|-------------|---------|------------------|
-| Updating `maxAttempts` | Changing from 3 to 5 retries | Only affects execution behavior, not graph structure |
-| Modifying `baseDelay` | Changing from 5s to 10s retry delay | Only affects timing, not data flow |
-| Changing `timeout` value | Extending timeout from 30s to 60s | Only affects execution parameters |
-| Optimizing step handler logic | Improving algorithm efficiency | Internal implementation detail |
-| Adding optional fields to return type | Adding optional metadata | Doesn't break dependent steps |
-| Updating documentation/comments | Adding explanation comments | No runtime impact |
-
-## Implementation Best Practices
-
-### When to Create a New Flow Version
-
-1. **Explicit Versioning**: When making breaking changes, use either:
-   - Semantic versioning: `analyze_website_v1.0.0` → `analyze_website_v1.1.0`
-   - Date-based versioning: `analyze_website_20231001` → `analyze_website_20231115`
-
-2. **Migration Strategy**: When introducing a new flow version:
-   ```typescript
-   // New version with breaking changes
-   export const AnalyzeWebsiteV2 = new Flow<Input>({
-     slug: 'analyze_website_v2', // Note the new slug
-     maxAttempts: 3,
-   })
-   .step(/* updated step definitions */);
-   
-   // Keep old version for in-flight workflows
-   export const AnalyzeWebsiteV1 = new Flow<Input>({
-     slug: 'analyze_website_v1',
-     maxAttempts: 3,
-   })
-   .step(/* original step definitions */);
-   ```
-
-3. **Transitioning to New Versions**:
-   - Run both versions in parallel during transition
-   - Direct new runs to new version
-   - Allow old version to complete in-flight runs
-   - Consider a formal deprecation period for old versions
-
-### How to Safely Update Non-Breaking Changes
-
-For runtime options like `maxAttempts`, `baseDelay`, or `timeout`:
-
-```typescript
-// Original flow with updated non-breaking parameters
-export const AnalyzeWebsite = new Flow<Input>({
-  slug: 'analyze_website',
-  maxAttempts: 5,  // Updated from 3
-  baseDelay: 10,   // Updated from 5
-})
-.step(
-  { 
-    slug: 'sentiment', 
-    dependsOn: ['website'], 
-    timeout: 60,   // Updated from 30
-    maxAttempts: 7 // Updated from 5
-  },
-  async (input) => await analyzeSentiment(input.website.content)
-);
-```
-
-### Runtime Options Handling
-
-The implementation should separate runtime execution parameters from the flow structure definition:
-
-```typescript
-// Implementation pseudocode for flow registration system
-
-function registerFlow(flow) {
-  const existingFlow = getFlowFromDatabase(flow.flowOptions.slug);
-  
-  if (existingFlow) {
-    // Check if graph shape is identical
-    if (!areGraphShapesIdentical(existingFlow, flow)) {
-      throw new Error(
-        `Flow with slug "${flow.flowOptions.slug}" already exists with different shape. ` +
-        `Use a new flow_slug for breaking changes.`
-      );
-    }
-    
-    // For non-breaking changes, update runtime options only
-    updateFlowRuntimeOptions(
-      flow.flowOptions.slug, 
-      flow.flowOptions
-    );
-    
-    // Update step-specific runtime options
-    Object.entries(flow.getSteps()).forEach(([stepSlug, stepDefinition]) => {
-      updateStepRuntimeOptions(
-        flow.flowOptions.slug,
-        stepSlug,
-        {
-          maxAttempts: stepDefinition.maxAttempts,
-          baseDelay: stepDefinition.baseDelay,
-          timeout: stepDefinition.timeout,
-        }
-      );
-    });
-    
-    console.log(`Updated runtime options for flow "${flow.flowOptions.slug}"`);
-  } else {
-    // Register new flow with full definition
-    createNewFlow(flow);
-  }
-}
-```
-
-## Technical Implementation Considerations
-
-### 1. Flow Shape Comparison
-
-To determine if a flow structure has changed (requiring a new slug):
-
-```typescript
-function areGraphShapesIdentical(flow1, flow2) {
-  // Check if both flows have the same steps
-  const steps1 = Object.keys(flow1.getSteps()).sort();
-  const steps2 = Object.keys(flow2.getSteps()).sort();
-  
-  if (!arraysEqual(steps1, steps2)) return false;
-  
-  // Check if dependencies are identical for each step
-  for (const stepSlug of steps1) {
-    const deps1 = flow1.getSteps()[stepSlug].dependencies.sort();
-    const deps2 = flow2.getSteps()[stepSlug].dependencies.sort();
-    
-    if (!arraysEqual(deps1, deps2)) return false;
-  }
-  
-  return true;
-}
-```
-
-### 2. Database Schema Support
-
-The database schema should separate immutable flow structure from mutable runtime options:
-
-```sql
--- Immutable flow structure (never changes for same flow_slug)
-CREATE TABLE pgflow.flows (
-  flow_slug TEXT PRIMARY KEY,
-  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
-);
-
-CREATE TABLE pgflow.flow_steps (
-  flow_slug TEXT REFERENCES pgflow.flows(flow_slug),
-  step_slug TEXT NOT NULL,
-  dependencies TEXT[] NOT NULL,
-  PRIMARY KEY (flow_slug, step_slug)
-);
-
--- Mutable runtime options (can be updated)
-CREATE TABLE pgflow.flow_options (
-  flow_slug TEXT REFERENCES pgflow.flows(flow_slug),
-  max_attempts INTEGER,
-  base_delay INTEGER,
-  timeout INTEGER,
-  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  PRIMARY KEY (flow_slug)
-);
-
-CREATE TABLE pgflow.step_options (
-  flow_slug TEXT,
-  step_slug TEXT,
-  max_attempts INTEGER,
-  base_delay INTEGER,
-  timeout INTEGER,
-  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
-  PRIMARY KEY (flow_slug, step_slug),
-  FOREIGN KEY (flow_slug, step_slug) REFERENCES pgflow.flow_steps(flow_slug, step_slug)
-);
-```
-
-### 3. CLI Support for Flow Diffing
-
-Provide a CLI tool to help developers identify breaking vs. non-breaking changes:
-
-```bash
-# Example CLI usage
-pgflow diff --current-flow analyze_website --new-flow-definition ./flows/analyze_website.ts
-
-# Output:
-# Breaking changes detected:
-# - Step 'validate' added
-# - Dependencies changed for step 'saveToDb' (added: 'validate')
-# 
-# Please create a new flow_slug to implement these changes.
-```
-
-## Workflow and Testing Recommendations
-
-### Development Workflow
-
-1. **Create a flow definition in TypeScript**
-2. **Validate changes**:
-   - Use the `pgflow diff` command to check if changes are breaking
-   - For breaking changes, create a new flow version with updated slug
-3. **Register the flow**:
-   - In development, use `pgflow deploy --dev` for quick iteration
-   - In production, follow proper migration processes
-
-### Testing Strategy
-
-1. **Unit test flow definitions** to verify graph structure
-2. **Integration tests** to ensure compatible message passing between steps
-3. **Version transition tests** to verify that both old and new versions run correctly during migration
-
-### Monitoring Version Transitions
-
-When transitioning between flow versions:
-
-1. Monitor completion of in-flight executions of old versions
-2. Track success rates of new flow versions
-3. Implement alerting for flow execution failures
-4. Consider a formal deprecation period before removing old versions
-
-## Conclusion
-
-By clearly separating breaking changes (requiring new flow slugs) from non-breaking updates (compatible with existing flow slugs), you can maintain a robust and reliable workflow orchestration system while still allowing for necessary evolution.
-
-Remember these key principles:
-- Changes to graph structure or data flow are breaking changes
-- Runtime behavior adjustments are generally non-breaking
-- Always verify compatibility before updating existing flows
-- Use explicit versioning for breaking changes
-- Document your versioning strategy for team alignment
-
-Following these guidelines will help ensure that your workflow orchestration system remains stable, predictable, and maintainable as it evolves over time.

package/docs/refactor-edge-worker.md

@@ -1,146 +0,0 @@
-## Analysis of Proposed Refactoring
-
-Below is a high-level analysis of the changes needed to achieve constructor-based dependency injection for the Worker and to generalize the polling and execution logic so that different backends (like pgflow vs. pgmq) can be plugged in.
-
----
-
-### 1. Constructor-Based DI for All Core Components
-
-**Goal:** Eliminate internal construction of objects in `Worker`. Pass in each dependency (Poller, Executor, Lifecycle manager, etc.) at construction time.
-
-1. **Worker**  
-   - Currently: The `Worker` class creates and configures its own `Queue`, `ExecutionController`, `BatchProcessor`, etc. internally.  
-   - Proposed: The `Worker` constructor receives fully configured instances of:
-     - A **Poller** (the component that retrieves tasks based on queue or flow logic).  
-     - A **PayloadExecutor** (the component that processes a single task, including archiving, retrying, etc.).  
-     - A **Lifecycle** (the component that manages worker states, heartbeats, etc.).  
-
-2. **Benefits**  
-   - **Loose Coupling**: The `Worker` no longer has knowledge of how the poller or executor are implemented.  
-   - **Testability**: Each component can be tested in isolation (mocks or stubs can be injected).  
-   - **Extensibility**: Swapping out a `Poller` or `PayloadExecutor` for a different backend becomes straightforward.  
-
----
-
-### 2. Rename and Generalize `MessageExecutor` → `PayloadExecutor`
-
-**Goal:** Abstract away the notion of "messages" vs. "tasks" and unify them into a more general "payload" concept.
-
-1. **Current State**:  
-   - `MessageExecutor` is tightly coupled to the `MessageRecord` shape from pgmq.  
-   - `execute()` includes logic for archiving or retrying through the pgmq queue.  
-
-2. **Proposed Adjustments**:  
-   - Rename `MessageExecutor` → `PayloadExecutor<TPayload>`.  
-   - Accept whatever “payload” shape is needed by the backend (`MessageRecord`, `worker_task`, etc.).  
-   - A `PayloadExecutor` specifically for pgflow would handle `pgflow_worker_task` shaped payloads (i.e. handle `complete_task` or `fail_task`).  
-   - Another `PayloadExecutor` could still handle `pgmq` messages in the old style of `archive`, `retry`, etc.  
-
-3. **Benefits**:  
-   - **Worker-Agnostic**: The `Worker` loops over generic payloads, calling `executor.execute()`. The business logic of success/failure is in the executor.  
-   - **Clearer Abstractions**: `PayloadExecutor` is now dedicated to “process this item and finalize in the store (archive/fail).”  
-
----
-
-### 3. Poller Abstraction
-
-**Goal:** Replace specialized poll logic inside the worker with a clean interface. For instance, one poller for pgmq (`ReadWithPollPoller`) and another for pgflow (`poll_for_tasks`).
-
-1. **Current State**:  
-   - `BatchProcessor` or `ReadWithPollPoller` is used specifically for pgmq-based reads.  
-   - `ListenNotifyPoller` is an alternative approach but still specific to pgmq.  
-
-2. **Proposed Adjustments**:  
-   - Define a generic `Poller<TPayload>` interface with a `poll(): Promise<TPayload[]>` method.  
-   - The worker loops over `poll()` calls on an injected `Poller`.  
-   - For pgflow:
-     - Implement a `PgFlowPoller` that calls `poll_for_tasks` and returns an array of `worker_task`.  
-   - For pgmq:
-     - Keep a `PgmqPoller` that calls `readWithPoll`, returning an array of `MessageRecord`.  
-
-3. **Benefits**:  
-   - Each poller fully encapsulates how to fetch items.  
-   - The `Worker` only needs `poll()`.
-
----
-
-### 4. Worker’s Main Loop Now Becomes Simple
-
-**Goal:** The Worker core loop is only responsible for:
-
-1. Checking lifecycle signals (e.g., if `abortSignal.aborted` or the worker state is “Stopping”).  
-2. Calling `poller.poll()` to retrieve tasks.  
-3. Passing each received payload to `executor.execute()`.  
-4. Occasionally sending heartbeats.  
-5. Handling errors in a top-level “catch” to keep the loop going.  
-
-In other words, no knowledge of *how* tasks are polled or how they are completed. All logic is delegated to the injected poller and executor.
-
----
-
-### 5. Impact on Lifecycle & Queries
-
-**Goal:** Lifecycle management (start, stop, heartbeat) remains the same but is injected as a ready-to-use instance.
-
-- **`WorkerLifecycle`** can still own references to `Queries` or `queueName` for heartbeats and updating “worker started/stopped.”  
-- It’s simply injected into the Worker constructor, along with the poller/executor, so that the Worker’s only lifecycle knowledge is:
-  - “I should call `lifecycle.acknowledgeStart()` before beginning.”  
-  - “I call `lifecycle.sendHeartbeat()` on each loop iteration.”  
-  - “I call `lifecycle.acknowledgeStop()` when done.”  
-
-This keeps lifecycle logic entirely separate from poller or execution logic.
-
----
-
-### 6. Larger Changes in pgflow
-
-**Context:** pgflow’s ability to orchestrate tasks with “complete/fail” steps means it needs a specialized poller (`poll_for_tasks`) and specialized executor (`complete_task` or `fail_task`). The result is conceptually different from the pgmq “archive” approach.
-
-**Recommendation**:  
-- Implement a new `PgFlowPoller` returning typed `WorkerTask[]`.  
-- Implement a new `PgFlowExecutor` that processes each `WorkerTask`.  
-  - On success, calls `complete_task`.  
-  - On error, calls `fail_task`.  
-
-Hence, the Worker remains ignorant of whether the tasks come from pgmq or pgflow.
-
----
-
-### 7. Summary of File-Level Changes
-
-- **`Worker.ts`**  
-  - Takes constructor args like `(poller: Poller<TPayload>, executor: PayloadExecutor<TPayload>, lifecycle: WorkerLifecycle, signal?: AbortSignal, ...)`.  
-  - Eliminates internal references to `Queue`, `BatchProcessor`, or `ExecutionController`.  
-  - Main loop is simplified: poll → execute → heartbeat → loop.  
-
-- **`MessageExecutor.ts`** → **`PayloadExecutor.ts`**  
-  - Parameterize: `class PayloadExecutor<TPayload> { ... }`  
-  - Implement specialized logic in separate classes if needed (e.g., `PgmqExecutor`, `PgFlowExecutor`).  
-
-- **`BatchProcessor`, `ListenNotifyPoller`, `ReadWithPollPoller`, etc.**  
-  - Potentially reworked or consolidated under `Poller<TPayload>` interface.  
-  - If still used for pgmq, keep them internal to a `pgmq` package or “default poller” implementation.  
-
-- **`ExecutionController`**  
-  - Could be replaced or dramatically simplified: if concurrency control is still needed, it can remain a private detail inside the chosen `PayloadExecutor` or integrated into the poller. Or keep a separate concurrency-limiting layer, but construct it outside the `Worker`.  
-
----
-
-### 8. Expected Outcomes
-
-1. **Separation of Concerns**: 
-   - Worker does only “loop + lifecycle”  
-   - Poller does “fetch tasks”  
-   - Executor does “execute tasks”  
-2. **Adaptability**: 
-   - We can create specialized pollers/executors for different backends (pgmq, pgflow, etc.) without altering `Worker`.  
-3. **Better Maintenance**: 
-   - Code is easier to extend or refactor because each piece has a well-defined responsibility.  
-
----
-
-## Conclusion
-
-By introducing constructor-based dependency injection and splitting out pollers/executors as standalone interfaces, we cleanly decouple the Worker from specific queue or flow implementations. This approach allows for multiple polling strategies (e.g., pgflow vs. pgmq), varied execution (archiving vs. step completion), and makes the Worker’s main loop focused on managing its own lifecycle and health checks. 
-
-Overall, these changes should yield a more modular, maintainable, and testable codebase that is ready to accommodate new backend features (like pgflow’s “complete/fail” tasks) without breaking the existing pgmq-based functionality.

package/docs/versioning.md

@@ -1,19 +0,0 @@
-# Versioning of the flows
-
-## Agreed on immutable flow definitions (similar to Temporal)
-
--	Once a flow is uploaded to DB, it remains unchanged
-- Versioning handled through flow slugs rather than explicit version numbers
-- Users responsible for managing changes in safe, organized manner
-
-## Benefits of immutable approach
-
-- Simplifies implementation
-- Provides natural versioning cascade for subflows
-- Makes version transitions explicit and intentional
-- Avoids "half-upgraded" scenarios
-
-## Consciously decided against "latest" aliases for now
-- Could introduce complexity and unpredictable behavior
-- Users can implement their own aliasing logic if needed
-- Explicit slugs provide clarity about which version is being used

package/eslint.config.cjs

@@ -1,22 +0,0 @@
-const baseConfig = require('../../eslint.config.cjs');
-
-module.exports = [
-  ...baseConfig,
-  {
-    files: ['**/*.json'],
-    rules: {
-      '@nx/dependency-checks': [
-        'error',
-        {
-          ignoredFiles: [
-            '{projectRoot}/eslint.config.{js,cjs,mjs}',
-            '{projectRoot}/vite.config.{js,ts,mjs,mts}*',
-          ],
-        },
-      ],
-    },
-    languageOptions: {
-      parser: require('jsonc-eslint-parser'),
-    },
-  },
-];

package/out-tsc/vitest/__tests__/runtime/flow.test.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=flow.test.d.ts.map

package/out-tsc/vitest/__tests__/runtime/flow.test.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"flow.test.d.ts","sourceRoot":"","sources":["../../../../__tests__/runtime/flow.test.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/runtime/steps.test.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=steps.test.d.ts.map

package/out-tsc/vitest/__tests__/runtime/steps.test.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"steps.test.d.ts","sourceRoot":"","sources":["../../../../__tests__/runtime/steps.test.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/runtime/utils.test.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=utils.test.d.ts.map

package/out-tsc/vitest/__tests__/runtime/utils.test.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"utils.test.d.ts","sourceRoot":"","sources":["../../../../__tests__/runtime/utils.test.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/dsl-types.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=dsl-types.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/dsl-types.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"dsl-types.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/dsl-types.test-d.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/example-flow.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=example-flow.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/example-flow.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"example-flow.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/example-flow.test-d.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/extract-flow-input.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=extract-flow-input.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/extract-flow-input.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"extract-flow-input.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/extract-flow-input.test-d.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/extract-flow-steps.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=extract-flow-steps.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/extract-flow-steps.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"extract-flow-steps.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/extract-flow-steps.test-d.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/getStepDefinition.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=getStepDefinition.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/getStepDefinition.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"getStepDefinition.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/getStepDefinition.test-d.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/step-input.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=step-input.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/step-input.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"step-input.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/step-input.test-d.ts"],"names":[],"mappings":""}

package/out-tsc/vitest/__tests__/types/step-output.test-d.d.ts

@@ -1,2 +0,0 @@
-export {};
-//# sourceMappingURL=step-output.test-d.d.ts.map

package/out-tsc/vitest/__tests__/types/step-output.test-d.d.ts.map

@@ -1 +0,0 @@
-{"version":3,"file":"step-output.test-d.d.ts","sourceRoot":"","sources":["../../../../__tests__/types/step-output.test-d.ts"],"names":[],"mappings":""}