@pgflow/dsl 0.0.0-0-10-0-ac3e1101-20251210151347

package/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2025 Wojciech Majewski <owner@pgflow.dev>
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

package/README.md

@@ -0,0 +1,346 @@
+# @pgflow/dsl
+
+The TypeScript Domain Specific Language (DSL) for defining type-safe workflow definitions in pgflow.
+
+> [!NOTE]
+> This project and all its components are licensed under [Apache 2.0](./LICENSE) license.
+
+## Overview
+
+`@pgflow/dsl` provides a type-safe, fluent interface for defining data-driven workflows with explicit dependencies. The DSL ensures that data flows correctly between steps and maintains type safety throughout the entire workflow definition.
+
+Key features:
+
+- **Type Safety** - Complete TypeScript type checking from flow inputs to outputs
+- **Fluent Interface** - Chainable method calls for defining steps and dependencies
+- **Functional Approach** - Clean separation between task implementation and flow orchestration
+- **JSON-Compatible** - All inputs and outputs are JSON-serializable for database storage
+- **Immutable Flow Definitions** - Each step operation returns a new Flow instance
+
+## Usage
+
+### Basic Example
+
+```typescript
+import { Flow } from '@pgflow/dsl';
+
+// Define input type for the flow
+type Input = {
+  url: string;
+};
+
+// Define a flow with steps and dependencies
+export const AnalyzeWebsite = new Flow<Input>({
+  slug: 'analyzeWebsite',
+  maxAttempts: 3,
+  baseDelay: 5,
+  timeout: 10,
+})
+  .step(
+    { slug: 'website' },
+    async (input) => await scrapeWebsite(input.run.url)
+  )
+  .step(
+    { slug: 'sentiment', dependsOn: ['website'] },
+    async (input) => await analyzeSentiment(input.website.content)
+  )
+  .step(
+    { slug: 'summary', dependsOn: ['website'] },
+    async (input) => await summarizeWithAI(input.website.content)
+  )
+  .step(
+    { slug: 'saveToDb', dependsOn: ['sentiment', 'summary'] },
+    async (input) => {
+      return await saveToDb({
+        websiteUrl: input.run.url,
+        sentiment: input.sentiment.score,
+        summary: input.summary.aiSummary,
+      });
+    }
+  );
+```
+
+### Understanding Data Flow
+
+In pgflow, each step receives an `input` object that contains:
+
+1. **`input.run`** - The original flow input (available to all steps)
+2. **`input.{stepName}`** - Outputs from dependency steps
+
+This design ensures:
+
+- Original flow parameters are accessible throughout the entire flow
+- Data doesn't need to be manually forwarded through intermediate steps
+- Steps can combine original input with processed data from previous steps
+
+### Step Methods
+
+The Flow DSL provides three methods for defining steps in your workflow:
+
+#### `.step()` - Regular Steps
+
+The standard method for adding steps to a flow. Each step processes input and returns output.
+
+```typescript
+.step(
+  { slug: 'process', dependsOn: ['previous'] },
+  async (input) => {
+    // Access input.run and input.previous
+    return { result: 'processed' };
+  }
+)
+```
+
+#### `.array()` - Array-Returning Steps
+
+A semantic wrapper around `.step()` that provides type enforcement for steps that return arrays. Useful for data fetching or collection steps that will be processed by map steps.
+
+```typescript
+// Fetch an array of items to be processed
+.array(
+  { slug: 'fetchItems' },
+  async () => [1, 2, 3, 4, 5]
+)
+
+// With dependencies - combining data from multiple sources
+.array(
+  { slug: 'combineResults', dependsOn: ['source1', 'source2'] },
+  async (input) => [...input.source1, ...input.source2]
+)
+```
+
+**Key Points:**
+- Return type is enforced to be an array at compile time
+- Commonly used as input for subsequent map steps
+- Can depend on other steps just like regular steps
+
+#### `.map()` - Array Processing Steps
+
+Processes arrays element-by-element, similar to JavaScript's `Array.map()`. The handler receives individual items instead of the full input object.
+
+**Two Modes of Operation:**
+
+1. **Root Map** (no `array:` property): Processes the flow's input array directly
+   - The flow input MUST be an array when using root maps
+   - Omitting the `array:` property tells pgflow to use the flow input
+
+2. **Dependent Map** (with `array:` property): Processes another step's array output
+   - The `array:` property specifies which step's output to process
+   - That step must return an array
+
+```typescript
+// ROOT MAP - No array: property means use flow input
+// Flow input MUST be an array (e.g., ["hello", "world"])
+new Flow<string[]>({ slug: 'processStrings' })
+  .map(
+    { slug: 'uppercase' }, // No array: property!
+    (item) => item.toUpperCase()
+  );
+// Each string in the input array gets uppercased in parallel
+
+// DEPENDENT MAP - array: property specifies the source step
+new Flow<{}>({ slug: 'dataPipeline' })
+  .array({ slug: 'numbers' }, () => [1, 2, 3])
+  .map(
+    { slug: 'double', array: 'numbers' }, // Processes 'numbers' output
+    (n) => n * 2
+  )
+  .map(
+    { slug: 'square', array: 'double' }, // Chains from 'double'
+    (n) => n * n
+  );
+// Results: numbers: [1,2,3] → double: [2,4,6] → square: [4,16,36]
+```
+
+**Key differences from regular steps:**
+- Uses `array:` to specify dependency (not `dependsOn:`)
+- When `array:` is omitted, uses flow input array (root map)
+- Handler signature is `(item, context) => result` instead of `(input, context) => result`
+- Return type is always an array
+- Map steps can have at most one dependency (the array source)
+- Generates SQL with `step_type => 'map'` parameter for pgflow's map processing
+
+**Type Safety:**
+The `.map()` method provides full TypeScript type inference for array elements:
+
+```typescript
+type User = { id: number; name: string };
+
+new Flow<{}>({ slug: 'userFlow' })
+  .array({ slug: 'users' }, (): User[] => [
+    { id: 1, name: 'Alice' },
+    { id: 2, name: 'Bob' }
+  ])
+  .map({ slug: 'greet', array: 'users' }, (user) => {
+    // TypeScript knows user is of type User
+    return `Hello, ${user.name} (ID: ${user.id})`;
+  });
+```
+
+**Common Patterns:**
+
+```typescript
+// Batch processing - process multiple items in parallel
+new Flow<number[]>({ slug: 'batchProcessor' })
+  .map({ slug: 'validate' }, (item) => {
+    if (item < 0) throw new Error('Invalid item');
+    return item;
+  })
+  .map({ slug: 'process', array: 'validate' }, async (item) => {
+    // Each item processed in its own task
+    return await expensiveOperation(item);
+  });
+
+// Data transformation pipeline
+new Flow<{}>({ slug: 'etlPipeline' })
+  .step({ slug: 'fetchUrls' }, () => ['url1', 'url2', 'url3'])
+  .map({ slug: 'scrape', array: 'fetchUrls' }, async (url) => {
+    return await fetchContent(url);
+  })
+  .map({ slug: 'extract', array: 'scrape' }, (html) => {
+    return extractData(html);
+  })
+  .step({ slug: 'aggregate', dependsOn: ['extract'] }, (input) => {
+    // input.extract is the aggregated array from all map tasks
+    return consolidateResults(input.extract);
+  });
+```
+
+**Limitations:**
+- Can only depend on a single array-returning step
+- TypeScript may not track type transformations between chained maps (use type assertions if needed)
+- Root maps require the entire flow input to be an array
+
+### Context Object
+
+Step handlers can optionally receive a second parameter - the **context object** - which provides access to platform resources and runtime information.
+
+```typescript
+.step(
+  { slug: 'saveToDb' },
+  async (input, context) => {
+    // Access platform resources through context
+    const result = await context.sql`SELECT * FROM users WHERE id = ${input.userId}`;
+    return result[0];
+  }
+)
+```
+
+#### Core Context Resources
+
+All platforms provide these core resources:
+
+- **`context.env`** - Environment variables (`Record<string, string | undefined>`)
+- **`context.shutdownSignal`** - AbortSignal for graceful shutdown handling
+- **`context.rawMessage`** - Original pgmq message with metadata
+  ```typescript
+  interface PgmqMessageRecord<T> {
+    msg_id: number;
+    read_ct: number;
+    enqueued_at: Date;
+    vt: Date;
+    message: T; // <-- this is your 'input'
+  }
+  ```
+- **`context.stepTask`** - Current step task details (flow handlers only)
+  ```typescript
+  interface StepTaskRecord<TFlow> {
+    flow_slug: string;
+    run_id: string;
+    step_slug: string;
+    input: StepInput<TFlow, StepSlug>; // <-- this is handler 'input'
+    msg_id: number;
+  }
+  ```
+- **`context.workerConfig`** - Resolved worker configuration with all defaults applied
+  ```typescript
+  // Provides access to worker settings like retry limits
+  const isLastAttempt = context.rawMessage.read_ct >= context.workerConfig.retry.limit;
+  ```
+
+#### Supabase Platform Resources
+
+When using the Supabase platform with EdgeWorker, additional resources are available:
+
+- **`context.sql`** - PostgreSQL client (postgres.js)
+- **`context.supabase`** - Supabase client with service role key for full database access
+
+To use Supabase resources, import the `Flow` class from the Supabase preset:
+
+```typescript
+import { Flow } from '@pgflow/dsl/supabase';
+
+const MyFlow = new Flow<{ userId: string }>({
+  slug: 'myFlow',
+}).step({ slug: 'process' }, async (input, context) => {
+  // TypeScript knows context includes Supabase resources
+  const { data } = await context.supabase
+    .from('users')
+    .select('*')
+    .eq('id', input.userId);
+
+  // Use SQL directly
+  const stats = await context.sql`
+      SELECT COUNT(*) as total FROM events 
+      WHERE user_id = ${input.userId}
+    `;
+
+  return { user: data[0], eventCount: stats[0].total };
+});
+```
+
+> [!NOTE]
+> Context is optional - handlers that don't need platform resources can omit the second parameter for backward compatibility.
+
+For more details on available resources and platform configuration, see the [@pgflow/edge-worker documentation](https://github.com/pgflow-org/pgflow/tree/main/pkgs/edge-worker#context-resources).
+
+### Flow Configuration
+
+Configure flows and steps with runtime options:
+
+```typescript
+new Flow<Input>({
+  slug: 'myFlow', // Required: Unique flow identifier
+  maxAttempts: 3, // Optional: Maximum retry attempts (default: 1)
+  baseDelay: 5, // Optional: Base delay in seconds for retries (default: 1)
+  timeout: 10, // Optional: Task timeout in seconds (default: 30)
+});
+```
+
+## Compiling Flows
+
+Use the `compileFlow` utility to convert a flow definition into SQL statements:
+
+```typescript
+import { compileFlow } from '@pgflow/dsl';
+
+const sqlStatements = compileFlow(MyFlow);
+console.log(sqlStatements.join('\n'));
+```
+
+Alternatively, use the pgflow CLI to compile flows directly to migration files:
+
+```bash
+npx pgflow compile path/to/flow.ts
+```
+
+## Requirements
+
+- All step inputs and outputs MUST be JSON-serializable
+- Use only: primitive types, plain objects, and arrays
+- Convert dates to ISO strings (`new Date().toISOString()`)
+- Avoid: class instances, functions, symbols, undefined values, and circular references
+
+## Building
+
+Run `nx build dsl` to build the library.
+
+## Running unit tests
+
+Run `nx test dsl` to execute the unit tests via [Vitest](https://vitest.dev/).
+
+## Documentation
+
+For detailed documentation on the Flow DSL, visit:
+
+- [Understanding the Flow DSL](https://pgflow.dev/explanations/flow-dsl/)