@pgflow/core 0.0.0-add-workerconfig-to-context--20250905094004-b98e1fec-20250905074005
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +202 -0
- package/README.md +393 -0
- package/dist/ATLAS.md +32 -0
- package/dist/CHANGELOG.md +639 -0
- package/dist/PgflowSqlClient.d.ts +17 -0
- package/dist/PgflowSqlClient.d.ts.map +1 -0
- package/dist/PgflowSqlClient.js +70 -0
- package/dist/README.md +393 -0
- package/dist/database-types.d.ts +828 -0
- package/dist/database-types.d.ts.map +1 -0
- package/dist/database-types.js +8 -0
- package/dist/index.d.ts +4 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +2 -0
- package/dist/package.json +32 -0
- package/dist/supabase/migrations/20250429164909_pgflow_initial.sql +579 -0
- package/dist/supabase/migrations/20250517072017_pgflow_fix_poll_for_tasks_to_use_separate_statement_for_polling.sql +101 -0
- package/dist/supabase/migrations/20250609105135_pgflow_add_start_tasks_and_started_status.sql +371 -0
- package/dist/supabase/migrations/20250610180554_pgflow_add_set_vt_batch_and_use_it_in_start_tasks.sql +127 -0
- package/dist/supabase/migrations/20250614124241_pgflow_add_realtime.sql +501 -0
- package/dist/supabase/migrations/20250619195327_pgflow_fix_fail_task_missing_realtime_event.sql +185 -0
- package/dist/supabase/migrations/20250627090700_pgflow_fix_function_search_paths.sql +6 -0
- package/dist/supabase/migrations/20250707210212_pgflow_add_opt_start_delay.sql +103 -0
- package/dist/supabase/migrations/20250719205006_pgflow_worker_deprecation.sql +2 -0
- package/dist/tsconfig.lib.tsbuildinfo +1 -0
- package/dist/types.d.ts +94 -0
- package/dist/types.d.ts.map +1 -0
- package/dist/types.js +1 -0
- package/package.json +32 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,202 @@
|
|
|
1
|
+
|
|
2
|
+
Apache License
|
|
3
|
+
Version 2.0, January 2004
|
|
4
|
+
http://www.apache.org/licenses/
|
|
5
|
+
|
|
6
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
7
|
+
|
|
8
|
+
1. Definitions.
|
|
9
|
+
|
|
10
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
11
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
12
|
+
|
|
13
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
14
|
+
the copyright owner that is granting the License.
|
|
15
|
+
|
|
16
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
17
|
+
other entities that control, are controlled by, or are under common
|
|
18
|
+
control with that entity. For the purposes of this definition,
|
|
19
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
20
|
+
direction or management of such entity, whether by contract or
|
|
21
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
22
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
23
|
+
|
|
24
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
25
|
+
exercising permissions granted by this License.
|
|
26
|
+
|
|
27
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
28
|
+
including but not limited to software source code, documentation
|
|
29
|
+
source, and configuration files.
|
|
30
|
+
|
|
31
|
+
"Object" form shall mean any form resulting from mechanical
|
|
32
|
+
transformation or translation of a Source form, including but
|
|
33
|
+
not limited to compiled object code, generated documentation,
|
|
34
|
+
and conversions to other media types.
|
|
35
|
+
|
|
36
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
37
|
+
Object form, made available under the License, as indicated by a
|
|
38
|
+
copyright notice that is included in or attached to the work
|
|
39
|
+
(an example is provided in the Appendix below).
|
|
40
|
+
|
|
41
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
42
|
+
form, that is based on (or derived from) the Work and for which the
|
|
43
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
44
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
45
|
+
of this License, Derivative Works shall not include works that remain
|
|
46
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
47
|
+
the Work and Derivative Works thereof.
|
|
48
|
+
|
|
49
|
+
"Contribution" shall mean any work of authorship, including
|
|
50
|
+
the original version of the Work and any modifications or additions
|
|
51
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
52
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
53
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
54
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
55
|
+
means any form of electronic, verbal, or written communication sent
|
|
56
|
+
to the Licensor or its representatives, including but not limited to
|
|
57
|
+
communication on electronic mailing lists, source code control systems,
|
|
58
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
59
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
60
|
+
excluding communication that is conspicuously marked or otherwise
|
|
61
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
62
|
+
|
|
63
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
64
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
65
|
+
subsequently incorporated within the Work.
|
|
66
|
+
|
|
67
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
68
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
69
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
70
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
71
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
72
|
+
Work and such Derivative Works in Source or Object form.
|
|
73
|
+
|
|
74
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
75
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
76
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
77
|
+
(except as stated in this section) patent license to make, have made,
|
|
78
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
79
|
+
where such license applies only to those patent claims licensable
|
|
80
|
+
by such Contributor that are necessarily infringed by their
|
|
81
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
82
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
83
|
+
institute patent litigation against any entity (including a
|
|
84
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
85
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
86
|
+
or contributory patent infringement, then any patent licenses
|
|
87
|
+
granted to You under this License for that Work shall terminate
|
|
88
|
+
as of the date such litigation is filed.
|
|
89
|
+
|
|
90
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
91
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
92
|
+
modifications, and in Source or Object form, provided that You
|
|
93
|
+
meet the following conditions:
|
|
94
|
+
|
|
95
|
+
(a) You must give any other recipients of the Work or
|
|
96
|
+
Derivative Works a copy of this License; and
|
|
97
|
+
|
|
98
|
+
(b) You must cause any modified files to carry prominent notices
|
|
99
|
+
stating that You changed the files; and
|
|
100
|
+
|
|
101
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
102
|
+
that You distribute, all copyright, patent, trademark, and
|
|
103
|
+
attribution notices from the Source form of the Work,
|
|
104
|
+
excluding those notices that do not pertain to any part of
|
|
105
|
+
the Derivative Works; and
|
|
106
|
+
|
|
107
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
108
|
+
distribution, then any Derivative Works that You distribute must
|
|
109
|
+
include a readable copy of the attribution notices contained
|
|
110
|
+
within such NOTICE file, excluding those notices that do not
|
|
111
|
+
pertain to any part of the Derivative Works, in at least one
|
|
112
|
+
of the following places: within a NOTICE text file distributed
|
|
113
|
+
as part of the Derivative Works; within the Source form or
|
|
114
|
+
documentation, if provided along with the Derivative Works; or,
|
|
115
|
+
within a display generated by the Derivative Works, if and
|
|
116
|
+
wherever such third-party notices normally appear. The contents
|
|
117
|
+
of the NOTICE file are for informational purposes only and
|
|
118
|
+
do not modify the License. You may add Your own attribution
|
|
119
|
+
notices within Derivative Works that You distribute, alongside
|
|
120
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
121
|
+
that such additional attribution notices cannot be construed
|
|
122
|
+
as modifying the License.
|
|
123
|
+
|
|
124
|
+
You may add Your own copyright statement to Your modifications and
|
|
125
|
+
may provide additional or different license terms and conditions
|
|
126
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
127
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
128
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
129
|
+
the conditions stated in this License.
|
|
130
|
+
|
|
131
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
132
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
133
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
134
|
+
this License, without any additional terms or conditions.
|
|
135
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
136
|
+
the terms of any separate license agreement you may have executed
|
|
137
|
+
with Licensor regarding such Contributions.
|
|
138
|
+
|
|
139
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
140
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
141
|
+
except as required for reasonable and customary use in describing the
|
|
142
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
143
|
+
|
|
144
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
145
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
146
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
147
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
148
|
+
implied, including, without limitation, any warranties or conditions
|
|
149
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
150
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
151
|
+
appropriateness of using or redistributing the Work and assume any
|
|
152
|
+
risks associated with Your exercise of permissions under this License.
|
|
153
|
+
|
|
154
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
155
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
156
|
+
unless required by applicable law (such as deliberate and grossly
|
|
157
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
158
|
+
liable to You for damages, including any direct, indirect, special,
|
|
159
|
+
incidental, or consequential damages of any character arising as a
|
|
160
|
+
result of this License or out of the use or inability to use the
|
|
161
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
162
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
163
|
+
other commercial damages or losses), even if such Contributor
|
|
164
|
+
has been advised of the possibility of such damages.
|
|
165
|
+
|
|
166
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
167
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
168
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
169
|
+
or other liability obligations and/or rights consistent with this
|
|
170
|
+
License. However, in accepting such obligations, You may act only
|
|
171
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
172
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
173
|
+
defend, and hold each Contributor harmless for any liability
|
|
174
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
175
|
+
of your accepting any such warranty or additional liability.
|
|
176
|
+
|
|
177
|
+
END OF TERMS AND CONDITIONS
|
|
178
|
+
|
|
179
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
180
|
+
|
|
181
|
+
To apply the Apache License to your work, attach the following
|
|
182
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
183
|
+
replaced with your own identifying information. (Don't include
|
|
184
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
185
|
+
comment syntax for the file format. We also recommend that a
|
|
186
|
+
file or class name and description of purpose be included on the
|
|
187
|
+
same "printed page" as the copyright notice for easier
|
|
188
|
+
identification within third-party archives.
|
|
189
|
+
|
|
190
|
+
Copyright 2025 Wojciech Majewski <owner@pgflow.dev>
|
|
191
|
+
|
|
192
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
193
|
+
you may not use this file except in compliance with the License.
|
|
194
|
+
You may obtain a copy of the License at
|
|
195
|
+
|
|
196
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
197
|
+
|
|
198
|
+
Unless required by applicable law or agreed to in writing, software
|
|
199
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
200
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
201
|
+
See the License for the specific language governing permissions and
|
|
202
|
+
limitations under the License.
|
package/README.md
ADDED
|
@@ -0,0 +1,393 @@
|
|
|
1
|
+
# pgflow SQL Core
|
|
2
|
+
|
|
3
|
+
PostgreSQL-native workflow engine for defining, managing, and tracking DAG-based workflows directly in your database.
|
|
4
|
+
|
|
5
|
+
> [!NOTE]
|
|
6
|
+
> This project and all its components are licensed under [Apache 2.0](./LICENSE) license.
|
|
7
|
+
|
|
8
|
+
> [!WARNING]
|
|
9
|
+
> This project uses [Atlas](https://atlasgo.io/docs) to manage the schemas and migrations.
|
|
10
|
+
> See [ATLAS.md](ATLAS.md) for more details.
|
|
11
|
+
|
|
12
|
+
## Table of Contents
|
|
13
|
+
|
|
14
|
+
- [Overview](#overview)
|
|
15
|
+
- [Key Features](#key-features)
|
|
16
|
+
- [Architecture](#architecture)
|
|
17
|
+
- [Schema Design](#schema-design)
|
|
18
|
+
- [Execution Model](#execution-model)
|
|
19
|
+
- [Example Flow and its life](#example-flow-and-its-life)
|
|
20
|
+
- [Defining a Workflow](#defining-a-workflow)
|
|
21
|
+
- [Starting a Workflow Run](#starting-a-workflow-run)
|
|
22
|
+
- [Workflow Execution](#workflow-execution)
|
|
23
|
+
- [Task Polling](#task-polling)
|
|
24
|
+
- [Task Completion](#task-completion)
|
|
25
|
+
- [Error Handling](#error-handling)
|
|
26
|
+
- [Retries and Timeouts](#retries-and-timeouts)
|
|
27
|
+
- [TypeScript Flow DSL](#typescript-flow-dsl)
|
|
28
|
+
- [Overview](#overview-1)
|
|
29
|
+
- [Type Inference System](#type-inference-system)
|
|
30
|
+
- [Basic Example](#basic-example)
|
|
31
|
+
- [How Payload Types Are Built](#how-payload-types-are-built)
|
|
32
|
+
- [Benefits of Automatic Type Inference](#benefits-of-automatic-type-inference)
|
|
33
|
+
- [Data Flow](#data-flow)
|
|
34
|
+
- [Input and Output Handling](#input-and-output-handling)
|
|
35
|
+
- [Run Completion](#run-completion)
|
|
36
|
+
|
|
37
|
+
## Overview
|
|
38
|
+
|
|
39
|
+
The pgflow SQL Core provides the data model, state machine, and transactional functions for workflow management. It treats workflows as Directed Acyclic Graphs (DAGs) of steps, each step being a simple state machine.
|
|
40
|
+
|
|
41
|
+
This package focuses on:
|
|
42
|
+
|
|
43
|
+
- Defining and storing workflow shapes
|
|
44
|
+
- Managing workflow state transitions
|
|
45
|
+
- Exposing transactional functions for workflow operations
|
|
46
|
+
- Providing two-phase APIs for reliable task polling and status updates
|
|
47
|
+
|
|
48
|
+
The actual execution of workflow tasks is handled by the [Edge Worker](../edge-worker/README.md), which calls back to the SQL Core to acknowledge task completion or failure.
|
|
49
|
+
|
|
50
|
+
## Key Features
|
|
51
|
+
|
|
52
|
+
- **Declarative Workflows**: Define flows and steps via SQL tables
|
|
53
|
+
- **Dependency Management**: Explicit step dependencies with atomic transitions
|
|
54
|
+
- **Configurable Behavior**: Per-flow and per-step options for timeouts, retries, and delays
|
|
55
|
+
- **Queue Integration**: Built on pgmq for reliable task processing
|
|
56
|
+
- **Transactional Guarantees**: All state transitions are ACID-compliant
|
|
57
|
+
|
|
58
|
+
## Architecture
|
|
59
|
+
|
|
60
|
+
### Schema Design
|
|
61
|
+
|
|
62
|
+
[Schema ERD Diagram (click to enlarge)](./assets/schema.svg)
|
|
63
|
+
|
|
64
|
+
<a href="./assets/schema.svg">
|
|
65
|
+
<img src="./assets/schema.svg" alt="Schema ERD Diagram" width="25%" height="25%">
|
|
66
|
+
</a>
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
The schema consists of two main categories of tables:
|
|
71
|
+
|
|
72
|
+
#### Static definition tables
|
|
73
|
+
|
|
74
|
+
- `flows` (just an identity for the workflow with some global options)
|
|
75
|
+
- `steps` (DAG nodes belonging to particular `flows`, with option overrides)
|
|
76
|
+
- `deps` (DAG edges between `steps`)
|
|
77
|
+
|
|
78
|
+
#### Runtime state tables
|
|
79
|
+
|
|
80
|
+
- `runs` (execution instances of `flows`)
|
|
81
|
+
- `step_states` (states of individual `steps` within a `run`)
|
|
82
|
+
- `step_tasks` (units of work for individual `steps` within a `run`, so we can have fanouts)
|
|
83
|
+
|
|
84
|
+
### Execution Model
|
|
85
|
+
|
|
86
|
+
The SQL Core handles the workflow lifecycle through these key operations:
|
|
87
|
+
|
|
88
|
+
1. **Definition**: Workflows are defined using `create_flow` and `add_step`
|
|
89
|
+
2. **Instantiation**: Workflow instances are started with `start_flow`, creating a new run
|
|
90
|
+
3. **Task Retrieval**: The [Edge Worker](../edge-worker/README.md) uses two-phase polling - first `read_with_poll` to reserve queue messages, then `start_tasks` to convert them to executable tasks
|
|
91
|
+
4. **State Transitions**: When the Edge Worker reports back using `complete_task` or `fail_task`, the SQL Core handles state transitions and schedules dependent steps
|
|
92
|
+
|
|
93
|
+
[Flow lifecycle diagram (click to enlarge)](./assets/flow-lifecycle.svg)
|
|
94
|
+
|
|
95
|
+
<a href="./assets/flow-lifecycle.svg"><img src="./assets/flow-lifecycle.svg" alt="Flow Lifecycle" width="25%" height="25%"></a>
|
|
96
|
+
|
|
97
|
+
## Example flow and its life
|
|
98
|
+
|
|
99
|
+
Let's walk through creating and running a workflow that fetches a website,
|
|
100
|
+
does summarization and sentiment analysis in parallel steps
|
|
101
|
+
and saves the results to a database.
|
|
102
|
+
|
|
103
|
+

|
|
104
|
+
|
|
105
|
+
### Defining a Workflow
|
|
106
|
+
|
|
107
|
+
Workflows are defined using two SQL functions: `create_flow` and `add_step`.
|
|
108
|
+
|
|
109
|
+
In this example, we'll create a workflow with:
|
|
110
|
+
|
|
111
|
+
- `website` as the entry point ("root step")
|
|
112
|
+
- `sentiment` and `summary` as parallel steps that depend on `website`
|
|
113
|
+
- `saveToDb` as the final step, depending on both parallel steps
|
|
114
|
+
|
|
115
|
+
```sql
|
|
116
|
+
-- Define workflow with parallel steps
|
|
117
|
+
SELECT pgflow.create_flow('analyze_website');
|
|
118
|
+
SELECT pgflow.add_step('analyze_website', 'website');
|
|
119
|
+
SELECT pgflow.add_step('analyze_website', 'sentiment', deps_slugs => ARRAY['website']);
|
|
120
|
+
SELECT pgflow.add_step('analyze_website', 'summary', deps_slugs => ARRAY['website']);
|
|
121
|
+
SELECT pgflow.add_step('analyze_website', 'saveToDb', deps_slugs => ARRAY['sentiment', 'summary']);
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
> [!WARNING]
|
|
125
|
+
> You need to call `add_step` in topological order, which is enforced by foreign key constraints.
|
|
126
|
+
|
|
127
|
+
> [!NOTE]
|
|
128
|
+
> You can have multiple "root steps" in a workflow. You can even create a root-steps-only workflow
|
|
129
|
+
> to process a single input in parallel, because at the end, all of the outputs from steps
|
|
130
|
+
> that does not have dependents ("final steps") are aggregated and saved as run's `output`.
|
|
131
|
+
|
|
132
|
+
### Starting a Workflow Run
|
|
133
|
+
|
|
134
|
+
To start a workflow, call `start_flow` with a flow slug and input arguments:
|
|
135
|
+
|
|
136
|
+
```sql
|
|
137
|
+
SELECT * FROM pgflow.start_flow(
|
|
138
|
+
flow_slug => 'analyze_website',
|
|
139
|
+
input => '{"url": "https://example.com"}'::jsonb
|
|
140
|
+
);
|
|
141
|
+
|
|
142
|
+
-- run_id | flow_slug | status | input | output | remaining_steps
|
|
143
|
+
-- ------------+-----------------+---------+--------------------------------+--------+-----------------
|
|
144
|
+
-- <run uuid> | analyze_website | started | {"url": "https://example.com"} | [NULL] | 4
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
When a workflow starts:
|
|
148
|
+
|
|
149
|
+
- A new `run` record is created
|
|
150
|
+
- Initial states for all steps are created
|
|
151
|
+
- Root steps are marked as `started`
|
|
152
|
+
- Tasks are created for root steps
|
|
153
|
+
- Messages are enqueued on PGMQ for worker processing
|
|
154
|
+
|
|
155
|
+
> [!NOTE]
|
|
156
|
+
> The `input` argument must be a valid JSONB object: string, number, boolean, array, object or null.
|
|
157
|
+
|
|
158
|
+
### Workflow Execution
|
|
159
|
+
|
|
160
|
+
#### Task Polling
|
|
161
|
+
|
|
162
|
+
The Edge Worker uses a two-phase approach to retrieve and start tasks:
|
|
163
|
+
|
|
164
|
+
**Phase 1 - Reserve Messages:**
|
|
165
|
+
```sql
|
|
166
|
+
SELECT * FROM pgflow.read_with_poll(
|
|
167
|
+
queue_name => 'analyze_website',
|
|
168
|
+
vt => 60, -- visibility timeout in seconds
|
|
169
|
+
qty => 5 -- maximum number of messages to fetch
|
|
170
|
+
);
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
**Phase 2 - Start Tasks:**
|
|
174
|
+
```sql
|
|
175
|
+
SELECT * FROM pgflow.start_tasks(
|
|
176
|
+
flow_slug => 'analyze_website',
|
|
177
|
+
msg_ids => ARRAY[101, 102, 103], -- message IDs from phase 1
|
|
178
|
+
worker_id => '550e8400-e29b-41d4-a716-446655440000'::uuid
|
|
179
|
+
);
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
**How it works:**
|
|
183
|
+
|
|
184
|
+
1. **read_with_poll** reserves raw queue messages and hides them from other workers
|
|
185
|
+
2. **start_tasks** finds matching step_tasks, increments attempts counter, and builds task inputs
|
|
186
|
+
3. Task metadata and input are returned to the worker for execution
|
|
187
|
+
|
|
188
|
+
This two-phase approach ensures tasks always exist before processing begins, eliminating race conditions that could occur with single-phase polling.
|
|
189
|
+
|
|
190
|
+
#### Task Completion
|
|
191
|
+
|
|
192
|
+
After successful processing, the worker acknowledges completion:
|
|
193
|
+
|
|
194
|
+
```sql
|
|
195
|
+
SELECT pgflow.complete_task(
|
|
196
|
+
run_id => '<run_uuid>',
|
|
197
|
+
step_slug => 'website',
|
|
198
|
+
task_index => 0, -- we will have multiple tasks for a step in the future
|
|
199
|
+
output => '{"content": "HTML content", "status": 200}'::jsonb
|
|
200
|
+
);
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
When a task completes:
|
|
204
|
+
|
|
205
|
+
1. The task status is updated to 'completed' and the output is saved
|
|
206
|
+
2. The message is archived in PGMQ
|
|
207
|
+
3. The step state is updated to 'completed'
|
|
208
|
+
4. Dependent steps with all dependencies completed are automatically started
|
|
209
|
+
5. The run's remaining_steps counter is decremented
|
|
210
|
+
6. If all steps are completed, the run is marked as completed with aggregated outputs
|
|
211
|
+
|
|
212
|
+
#### Error Handling
|
|
213
|
+
|
|
214
|
+
If a task fails, the worker acknowledges this using `fail_task`:
|
|
215
|
+
|
|
216
|
+
```sql
|
|
217
|
+
SELECT pgflow.fail_task(
|
|
218
|
+
run_id => '<run_uuid>',
|
|
219
|
+
step_slug => 'website',
|
|
220
|
+
task_index => 0,
|
|
221
|
+
error_message => 'Connection timeout when fetching URL'::text
|
|
222
|
+
);
|
|
223
|
+
```
|
|
224
|
+
|
|
225
|
+
The system handles failures by:
|
|
226
|
+
|
|
227
|
+
1. Checking if retry attempts are available
|
|
228
|
+
2. For available retries:
|
|
229
|
+
- Keeping the task in 'queued' status
|
|
230
|
+
- Applying exponential backoff for visibility
|
|
231
|
+
- Preventing processing until the visibility timeout expires
|
|
232
|
+
3. When retries are exhausted:
|
|
233
|
+
- Marking the task as 'failed'
|
|
234
|
+
- Marking the step as 'failed'
|
|
235
|
+
- Marking the run as 'failed'
|
|
236
|
+
- Archiving the message in PGMQ
|
|
237
|
+
- Notifying workers to abort pending tasks (future feature)
|
|
238
|
+
|
|
239
|
+
#### Retries and Timeouts
|
|
240
|
+
|
|
241
|
+
Retry behavior can be configured at both the flow and step level:
|
|
242
|
+
|
|
243
|
+
```sql
|
|
244
|
+
-- Flow-level defaults
|
|
245
|
+
SELECT pgflow.create_flow(
|
|
246
|
+
flow_slug => 'analyze_website',
|
|
247
|
+
max_attempts => 3, -- Maximum retry attempts (including first attempt)
|
|
248
|
+
base_delay => 5, -- Base delay in seconds for exponential backoff
|
|
249
|
+
timeout => 60 -- Task timeout in seconds
|
|
250
|
+
);
|
|
251
|
+
|
|
252
|
+
-- Step-level overrides
|
|
253
|
+
SELECT pgflow.add_step(
|
|
254
|
+
flow_slug => 'analyze_website',
|
|
255
|
+
step_slug => 'sentiment',
|
|
256
|
+
deps_slugs => ARRAY['website']::text[],
|
|
257
|
+
max_attempts => 5, -- Override max attempts for this step
|
|
258
|
+
base_delay => 2, -- Override base delay for exponential backoff
|
|
259
|
+
timeout => 30 -- Override timeout for this step
|
|
260
|
+
);
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
The system applies exponential backoff for retries using the formula:
|
|
264
|
+
|
|
265
|
+
```
|
|
266
|
+
delay = base_delay * (2 ^ attempts_count)
|
|
267
|
+
```
|
|
268
|
+
|
|
269
|
+
Timeouts are enforced by setting the message visibility timeout to the step's timeout value plus a small buffer. If a worker doesn't acknowledge completion or failure within this period, the task becomes visible again and can be retried.
|
|
270
|
+
|
|
271
|
+
## TypeScript Flow DSL
|
|
272
|
+
|
|
273
|
+
> [!NOTE]
|
|
274
|
+
> TypeScript Flow DSL is a Work In Progress and is not ready yet!
|
|
275
|
+
|
|
276
|
+
### Overview
|
|
277
|
+
|
|
278
|
+
While the SQL Core engine handles workflow definitions and state management, the primary way to define and work with your workflow logic is via the Flow DSL in TypeScript. This DSL offers a fluent API that makes it straightforward to outline the steps in your flow with full type safety.
|
|
279
|
+
|
|
280
|
+
### Type Inference System
|
|
281
|
+
|
|
282
|
+
The most powerful feature of the Flow DSL is its **automatic type inference system**:
|
|
283
|
+
|
|
284
|
+
1. You only need to annotate the initial Flow input type
|
|
285
|
+
2. The return type of each step is automatically inferred from your handler function
|
|
286
|
+
3. These return types become available in the payload of dependent steps
|
|
287
|
+
4. The TypeScript compiler builds a complete type graph matching your workflow DAG
|
|
288
|
+
|
|
289
|
+
This means you get full IDE autocompletion and type checking throughout your workflow without manual type annotations.
|
|
290
|
+
|
|
291
|
+
### Basic Example
|
|
292
|
+
|
|
293
|
+
Here's an example that matches our website analysis workflow:
|
|
294
|
+
|
|
295
|
+
```ts
|
|
296
|
+
// Provide a type for the input of the Flow
|
|
297
|
+
type Input = {
|
|
298
|
+
url: string;
|
|
299
|
+
};
|
|
300
|
+
|
|
301
|
+
const AnalyzeWebsite = new Flow<Input>({
|
|
302
|
+
slug: 'analyze_website',
|
|
303
|
+
maxAttempts: 3,
|
|
304
|
+
baseDelay: 5,
|
|
305
|
+
timeout: 10,
|
|
306
|
+
})
|
|
307
|
+
.step(
|
|
308
|
+
{ slug: 'website' },
|
|
309
|
+
async (input) => await scrapeWebsite(input.run.url)
|
|
310
|
+
)
|
|
311
|
+
.step(
|
|
312
|
+
{ slug: 'sentiment', dependsOn: ['website'], timeout: 30, maxAttempts: 5 },
|
|
313
|
+
async (input) => await analyzeSentiment(input.website.content)
|
|
314
|
+
)
|
|
315
|
+
.step(
|
|
316
|
+
{ slug: 'summary', dependsOn: ['website'] },
|
|
317
|
+
async (input) => await summarizeWithAI(input.website.content)
|
|
318
|
+
)
|
|
319
|
+
.step(
|
|
320
|
+
{ slug: 'saveToDb', dependsOn: ['sentiment', 'summary'] },
|
|
321
|
+
async (input) =>
|
|
322
|
+
await saveToDb({
|
|
323
|
+
websiteUrl: input.run.url,
|
|
324
|
+
sentiment: input.sentiment.score,
|
|
325
|
+
summary: input.summary,
|
|
326
|
+
}).status
|
|
327
|
+
);
|
|
328
|
+
```
|
|
329
|
+
|
|
330
|
+
### How Payload Types Are Built
|
|
331
|
+
|
|
332
|
+
The payload object for each step is constructed dynamically based on:
|
|
333
|
+
|
|
334
|
+
1. **The `run` property**: Always contains the original workflow input
|
|
335
|
+
2. **Dependency outputs**: Each dependency's output is available under a key matching the dependency's ID
|
|
336
|
+
3. **DAG structure**: Only outputs from direct dependencies are included in the payload
|
|
337
|
+
|
|
338
|
+
This means your step handlers receive exactly the data they need, properly typed, without any manual type declarations beyond the initial Flow input type.
|
|
339
|
+
|
|
340
|
+
### Benefits of Automatic Type Inference
|
|
341
|
+
|
|
342
|
+
- **Refactoring safety**: Change a step's output, and TypeScript will flag all dependent steps that need updates
|
|
343
|
+
- **Discoverability**: IDE autocompletion shows exactly what data is available in each step
|
|
344
|
+
- **Error prevention**: Catch typos and type mismatches at compile time, not runtime
|
|
345
|
+
- **Documentation**: The types themselves serve as living documentation of your workflow's data flow
|
|
346
|
+
|
|
347
|
+
## Data Flow
|
|
348
|
+
|
|
349
|
+
### Input and Output Handling
|
|
350
|
+
|
|
351
|
+
Handlers in pgflow **must return** JSON-serializable values that are captured and saved when `complete_task` is called. These outputs become available as inputs to dependent steps, allowing data to flow through your workflow pipeline.
|
|
352
|
+
|
|
353
|
+
When a step is executed, it receives an input object where:
|
|
354
|
+
|
|
355
|
+
- Each key is a step_slug of a completed dependency
|
|
356
|
+
- Each value is that step's output
|
|
357
|
+
- A special "run" key contains the original workflow input
|
|
358
|
+
|
|
359
|
+
#### Example: `sentiment`
|
|
360
|
+
|
|
361
|
+
When the `sentiment` step runs, it receives:
|
|
362
|
+
|
|
363
|
+
```json
|
|
364
|
+
{
|
|
365
|
+
"run": { "url": "https://example.com" },
|
|
366
|
+
"website": { "content": "HTML content", "status": 200 }
|
|
367
|
+
}
|
|
368
|
+
```
|
|
369
|
+
|
|
370
|
+
#### Example: `saveToDb`
|
|
371
|
+
|
|
372
|
+
The `saveToDb` step depends on both `sentiment` and `summary`:
|
|
373
|
+
|
|
374
|
+
```json
|
|
375
|
+
{
|
|
376
|
+
"run": { "url": "https://example.com" },
|
|
377
|
+
"sentiment": { "score": 0.85, "label": "positive" },
|
|
378
|
+
"summary": "This website discusses various topics related to technology and innovation."
|
|
379
|
+
}
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
### Run Completion
|
|
383
|
+
|
|
384
|
+
When all steps in a run are completed, the run status is automatically updated to 'completed' and its output is set. The output is an aggregation of all the outputs from final steps (steps that have no dependents):
|
|
385
|
+
|
|
386
|
+
```sql
|
|
387
|
+
-- Example of a completed run with output
|
|
388
|
+
SELECT run_id, status, output FROM pgflow.runs WHERE run_id = '<run_uuid>';
|
|
389
|
+
|
|
390
|
+
-- run_id | status | output
|
|
391
|
+
-- ------------+-----------+-----------------------------------------------------
|
|
392
|
+
-- <run uuid> | completed | {"saveToDb": {"status": "success"}}
|
|
393
|
+
```
|
package/dist/ATLAS.md
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Atlas setup
|
|
2
|
+
|
|
3
|
+
We use [Atlas](https://atlasgo.io/docs) to generate migrations from the declarative schemas stored in `./schemas/` folder.
|
|
4
|
+
|
|
5
|
+
## Configuration
|
|
6
|
+
|
|
7
|
+
The setup is configured in `atlas.hcl`.
|
|
8
|
+
|
|
9
|
+
It is set to compare `schemas/` to what is in `supabase/migrations/`.
|
|
10
|
+
|
|
11
|
+
### Docker dev image
|
|
12
|
+
|
|
13
|
+
Atlas requires a dev database to be available for computing diffs.
|
|
14
|
+
The database must be empty, but contain everything needed for the schemas to apply.
|
|
15
|
+
|
|
16
|
+
We need a configured [PGMQ](https://github.com/tembo-io/pgmq) extension, which Atlas does not support
|
|
17
|
+
in their dev images.
|
|
18
|
+
|
|
19
|
+
That's why this setup relies on a custom built image `jumski/postgres-15-pgmq:latest`.
|
|
20
|
+
|
|
21
|
+
Inspect `Dockerfile.atlas` to see how it is built.
|
|
22
|
+
|
|
23
|
+
See also `./scripts/build-atlas-postgres-image` and `./scripts/push-atlas-postgres-image` scripts for building and pushing the image.
|
|
24
|
+
|
|
25
|
+
## Workflow
|
|
26
|
+
|
|
27
|
+
1. Make sure you start with a clean database (`pnpm supabase db reset`).
|
|
28
|
+
1. Modify the schemas in `schemas/` to a desired state.
|
|
29
|
+
1. Run `./scripts/atlas-migrate-diff <migration-name>` to create a new migration based on the diff.
|
|
30
|
+
1. Run `pnpm supabase migration up` to apply the migration.
|
|
31
|
+
1. In case of any errors, remove the generated migration file, make changes in `schemas/` and repeat the process.
|
|
32
|
+
1. After the migration is applied, verify it does not break tests with `nx test:pgtap`
|