forktex-flow 0.2.0__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- forktex_flow-0.4.0/PKG-INFO +415 -0
- forktex_flow-0.4.0/README.md +380 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/pyproject.toml +8 -2
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/__init__.py +11 -31
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/domain/__init__.py +1 -11
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/domain/config.py +7 -26
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/domain/definition.py +4 -47
- forktex_flow-0.4.0/src/forktex_flow/domain/filter.py +45 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/domain/node.py +3 -32
- forktex_flow-0.4.0/src/forktex_flow/domain/state.py +101 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/domain/types.py +31 -21
- forktex_flow-0.4.0/src/forktex_flow/errors.py +82 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/extension.py +7 -30
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/flow.py +56 -121
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/__init__.py +3 -12
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/mappers.py +1 -7
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/migrations/__init__.py +4 -16
- forktex_flow-0.4.0/src/forktex_flow/persist/migrations/v0001__workflow.sql +20 -0
- forktex_flow-0.4.0/src/forktex_flow/persist/migrations/v0002__run.sql +50 -0
- forktex_flow-0.4.0/src/forktex_flow/persist/migrations/v0003__step_run.sql +39 -0
- forktex_flow-0.4.0/src/forktex_flow/persist/migrations/v0004__run_event.sql +17 -0
- forktex_flow-0.4.0/src/forktex_flow/persist/migrations/v0005__scheduled_run.sql +21 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/models.py +39 -41
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/runs.py +89 -79
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/steps.py +55 -93
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/persist/store.py +13 -20
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/read/__init__.py +2 -5
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/read/instance.py +33 -98
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/read/query.py +46 -71
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/__init__.py +4 -10
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/compiler.py +25 -52
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/context.py +0 -9
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/ctx.py +8 -35
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/driver.py +101 -79
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/execution.py +81 -81
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/runtime/registry.py +1 -18
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/testing.py +6 -57
- forktex_flow-0.2.0/PKG-INFO +0 -363
- forktex_flow-0.2.0/README.md +0 -328
- forktex_flow-0.2.0/src/forktex_flow/domain/filter.py +0 -71
- forktex_flow-0.2.0/src/forktex_flow/domain/state.py +0 -107
- forktex_flow-0.2.0/src/forktex_flow/errors.py +0 -108
- forktex_flow-0.2.0/src/forktex_flow/persist/migrations/v0001__schema.sql +0 -125
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/LICENSE +0 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/NOTICE +0 -0
- {forktex_flow-0.2.0 → forktex_flow-0.4.0}/src/forktex_flow/py.typed +0 -0
|
@@ -0,0 +1,415 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: forktex-flow
|
|
3
|
+
Version: 0.4.0
|
|
4
|
+
Summary: Postgres-native durable workflow execution — pipelines and cron-scheduled runs, built on the forktex substrate.
|
|
5
|
+
License-Expression: AGPL-3.0-or-later OR LicenseRef-ForkTex-Commercial
|
|
6
|
+
License-File: LICENSE
|
|
7
|
+
License-File: NOTICE
|
|
8
|
+
Author: FORKTEX
|
|
9
|
+
Author-email: info@forktex.com
|
|
10
|
+
Requires-Python: >=3.14,<4.0
|
|
11
|
+
Classifier: Development Status :: 4 - Beta
|
|
12
|
+
Classifier: Intended Audience :: Developers
|
|
13
|
+
Classifier: Operating System :: OS Independent
|
|
14
|
+
Classifier: Programming Language :: Python :: 3
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
17
|
+
Classifier: Topic :: Database
|
|
18
|
+
Classifier: Topic :: Software Development
|
|
19
|
+
Classifier: Topic :: System :: Distributed Computing
|
|
20
|
+
Classifier: Typing :: Typed
|
|
21
|
+
Provides-Extra: testing
|
|
22
|
+
Requires-Dist: asyncpg (>=0.31)
|
|
23
|
+
Requires-Dist: croniter (>=2.0)
|
|
24
|
+
Requires-Dist: forktex (>=0.14,<0.15)
|
|
25
|
+
Requires-Dist: pydantic (>=2.0)
|
|
26
|
+
Requires-Dist: sqlalchemy[asyncio] (>=2.0)
|
|
27
|
+
Requires-Dist: testcontainers[postgres] (>=4.0) ; extra == "testing"
|
|
28
|
+
Project-URL: Bug Tracker, https://github.com/forktex/forktex-flow/issues
|
|
29
|
+
Project-URL: Changelog, https://github.com/forktex/forktex-flow/blob/master/CHANGELOG.md
|
|
30
|
+
Project-URL: Documentation, https://github.com/forktex/forktex-flow/tree/master/docs
|
|
31
|
+
Project-URL: Homepage, https://forktex.com
|
|
32
|
+
Project-URL: Repository, https://github.com/forktex/forktex-flow
|
|
33
|
+
Description-Content-Type: text/markdown
|
|
34
|
+
|
|
35
|
+
# forktex-flow
|
|
36
|
+
|
|
37
|
+
Durable workflows in Postgres. No broker, no scheduler process, no separate
|
|
38
|
+
control plane — the database you already run, plus your own worker processes, is
|
|
39
|
+
the whole infrastructure.
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
pip install forktex-flow
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
Python 3.14, PostgreSQL 14+. Built on the
|
|
46
|
+
[`forktex`](https://github.com/forktex/forktex-py) substrate, which pip resolves
|
|
47
|
+
for you.
|
|
48
|
+
|
|
49
|
+
## What it does
|
|
50
|
+
|
|
51
|
+
You write ordinary async functions. The library makes them survive a crash.
|
|
52
|
+
|
|
53
|
+
```python
|
|
54
|
+
import operator
|
|
55
|
+
from typing import Annotated, TypedDict
|
|
56
|
+
|
|
57
|
+
from forktex_flow import Ctx, Flow, step
|
|
58
|
+
|
|
59
|
+
flow = Flow(database_url="postgresql+asyncpg://user:pass@host/db")
|
|
60
|
+
|
|
61
|
+
|
|
62
|
+
class DeployState(TypedDict):
|
|
63
|
+
logs: Annotated[list[str], operator.add] # accumulates across steps
|
|
64
|
+
server_id: str # last write wins
|
|
65
|
+
dns_enabled: bool
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
async def provision(ctx: Ctx, state: DeployState) -> dict:
|
|
69
|
+
server = await hetzner.create_server()
|
|
70
|
+
return {"server_id": server.id, "logs": [f"provisioned {server.id}"]}
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
async def configure_dns(ctx: Ctx, state: DeployState) -> dict:
|
|
74
|
+
await dns.point(state["server_id"])
|
|
75
|
+
return {"logs": ["dns configured"]}
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
@flow.pipeline("deploy.apply", version=1, state=DeployState)
|
|
79
|
+
class DeployApply:
|
|
80
|
+
steps = [provision, step(configure_dns, when=lambda s: s["dns_enabled"])]
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Start a worker and dispatch — Postgres must be running and reachable at that DSN:
|
|
84
|
+
|
|
85
|
+
```python
|
|
86
|
+
await flow.start_driver() # migrates the schema, then competes for leadership
|
|
87
|
+
|
|
88
|
+
instance = await flow.run("deploy.apply", state={"dns_enabled": True, "logs": []})
|
|
89
|
+
final = await instance.wait(timeout=600)
|
|
90
|
+
print(final.status, final.state["logs"])
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
If the worker dies after `provision` and before `configure_dns`, another worker
|
|
94
|
+
picks the run up, `provision` is **not** re-run — its recorded output is
|
|
95
|
+
returned — and `configure_dns` executes for the first time. No second server
|
|
96
|
+
gets created.
|
|
97
|
+
|
|
98
|
+
## The model, in four sentences
|
|
99
|
+
|
|
100
|
+
A workflow is an **ordered list of steps**. A step returns the *delta* it wants
|
|
101
|
+
merged into the run's state, not the whole state. Every step is memoised on
|
|
102
|
+
`(run_id, step, input state)`, so a run picked up after a crash re-executes from
|
|
103
|
+
the top and skips past work that already happened. There is no graph, no router
|
|
104
|
+
and no signal inbox: a run visits every step once, in order, executing the ones
|
|
105
|
+
whose guard passes.
|
|
106
|
+
|
|
107
|
+
Less than a general workflow engine offers, and what a deploy pipeline, a nightly
|
|
108
|
+
reconcile or an ingest job actually needs.
|
|
109
|
+
|
|
110
|
+
## Installing
|
|
111
|
+
|
|
112
|
+
```bash
|
|
113
|
+
pip install forktex-flow
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
To test your own workflows, take the extra — it brings in `testcontainers`:
|
|
117
|
+
|
|
118
|
+
```bash
|
|
119
|
+
pip install "forktex-flow[testing]"
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Install with Python 3.14.** `forktex` and `forktex-flow` both declare
|
|
123
|
+
`requires-python = ">=3.14"`, and pip hides a release whose floor your
|
|
124
|
+
interpreter does not meet rather than saying why — so an older interpreter
|
|
125
|
+
reports an unrelated, much older lineage and the dependency looks
|
|
126
|
+
unsatisfiable. It is not; the interpreter is.
|
|
127
|
+
|
|
128
|
+
Developing against a local checkout of the substrate:
|
|
129
|
+
|
|
130
|
+
```bash
|
|
131
|
+
pip install -e ../forktex-py
|
|
132
|
+
pip install -e .
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## Wiring it into your app
|
|
136
|
+
|
|
137
|
+
The library runs inside a process you already have. In FastAPI that is the
|
|
138
|
+
lifespan.
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from contextlib import asynccontextmanager
|
|
142
|
+
|
|
143
|
+
from fastapi import FastAPI
|
|
144
|
+
from forktex_flow import Flow
|
|
145
|
+
|
|
146
|
+
flow = Flow(database_url=settings.database_url)
|
|
147
|
+
import myapp.workflows # noqa: F401 — importing runs the @flow.pipeline decorators
|
|
148
|
+
|
|
149
|
+
|
|
150
|
+
@asynccontextmanager
|
|
151
|
+
async def lifespan(app: FastAPI):
|
|
152
|
+
await flow.start_driver() # migrates, then competes for the leader lock
|
|
153
|
+
yield
|
|
154
|
+
+
|
|
155
|
+
await flow.close() # stops driving, releases the pool
|
|
156
|
+
|
|
157
|
+
|
|
158
|
+
app = FastAPI(lifespan=lifespan)
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
Run this on **every** replica. Exactly one wins the advisory lock and executes;
|
|
162
|
+
the rest stand by and take over if it dies. Workflows must be imported before
|
|
163
|
+
`start_driver()` — a decorator only runs when its module does.
|
|
164
|
+
|
|
165
|
+
`start_driver()` calls `init()` for you. Call `init()` alone when you want the
|
|
166
|
+
schema migrated by something that does not execute anything, such as a deploy
|
|
167
|
+
hook.
|
|
168
|
+
|
|
169
|
+
## Every option you have
|
|
170
|
+
|
|
171
|
+
The sections below are the whole consumer surface, in the order you meet it:
|
|
172
|
+
declare a workflow, build a `Flow`, run it, watch it, tune it. Exact semantics
|
|
173
|
+
for each are in [`docs/flow.md`](docs/flow.md); failure behaviour is in
|
|
174
|
+
[`docs/operating.md`](docs/operating.md).
|
|
175
|
+
|
|
176
|
+
### 1. Declaring a workflow
|
|
177
|
+
|
|
178
|
+
Two forms, and nothing else:
|
|
179
|
+
|
|
180
|
+
| Form | Declares |
|
|
181
|
+
|---|---|
|
|
182
|
+
| `@flow.pipeline(name, *, version, state=None, cron=None)` | A class with a `steps` list. `cron=` also schedules it. |
|
|
183
|
+
| `@flow.scheduled(name, *, version, cron, state=None, enabled=True)` | One async function on a cron. |
|
|
184
|
+
|
|
185
|
+
`enabled=False` registers the schedule without arming its timer. The flag is
|
|
186
|
+
read only when the row is first written; the database owns it afterwards, so no
|
|
187
|
+
deploy re-arms or disarms a schedule behind an operator.
|
|
188
|
+
|
|
189
|
+
### 2. Declaring a step
|
|
190
|
+
|
|
191
|
+
A step is `async def (ctx: Ctx, state: YourState) -> dict`. Bare, it takes the
|
|
192
|
+
engine's retry defaults. Wrapped in `step(...)`, it takes its own:
|
|
193
|
+
|
|
194
|
+
| Option | Effect |
|
|
195
|
+
|---|---|
|
|
196
|
+
| `step(fn, when=predicate)` | Run only if `predicate(state)` is true. Evaluated immediately before dispatch. |
|
|
197
|
+
| `step(fn, max_attempts=n)` | Attempts for this step, overriding `RuntimeConfig.max_attempts`. |
|
|
198
|
+
| `step(fn, backoff=(a, b, c))` | Seconds between attempts. The last entry repeats once attempts run past the tuple. |
|
|
199
|
+
| `@step(max_attempts=…, backoff=…)` | The same as a decorator, when the step needs no `when=`. |
|
|
200
|
+
|
|
201
|
+
### 3. Declaring state
|
|
202
|
+
|
|
203
|
+
A `TypedDict`. A step returns the **delta** it wants merged, not the whole
|
|
204
|
+
state. A field annotated with a reducer accumulates; everything else is
|
|
205
|
+
last-write-wins:
|
|
206
|
+
|
|
207
|
+
```python
|
|
208
|
+
class DeployState(TypedDict):
|
|
209
|
+
logs: Annotated[list[str], operator.add] # merged with operator.add
|
|
210
|
+
server_id: str # overwritten
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
State must be JSON-serialisable — it is the memoisation key and it round-trips
|
|
214
|
+
through JSONB. A value JSONB cannot take raises `StateNotSerialisable`.
|
|
215
|
+
|
|
216
|
+
### 4. Building a `Flow`
|
|
217
|
+
|
|
218
|
+
```python
|
|
219
|
+
Flow(database_url=None, *, database=None, schema="forktex_flow",
|
|
220
|
+
extensions=None, config=None, echo=False, **engine_kwargs)
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
| Option | Default | What it does |
|
|
224
|
+
|---|---|---|
|
|
225
|
+
| `database_url` | — | A DSN. The pool is this `Flow`'s and `close()` disposes it. |
|
|
226
|
+
| `database` | — | A `forktex` `Database` you own. `close()` leaves it alone. Exactly one of these two. |
|
|
227
|
+
| `schema` | `forktex_flow` | The Postgres schema. Two `Flow`s on different schemas share nothing. |
|
|
228
|
+
| `extensions` | `None` | `FlowExtension` objects, called on lifecycle transitions. |
|
|
229
|
+
| `config` | `RuntimeConfig()` | Retry, polling and concurrency behaviour. |
|
|
230
|
+
| `echo` | `False` | SQLAlchemy statement logging. |
|
|
231
|
+
| `**engine_kwargs` | — | Passed verbatim to `create_async_engine` (`pool_size`, `pool_pre_ping`, …). |
|
|
232
|
+
|
|
233
|
+
Construction is lazy; nothing connects until something executes.
|
|
234
|
+
|
|
235
|
+
### 5. Lifecycle
|
|
236
|
+
|
|
237
|
+
| Call | When |
|
|
238
|
+
|---|---|
|
|
239
|
+
| `await flow.init()` | Create the schema and apply migrations. Idempotent, and safe from every worker at once. |
|
|
240
|
+
| `await flow.verify()` | Probe the pool, creating nothing. For a readiness check. |
|
|
241
|
+
| `await flow.start_driver()` | Begin competing for leadership. Returns at once. |
|
|
242
|
+
| `await flow.stop_driver()` | Stop cleanly: waits up to 30s for in-flight runs, then cancels what is left mid-step. |
|
|
243
|
+
| `await flow.close()` | `stop_driver()`, then dispose the pool if this `Flow` made it. Every entry point raises `FlowClosed` after. |
|
|
244
|
+
|
|
245
|
+
`apply_migrations(engine, schema)` brings a schema up without building a `Flow`
|
|
246
|
+
— for a migration job that does not run workflows.
|
|
247
|
+
|
|
248
|
+
### 6. Starting a run
|
|
249
|
+
|
|
250
|
+
```python
|
|
251
|
+
instance = await flow.run("deploy.apply", state={"logs": []}, metadata={"actor": "ci"})
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
| Argument | Effect |
|
|
255
|
+
|---|---|
|
|
256
|
+
| `name` | The registered workflow name. |
|
|
257
|
+
| `state` | The initial state. Becomes `run.input`, which a resume replays from. |
|
|
258
|
+
| `metadata` | Arbitrary JSON, queryable with `.metadata(**kv)` against a GIN index. |
|
|
259
|
+
| `version` | A specific version. Omitted, the highest registered one. |
|
|
260
|
+
|
|
261
|
+
There is no idempotency key: calling `run()` twice queues two runs.
|
|
262
|
+
`flow.registered()` lists every name declared in this process.
|
|
263
|
+
|
|
264
|
+
### 7. Watching one run
|
|
265
|
+
|
|
266
|
+
`run()` and `flow.get(run_id)` both return a `WorkflowInstance` — a snapshot
|
|
267
|
+
plus four calls:
|
|
268
|
+
|
|
269
|
+
| Call | Returns |
|
|
270
|
+
|---|---|
|
|
271
|
+
| `await instance.wait(timeout=None)` | The snapshot once terminal, or at timeout. A timeout is not an error. |
|
|
272
|
+
| `await instance.refresh()` | A fresh snapshot now. |
|
|
273
|
+
| `async for snap in instance.stream()` | A snapshot per change, over `LISTEN`/`NOTIFY`. |
|
|
274
|
+
| `await instance.cancel()` | `True` if it took effect; `False` if the run had already finished. |
|
|
275
|
+
|
|
276
|
+
Its fields: `instance_id`, `workflow_name`, `workflow_version`, `status`,
|
|
277
|
+
`state`, `metadata`, `current_node`, `started_at`, `finished_at`, and `nodes` —
|
|
278
|
+
a `NodeInstance` per step with `name`, `status`, `attempt`, `started_at`,
|
|
279
|
+
`finished_at`, `duration`, `state_delta`, `error`.
|
|
280
|
+
|
|
281
|
+
### 8. Finding runs
|
|
282
|
+
|
|
283
|
+
`flow.query()` returns an `InstanceQuery`. Filters chain and **replace** on
|
|
284
|
+
repeat, except `.metadata()` and `.state()`, which accumulate:
|
|
285
|
+
|
|
286
|
+
| Filter | Matches |
|
|
287
|
+
|---|---|
|
|
288
|
+
| `.workflow(name, version=None)` | One workflow, optionally one version. |
|
|
289
|
+
| `.status(*statuses)` | Runs in any of these statuses. |
|
|
290
|
+
| `.triggered_by(*triggers)` | `manual`, `schedule`, or both. |
|
|
291
|
+
| `.metadata(**kv)` | `metadata @> {...}` containment. |
|
|
292
|
+
| `.state(**kv)` | `input @> {...}` containment. |
|
|
293
|
+
| `.current_node(*names)` | Runs whose running step is one of these. |
|
|
294
|
+
| `.since(dt)` / `.until(dt)` | Started at/after, or before. |
|
|
295
|
+
| `.limit(n)` | Page size. Default 50. |
|
|
296
|
+
|
|
297
|
+
And four terminals:
|
|
298
|
+
|
|
299
|
+
| Terminal | Returns |
|
|
300
|
+
|---|---|
|
|
301
|
+
| `await q.fetch(cursor=None)` | An `InstancePage`: `items`, `has_more`, `next_cursor`, `total`. Keyset-paged. |
|
|
302
|
+
| `await q.first()` | The newest match, or `None`. |
|
|
303
|
+
| `await q.count()` | How many match, without fetching them. |
|
|
304
|
+
| `await q.summary()` | A `RunSummary`: `total`, `by_status`, `avg_duration_seconds`, `p95_duration_seconds`, `oldest_started_at`, `newest_started_at`. |
|
|
305
|
+
|
|
306
|
+
### 9. Inside a step
|
|
307
|
+
|
|
308
|
+
The `Ctx` a step is handed carries `run_id`, `workflow_name`,
|
|
309
|
+
`workflow_version`, `node_name`, `attempt` and `metadata`, plus two calls:
|
|
310
|
+
|
|
311
|
+
| Call | Effect |
|
|
312
|
+
|---|---|
|
|
313
|
+
| `await ctx.emit(event_type, payload=None)` | Append a row to `run_event` — progress from inside a long step. |
|
|
314
|
+
| `await ctx.sleep(seconds)` | Sleep. **Not durable**: a worker dying mid-sleep restarts the run and the sleep begins again. |
|
|
315
|
+
|
|
316
|
+
### 10. Tuning the engine
|
|
317
|
+
|
|
318
|
+
```python
|
|
319
|
+
Flow(url, config=RuntimeConfig(poll_interval=0.5, max_concurrent_runs=25))
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
| Field | Default | Controls |
|
|
323
|
+
|---|---|---|
|
|
324
|
+
| `poll_interval` | `1.0` | Seconds between driver ticks while it leads. |
|
|
325
|
+
| `election_interval` | `2.0` | Seconds a loser waits before standing again — the bound on takeover time. |
|
|
326
|
+
| `heartbeat_interval` | `10.0` | Seconds between heartbeat writes on a running step and run. |
|
|
327
|
+
| `stale_threshold` | `60.0` | Silence after which a step or run is reclaimed. Must exceed `heartbeat_interval`. |
|
|
328
|
+
| `max_attempts` | `3` | Default attempts per step. |
|
|
329
|
+
| `max_concurrent_runs` | `10` | Runs one worker executes at once. |
|
|
330
|
+
| `backoff` | `(30.0, 120.0, 300.0)` | Default seconds between attempts. |
|
|
331
|
+
| `leader_lock_key` | `0` | Advisory lock key. Derived from the schema when left at `0`. |
|
|
332
|
+
|
|
333
|
+
### 11. Reacting to runs
|
|
334
|
+
|
|
335
|
+
A `FlowExtension` is any object with one or more of these. A failure in a hook
|
|
336
|
+
is logged and never changes the run's outcome:
|
|
337
|
+
|
|
338
|
+
| Hook | Called | Can |
|
|
339
|
+
|---|---|---|
|
|
340
|
+
| `before_start(name, version, input, metadata)` | Before a run is inserted | Return a dict merged into its metadata. |
|
|
341
|
+
| `after_complete(run, output)` | Once terminal `completed` | Observe. `run` is the run as it ended. |
|
|
342
|
+
| `after_fail(run, error)` | Once terminal `failed` or `cancelled` | Observe. |
|
|
343
|
+
|
|
344
|
+
### 12. Testing your workflows
|
|
345
|
+
|
|
346
|
+
| Helper | Gives you |
|
|
347
|
+
|---|---|
|
|
348
|
+
| `make_ctx(**overrides)` | A `Ctx` with no database, to await a step directly. `ctx.emit()` raises rather than dropping an event. |
|
|
349
|
+
| `isolated_flow(url)` | An async context manager: a migrated `Flow` on a throwaway schema, dropped after. |
|
|
350
|
+
| `run_to_completion(flow, name, state=…)` | Dispatch and execute a run inline, with no driver. |
|
|
351
|
+
| `apply_state_update(current, update, reducers)` · `extract_reducers(cls)` | The reducer machinery, to test your own state classes. |
|
|
352
|
+
|
|
353
|
+
```python
|
|
354
|
+
async with isolated_flow(postgres_url) as flow:
|
|
355
|
+
register_my_workflows(flow)
|
|
356
|
+
final = await run_to_completion(flow, "deploy.apply", state={"logs": []})
|
|
357
|
+
assert final.state["logs"] == ["provisioned", "dns configured"]
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
### 13. Reading the engine's own tables
|
|
361
|
+
|
|
362
|
+
`flow.session()` is public — tailing `run_event` for a progress stream needs no
|
|
363
|
+
second pool. `flow.schema` and `flow.engine` are there for DDL, advisory locks
|
|
364
|
+
and migrations. The engine owns the transaction, so `commit()` raises.
|
|
365
|
+
|
|
366
|
+
## Errors
|
|
367
|
+
|
|
368
|
+
Everything this library raises derives from `forktex.error.AppError`, so a
|
|
369
|
+
transport that renders an `AppError` renders these with the right code rather
|
|
370
|
+
than a masked 500. The table of what is raised when is in
|
|
371
|
+
[`docs/flow.md`](docs/flow.md#errors).
|
|
372
|
+
|
|
373
|
+
## Stability
|
|
374
|
+
|
|
375
|
+
**The public surface is `forktex_flow.__all__`** — 28 names, asserted as an exact
|
|
376
|
+
set by `tests/test_architecture/test_public_surface.py`. Adding or removing one
|
|
377
|
+
is a deliberate edit to that test. Anything reachable but not in `__all__` is
|
|
378
|
+
internal and may change in any release.
|
|
379
|
+
|
|
380
|
+
**This is 0.x, but not a free-for-all.** A public name is removed only after one
|
|
381
|
+
minor release in which it still works and its docstring says it is deprecated.
|
|
382
|
+
Behaviour changes a correct consumer would notice are called out under *Changed*
|
|
383
|
+
in `CHANGELOG.md`.
|
|
384
|
+
|
|
385
|
+
**`FlowExtension`'s hook signatures are frozen.** New lifecycle points arrive as
|
|
386
|
+
new optional methods, never as new parameters on existing ones — so an extension
|
|
387
|
+
written today keeps working.
|
|
388
|
+
|
|
389
|
+
**Migrations are forward-only, and an applied one is never edited.** A schema
|
|
390
|
+
change is a new `v{NNNN}`; `tests/test_persist/test_migration_immutability.py`
|
|
391
|
+
pins the content hash of every shipped migration, and
|
|
392
|
+
`test_schema_drift.py` proves the ORM and the SQL describe the same columns.
|
|
393
|
+
|
|
394
|
+
**Two things outside the Python API are also contracts:** the `forktex_flow`
|
|
395
|
+
schema name, and `run_event(id, ts, event_type, payload, run_id)`, which
|
|
396
|
+
consumers read directly by SQL without importing this package.
|
|
397
|
+
|
|
398
|
+
**Absent by design:** graph workflows,
|
|
399
|
+
`step_template`, config-defined workflows, signals, `parallel`, and child
|
|
400
|
+
workflows. Each was measured to have zero call sites before it was cut; see
|
|
401
|
+
*Removed* in `CHANGELOG.md`.
|
|
402
|
+
|
|
403
|
+
## Documentation
|
|
404
|
+
|
|
405
|
+
[`docs/flow.md`](docs/flow.md) — the reference: every public name, the API and
|
|
406
|
+
the gotchas.
|
|
407
|
+
[`docs/operating.md`](docs/operating.md) — durability and cancellation
|
|
408
|
+
semantics, leadership, what to watch, and the schema.
|
|
409
|
+
[`docs/development.md`](docs/development.md) — contributing.
|
|
410
|
+
|
|
411
|
+
## Licence
|
|
412
|
+
|
|
413
|
+
AGPL-3.0-or-later, or a commercial licence from FORKTEX S.R.L.
|
|
414
|
+
(info@forktex.com). See `LICENSE` and `NOTICE`.
|
|
415
|
+
|