toro-queue 0.2.0__tar.gz → 0.4.0__tar.gz

{toro_queue-0.2.0 → toro_queue-0.4.0}/.github/workflows/pr-check.yaml

@@ -9,7 +9,7 @@ on:
   pull_request:
     types: [opened, synchronize, reopened]
 
-# Read-only by default — nothing here writes to the repo. SonarCloud PR
+# Read-only by default - nothing here writes to the repo. SonarCloud PR
 # decoration comes from the SonarCloud GitHub App, not GITHUB_TOKEN write scopes.
 permissions:
   contents: read
@@ -68,7 +68,7 @@ jobs:
         run: uv run pytest -m "unit or integration" --cov=toro --cov-report=xml
 
       # Runs only when SONAR_TOKEN is set (skipped on forks / before setup, so the
-      # check stays green), and only once per matrix — one coverage upload.
+      # check stays green), and only once per matrix - one coverage upload.
       # Config is passed inline; there is no sonar-project.properties.
       - name: SonarCloud scan
         if: matrix.python-version == '3.13' && env.SONAR_TOKEN

{toro_queue-0.2.0 → toro_queue-0.4.0}/.github/workflows/release.yml

@@ -1,7 +1,7 @@
 name: Release
 
 # A version tag (v*) builds and publishes to PyPI via trusted publishing (OIDC)
-# — no API tokens stored. The publish job runs in the `pypi` environment, which
+# - no API tokens stored. The publish job runs in the `pypi` environment, which
 # must match the trusted publisher registered on PyPI.
 on:
   push:

toro_queue-0.4.0/.pre-commit-config.yaml

@@ -0,0 +1,26 @@
+# All hooks run through `uv run`, so versions come from the lockfile - one
+# source of truth shared by pre-commit, CI and dev, nothing to drift.
+# Install once with:  uvx pre-commit install
+repos:
+  - repo: local
+    hooks:
+      - id: ruff-check
+        name: ruff check
+        entry: uv run ruff check --force-exclude
+        language: system
+        types_or: [python, pyi]
+        require_serial: true
+
+      - id: ruff-format
+        name: ruff format
+        entry: uv run ruff format --check --force-exclude
+        language: system
+        types_or: [python, pyi]
+        require_serial: true
+
+      - id: ty
+        name: ty type check
+        entry: uv run ty check
+        language: system
+        pass_filenames: false
+        types: [python]

{toro_queue-0.2.0 → toro_queue-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: toro-queue
-Version: 0.2.0
+Version: 0.4.0
 Summary: An async-first, Redis-backed job queue for Python.
 Project-URL: Homepage, https://github.com/ilovepixelart/toro
 Project-URL: Repository, https://github.com/ilovepixelart/toro
@@ -55,12 +55,12 @@ Pairs with **[matador](https://github.com/ilovepixelart/matador)**, a live web d
 
 ## Why toro
 
-- **Async-native.** Enqueue and process with `async`/`await` — no thread pools,
+- **Async-native.** Enqueue and process with `async`/`await` - no thread pools,
   no sync bridge. A natural fit for FastAPI, aiohttp, or any asyncio app.
 - **Atomic by construction.** Claims, retries, promotions and finishes are Lua
   scripts, so a job can't be lost or double-committed between two round trips.
 - **At-least-once delivery.** Per-job locks + a background mark-and-sweep recover
-  jobs from workers that crashed — without the visibility-timeout double-delivery
+  jobs from workers that crashed - without the visibility-timeout double-delivery
   trap of some other queues.
 - **Typed.** Ships `py.typed`; the public API is fully annotated.
 
@@ -71,13 +71,14 @@ Pairs with **[matador](https://github.com/ilovepixelart/matador)**, a live web d
 | **Enqueue** | delayed jobs, global **priorities** (FIFO within a band) |
 | **Retries** | fixed or exponential **backoff**, capped attempts |
 | **Schedules** | repeatable **cron** and fixed-interval (`every`) jobs |
+| **Flows** | parent/child job trees: fan-out/fan-in, failure policies, flow-aware retry |
 | **Rate limiting** | queue-wide token bucket shared across all workers |
 | **Dedup** | custom (idempotent) job ids + a throttle window (`{id, ttl}`) |
 | **Auto-removal** | keep the last N and/or finished-within-age completed/failed |
 | **Reliability** | per-job locks, lock renewal, stalled-job recovery |
 | **Observability** | progress, per-job logs, lifecycle events, `await result()` |
 | **Lifecycle** | pause / resume, graceful shutdown that drains in-flight jobs |
-| **Dashboard** | [matador](https://github.com/ilovepixelart/matador) — a live web UI |
+| **Dashboard** | [matador](https://github.com/ilovepixelart/matador) - a live web UI |
 
 ## Quick start
 
@@ -113,6 +114,11 @@ await queue.add("charge", data, job_id="order-1234")
 # A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
 await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")
 
+# A flow: children run first (fan-out), the parent runs on their results (fan-in)
+from toro import FlowChild as c
+report = await queue.add_flow("report", {"q": 3},
+                              children=[c("fetch", {"shard": i}) for i in range(3)])
+
 # Queue-wide rate limit: at most 100 jobs / second across every worker
 worker = Worker("emails", process, rate_limit={"max": 100, "duration": 1000})
 
@@ -121,6 +127,15 @@ job = await queue.add("resize", {"src": "a.png"})
 print(await job.result(timeout=30))
 ```
 
+## Flows
+
+A flow enqueues a parent and its children as one atomic tree. The children run
+first (fan-out, nested arbitrarily); the parent parks until every child has
+settled, then runs and reads their results (fan-in). One primitive covers
+fan-out/fan-in and chained steps, with per-child failure policies and
+flow-aware retry that recovers a whole failed flow in one shot. Full guide:
+[docs/flows.md](docs/flows.md).
+
 ## Develop
 
 Managed with [uv](https://astral.sh/uv); the Astral toolchain throughout.
@@ -134,7 +149,7 @@ uv run pytest -m "unit or integration"   # tests (integration needs Redis on :63
 uv run python examples/basic.py
 ```
 
-The suite is a pyramid — `-m unit` (fast, no Redis), `-m integration` (Redis),
+The suite is a pyramid - `-m unit` (fast, no Redis), `-m integration` (Redis),
 and `-m load` (the open-loop benchmark harness in `tests/load/`).
 
 ## License

{toro_queue-0.2.0 → toro_queue-0.4.0}/README.md

@@ -28,12 +28,12 @@ Pairs with **[matador](https://github.com/ilovepixelart/matador)**, a live web d
 
 ## Why toro
 
-- **Async-native.** Enqueue and process with `async`/`await` — no thread pools,
+- **Async-native.** Enqueue and process with `async`/`await` - no thread pools,
   no sync bridge. A natural fit for FastAPI, aiohttp, or any asyncio app.
 - **Atomic by construction.** Claims, retries, promotions and finishes are Lua
   scripts, so a job can't be lost or double-committed between two round trips.
 - **At-least-once delivery.** Per-job locks + a background mark-and-sweep recover
-  jobs from workers that crashed — without the visibility-timeout double-delivery
+  jobs from workers that crashed - without the visibility-timeout double-delivery
   trap of some other queues.
 - **Typed.** Ships `py.typed`; the public API is fully annotated.
 
@@ -44,13 +44,14 @@ Pairs with **[matador](https://github.com/ilovepixelart/matador)**, a live web d
 | **Enqueue** | delayed jobs, global **priorities** (FIFO within a band) |
 | **Retries** | fixed or exponential **backoff**, capped attempts |
 | **Schedules** | repeatable **cron** and fixed-interval (`every`) jobs |
+| **Flows** | parent/child job trees: fan-out/fan-in, failure policies, flow-aware retry |
 | **Rate limiting** | queue-wide token bucket shared across all workers |
 | **Dedup** | custom (idempotent) job ids + a throttle window (`{id, ttl}`) |
 | **Auto-removal** | keep the last N and/or finished-within-age completed/failed |
 | **Reliability** | per-job locks, lock renewal, stalled-job recovery |
 | **Observability** | progress, per-job logs, lifecycle events, `await result()` |
 | **Lifecycle** | pause / resume, graceful shutdown that drains in-flight jobs |
-| **Dashboard** | [matador](https://github.com/ilovepixelart/matador) — a live web UI |
+| **Dashboard** | [matador](https://github.com/ilovepixelart/matador) - a live web UI |
 
 ## Quick start
 
@@ -86,6 +87,11 @@ await queue.add("charge", data, job_id="order-1234")
 # A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
 await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")
 
+# A flow: children run first (fan-out), the parent runs on their results (fan-in)
+from toro import FlowChild as c
+report = await queue.add_flow("report", {"q": 3},
+                              children=[c("fetch", {"shard": i}) for i in range(3)])
+
 # Queue-wide rate limit: at most 100 jobs / second across every worker
 worker = Worker("emails", process, rate_limit={"max": 100, "duration": 1000})
 
@@ -94,6 +100,15 @@ job = await queue.add("resize", {"src": "a.png"})
 print(await job.result(timeout=30))
 ```
 
+## Flows
+
+A flow enqueues a parent and its children as one atomic tree. The children run
+first (fan-out, nested arbitrarily); the parent parks until every child has
+settled, then runs and reads their results (fan-in). One primitive covers
+fan-out/fan-in and chained steps, with per-child failure policies and
+flow-aware retry that recovers a whole failed flow in one shot. Full guide:
+[docs/flows.md](docs/flows.md).
+
 ## Develop
 
 Managed with [uv](https://astral.sh/uv); the Astral toolchain throughout.
@@ -107,7 +122,7 @@ uv run pytest -m "unit or integration"   # tests (integration needs Redis on :63
 uv run python examples/basic.py
 ```
 
-The suite is a pyramid — `-m unit` (fast, no Redis), `-m integration` (Redis),
+The suite is a pyramid - `-m unit` (fast, no Redis), `-m integration` (Redis),
 and `-m load` (the open-loop benchmark harness in `tests/load/`).
 
 ## License

{toro_queue-0.2.0 → toro_queue-0.4.0}/docs/architecture.md

@@ -9,12 +9,13 @@ every job is durable in Redis.
 ## Atomic state transitions via Lua
 
 Every state move (`wait→active`, `active→completed/failed/delayed`,
-`delayed→wait`) is a single Redis Lua script, run atomically, so multi-key
+`delayed→wait`, and the flow transitions around `waiting-children` -
+see [Flows](flows.md)) is a single Redis Lua script, run atomically, so multi-key
 "check-then-act" sequences can't interleave. That removes whole classes of race:
 
-- **pop-then-lock gap** — two workers claiming the same job: the claim pops from
+- **pop-then-lock gap** - two workers claiming the same job: the claim pops from
   the priority set and sets the lock inside one script.
-- **finish-after-steal** — a worker committing a result for a job a stalled sweep
+- **finish-after-steal** - a worker committing a result for a job a stalled sweep
   already re-queued: guarded by a token check plus `LREM active` returning 0.
 
 Scripts live in `scripts.py`, registered with `redis.asyncio`'s `register_script`.
@@ -23,7 +24,7 @@ The Python side only assembles KEYS/ARGV; the guarantees live in the Lua.
 ## Claiming a job: the prioritized set + a wakeup marker
 
 All waiting jobs live in one `prioritized` ZSET, scored
-`(PRIORITY_OFFSET - priority) * 2^32 + seq` — a single global order where higher
+`(PRIORITY_OFFSET - priority) * 2^32 + seq` - a single global order where higher
 priority is more urgent and ties stay FIFO (`seq` is a per-queue counter). This
 *is* the `wait` state; there is no separate fast-lane list, so a low-priority job
 can't starve a high-priority one.
@@ -74,20 +75,24 @@ due jobs into the prioritized set.
 
 ## Higher-level features
 
-- **Priorities** — every job is in the one prioritized ZSET above, so priority is
+- **Priorities** - every job is in the one prioritized ZSET above, so priority is
   a single global order with no starvation, FIFO within a band.
-- **Repeatable / cron** — `add_scheduler(every=ms | cron=...)` stores a template
+- **Repeatable / cron** - `add_scheduler(every=ms | cron=...)` stores a template
   and enqueues the first occurrence as a delayed job; each occurrence mints its
   successor with a deterministic id when a worker picks it up. `trigger_scheduler`
   runs one now, `remove_scheduler` stops the chain. See [Scheduling](scheduling.md).
-- **Rate limiting** — a queue-wide token bucket in Redis
+- **Rate limiting** - a queue-wide token bucket in Redis
   (`Worker(rate_limit={"max": N, "duration": ms})`), shared by every worker on the
   queue. An over-limit claim returns a sentinel and the worker waits out the window.
-- **Events** — Redis pub/sub on an `events` channel (`added`, `progress`,
+- **Events** - Redis pub/sub on an `events` channel (`added`, `progress`,
   `completed`, `failed`); `Queue.result()` awaits the terminal event and
   `Worker.on(event, fn)` exposes in-process hooks. See [Concepts](concepts.md).
-- **Auto-removal** — `remove_on_complete` / `remove_on_fail` (bool / count /
+- **Auto-removal** - `remove_on_complete` / `remove_on_fail` (bool / count /
   `{count, age}`) enforced inside the finish script, not by a separate sweeper.
+- **Flows** - `add_flow()` creates a parent/child tree atomically; children
+  settle into the parent's `:deps` barrier inside the same finish scripts that
+  commit their own transitions, so the fan-in resolves on the crash path too.
+  See [Flows](flows.md).
 
 ## The Lua scripts
 
@@ -104,40 +109,44 @@ The scripts share a small library of routines:
 | `acquireNext` | Pops the top prioritized job into `active` and locks it, honoring the rate limit. |
 | `tryRateLimit` | Token bucket: ms until a token frees, or 0 to proceed. |
 | `recordFinished` | Records a terminal job in `completed`/`failed` and applies auto-removal. |
+| `settleChildCompleted` / `settleChildFailed` / `releaseParent` | A finishing flow child settles into its parent's `:deps` barrier; the last one releases the parent - or fails it eagerly, per `on_fail`. |
+| `keepArgsFromOpts` | The Lua twin of `JobOptions.keep_args`, for eager parent failures that have no Python caller. |
 
 And the scripts themselves:
 
 | Script | Caller | Does |
 |---|---|---|
 | `ADD_JOB` | producer | Mint/accept an id, write the hash, enqueue or delay, dedup, publish `added`. |
+| `ADD_FLOW` | producer | Create a whole flow tree atomically: leaves enqueued/delayed, parents parked with their `:deps` barrier. |
 | `MOVE_TO_ACTIVE` | worker wakeup | Claim the next job: `ZPOPMIN prioritized` → `active` → lock + load. |
-| `MOVE_TO_COMPLETED` | worker finish | Commit the result and fetch-next in one round trip. |
-| `MOVE_TO_FAILED` | worker finish | Retry (to `wait`/`delayed`) or terminally fail, and fetch-next. |
+| `MOVE_TO_COMPLETED` | worker finish | Commit the result (settling a flow child into its parent) and fetch-next in one round trip. |
+| `MOVE_TO_FAILED` | worker finish | Retry (to `wait`/`delayed`) or terminally fail (applying a flow child's `on_fail`), and fetch-next. |
 | `EXTEND_LOCK` | renewer | Token-guarded lock renewal; clears the job from `stalled`. |
 | `MOVE_STALLED` | sweep | Mark-and-sweep recovery of jobs whose lock expired. |
 | `PROMOTE_DELAYED` | promote loop | Move up to `PROMOTE_BATCH` (1000) due delayed jobs to `prioritized`. |
 | `ADD_SCHEDULED` | scheduler | Enqueue a scheduler occurrence under a deterministic id (idempotent). |
-| `PROMOTE_JOB` / `RETRY_JOB` / `REMOVE_JOB` | dashboard | Run a delayed job now / re-enqueue a failed one / delete a job with its lock and logs. |
+| `PROMOTE_JOB` / `RETRY_JOB` / `REMOVE_JOB` | dashboard | Run a delayed job now / re-enqueue a failed one (flow-aware: a parent with unsettled children re-parks, a child re-joins the barrier) / delete a job with its lock, logs and flow aux keys (a flow parent takes its subtree). |
 
 ### Lua → Python return protocol
 
 Scripts signal outcomes with sentinels the worker decodes:
 
-- `RL_SENTINEL` (`"__rl__"`) — a claim hit the rate limiter; the second value is
+- `RL_SENTINEL` (`"__rl__"`) - a claim hit the rate limiter; the second value is
   ms until a token frees, so the worker waits instead of busy-spinning.
-- `LOCK_LOST` (`-2`) — a finish ran but the worker no longer held the lock (the
+- `LOCK_LOST` (`-2`) - a finish ran but the worker no longer held the lock (the
   job was reclaimed); the result is dropped.
-- `NOT_ACTIVE` (`-3`) — a finish ran but the job was no longer in `active`.
-- `OUTCOME_FAILED` (`1`) vs `0` — `MOVE_TO_FAILED` telling the worker whether the
+- `NOT_ACTIVE` (`-3`) - a finish ran but the job was no longer in `active`.
+- `OUTCOME_FAILED` (`1`) vs `0` - `MOVE_TO_FAILED` telling the worker whether the
   job terminally failed or will retry.
 
 Scores are packed under 2^53 (`PRIORITY_OFFSET = 2^20`, `SEQ_MOD = 2^32`) so ZSET
-double scores stay exact, and the scripts use only plain JSON and integer ARGV —
-no `cmsgpack` / `bit` / `cjson` — so they run on any Redis build.
+double scores stay exact, and the scripts use only plain JSON and integer ARGV with the built-in
+`cjson` for encode/decode - no `cmsgpack` / `bit` - so they run on any
+Redis build.
 
 ## Python-specific choices
 
-- **async-first** — `redis.asyncio`, `async def` processors, one event loop;
+- **async-first** - `redis.asyncio`, `async def` processors, one event loop;
   concurrency is N `asyncio` tasks sharing the loop.
-- **Cluster** — a `{braces}` hash-tag in the prefix keeps all of a queue's keys on
+- **Cluster** - a `{braces}` hash-tag in the prefix keeps all of a queue's keys on
   one slot, which the multi-key Lua scripts require.

{toro_queue-0.2.0 → toro_queue-0.4.0}/docs/concepts.md

@@ -20,7 +20,7 @@ toro has a clean producer/consumer split, and both talk to the same Redis.
   bookkeeping the system fills in: `state`, `attempts_made`, timestamps
   (`timestamp`, `processed_on`, `finished_on`), `progress`, `stacktrace`, and
   either a `returnvalue` or a `failed_reason`. (A job's log lines and its lock
-  live in separate Redis keys, not as fields on the `Job` — see the
+  live in separate Redis keys, not as fields on the `Job` - see the
   [data model](data-model.md).)
 
 Producers and consumers never call each other. They coordinate only through
@@ -37,15 +37,17 @@ type, `JobState`:
 | `wait` | Ready to run, waiting for a free worker. (Stored in the priority-ordered set, so "wait" and "prioritized" are the same place.) |
 | `delayed` | Scheduled for the future; not yet runnable. Promoted to `wait` when due. |
 | `active` | Claimed by a worker and currently running. |
+| `waiting-children` | A flow parent, parked until every child settles; released to `wait` by its last child. |
 | `completed` | Finished successfully; `returnvalue` holds the result. |
 | `failed` | Exhausted its retry attempts; `failed_reason` holds the error. |
 
 The normal path is `wait → active → completed`. A failure with retries left goes
 `active → wait` (or `active → delayed`, if a backoff delay applies) and tries
 again; only after the last attempt does it land in `failed`. A delayed or
-repeatable job starts in `delayed`. See [Job lifecycle](architecture.md) for the
-exact transitions and [Producing jobs](producing.md) for how delay and retries
-are configured.
+repeatable job starts in `delayed`; a flow parent starts in `waiting-children`
+(see [Flows](flows.md)). See [Job lifecycle](architecture.md) for the exact
+transitions and [Producing jobs](producing.md) for how delay and retries are
+configured.
 
 ## Workers vs. slots
 
@@ -91,7 +93,7 @@ not on a retry. Two things consume the channel:
 
 `Worker.on(event, fn)` lets a worker react to its own lifecycle with in-process
 callbacks (`completed`, `failed`, `retrying`, `stalled`, `lock-lost`,
-`rate-limited`) — separate from the pub/sub channel above. See
+`rate-limited`) - separate from the pub/sub channel above. See
 [Processing jobs](processing.md).
 
 ## Reliability in one sentence

{toro_queue-0.2.0 → toro_queue-0.4.0}/docs/data-model.md

@@ -29,6 +29,7 @@ Redis Cluster slot, which the multi-key Lua scripts require.
 | `delayed` | ZSET | Ids scored by their process-at timestamp (ms); promoted to `prioritized` when due. |
 | `completed` | ZSET | Successfully-finished ids, scored by finish time (for auto-removal + listing). |
 | `failed` | ZSET | Terminally-failed ids, scored by finish time. |
+| `waiting-children` | ZSET | Flow parents parked until their children settle, scored by enqueue time. |
 | `meta-paused` | string (flag) | Exists only while the queue is paused; workers stop claiming new jobs. |
 | `events` | pub/sub channel | Carries `added` / `progress` / `completed` / `failed`; drives `result()` and live dashboards. |
 | `limiter` | HASH | The queue-wide rate-limit token bucket (`{tokens, ts}`), shared by every worker. |
@@ -37,6 +38,8 @@ Redis Cluster slot, which the multi-key Lua scripts require.
 | `repeat` | ZSET | Scheduler id -> next-run timestamp. |
 | `workers` | ZSET | Live worker id -> last-heartbeat ms; stale entries pruned lazily on read. |
 | `departed` | LIST (capped) | Recent worker departures: graceful `stopped` or `lost` (crashed). |
+| `metrics:<minute>` | HASH | Per-minute counters (`added`/`completed`/`failed`/`ms`, per-name fields, histograms); self-expiring. |
+| `de:<dedupId>` | string (PX) | A live deduplication throttle window; holds the already-queued job's id. |
 
 ## Per-scheduler, per-worker, per-job keys
 
@@ -44,9 +47,12 @@ Redis Cluster slot, which the multi-key Lua scripts require.
 |---|---|---|
 | `repeat:<schedulerId>` | HASH | A scheduler's template: `name`, `every`/`cron`, `data`, `opts`. |
 | `worker:<workerId>` | HASH | A worker's presence record: host, pid, concurrency, current jobs, processed/failed counts, state. |
-| `<jobId>` | HASH | The job itself: `name`, `data`, `opts`, `state`, `attemptsMade`, timestamps, `returnvalue`/`failedReason`, `progress`, `stacktrace`, ... |
+| `<jobId>` | HASH | The job itself: `name`, `data`, `opts`, `state`, `attemptsMade`, timestamps, `returnvalue`/`failedReason`, `progress`, `stacktrace`, plus flow linkage on flow jobs: `parentId`/`onFail` (children), `children` (parents). |
 | `<jobId>:lock` | string (token, PX) | The per-job lock: the owning worker's token with an expiry. Only the holder may finish or renew it. |
 | `<jobId>:logs` | LIST | Log lines appended by `job.log(...)` from inside a processor. |
+| `<jobId>:deps` | SET | A flow parent's still-pending child ids - the fan-in barrier; the parent releases when it empties. |
+| `<jobId>:results` | HASH | Child id → returnvalue JSON, written as each child completes. |
+| `<jobId>:cfail` | HASH | Child id → failure reason for children failed under `on_fail="continue"`. |
 
 Note the job hash key is just `<prefix>:<name>:<jobId>` (no extra segment), so a job
 `5` on `toro:emails:` is the hash `toro:emails:5`, with `toro:emails:5:lock` and
@@ -54,9 +60,11 @@ Note the job hash key is just `<prefix>:<name>:<jobId>` (no extra segment), so a
 
 ## How the pieces connect
 
-- A job moves between `prioritized` / `active` / `delayed` / `completed` / `failed`
-  as its state changes; the move and the hash update happen in one Lua script. See
-  [Architecture](architecture.md).
+- A job moves between `prioritized` / `active` / `delayed` / `waiting-children` /
+  `completed` / `failed` as its state changes; the move and the hash update happen
+  in one Lua script. See [Architecture](architecture.md).
+- `:deps` + `:results` + `:cfail` are the flow fan-in machinery - children settle
+  into them as they finish. See [Flows](flows.md).
 - The `lock` + `stalled` keys are the at-least-once machinery. See
   [Reliability](reliability.md).
 - `repeat` + `repeat:<id>` drive [scheduling](scheduling.md); `workers` +

toro_queue-0.4.0/docs/flows-design.md

@@ -0,0 +1,192 @@
+# Flows: design notes
+
+> Status: SHIPPED. This documents the design decisions and the research behind
+> them; the user guide is [flows.md](flows.md).
+
+A flow is a parent job enqueued together with its children. Children run first
+(in parallel, nesting allowed); the parent becomes runnable only when every
+child has settled. This gives fan-out/fan-in ("fetch 10 parts, then summarize")
+and chains ("A, then B") with one primitive.
+
+## Why this design: lessons from the landscape
+
+Every mature queue eventually grew a dependency story, and each one left a
+public trail of what went wrong. Before building, we surveyed the dependency /
+workflow features of the established queues across the Node.js, Python and
+Postgres ecosystems - both how they model things and what their issue trackers
+say users keep hitting. The design below is shaped by that trail.
+
+What the survey says, distilled:
+
+1. **Fan-in barriers die at the crash boundary** unless the bookkeeping is
+   atomic with the child's settle. Counter- and callback-based barriers
+   coordinated from the client side have produced a decade of "the callback
+   never fired" bugs: a worker killed mid-job, a nested group counted wrong,
+   a result expiring before the barrier read it. toro already funnels every
+   settle through one Lua script - the barrier belongs inside it, crash path
+   included.
+2. **Implicit result injection ages badly.** Passing the previous job's return
+   value as a magic argument (first or last position, depending on the
+   framework) always grows an opt-out flag and confuses arity forever.
+   Explicit pull by handle is the shape that aged well - provided the
+   ergonomics are good; "fetch the dependency object and read `.result` off
+   it" is the unergonomic version users complain about.
+3. **"Wait forever" as a failure default is universally hated.** Two major
+   queues default to leaving the dependent/parent parked indefinitely when a
+   dependency fails terminally, silently; in both, it's the single most
+   complained-about behavior of the feature. Failure must propagate by
+   default.
+4. **Graph state lives in the datastore, not the message.** Serializing the
+   remaining workflow into message headers means bloat, no nesting, and no
+   visibility (one framework's own redesign RFC concedes exactly this). Keys
+   in the datastore are also the only way a dashboard can show a flow.
+5. **Cleanup must be structural.** The most mature implementation leaks its
+   children-results hashes (no TTL, surviving auto-removal - tens of
+   thousands of stale keys reported in production) because aux keys are
+   cleaned on a separate path from job removal. Cleanup has to ride the
+   removal paths that already exist.
+6. **Eager beats lazy on parent failure.** Failing the parent "lazily" -
+   marking it and waiting for a worker on the parent's queue to actually
+   transition it - surprises people whenever no such worker exists. The
+   parent should fail in the same atomic step as the child.
+7. **Failure policy frozen at enqueue time** (denormalized into the stored
+   job) is a recurring complaint: teams want to change their minds after the
+   flow exists. Storing the policy as a plain mutable field costs nothing.
+
+## Design
+
+### Scope and shape
+
+- **Trees, not DAGs.** A child has exactly one parent. Multi-parent is rare,
+  expensive, and even the most mature implementation declined it;
+  fan-out/fan-in plus nesting covers the real use cases.
+- **Single queue per flow (v1).** toro workers dispatch on `job.name` inside
+  one processor, so flow steps are just different names on one queue. This
+  removes the cross-queue hazard class outright (scripts reaching into other
+  queues' keys; a purged children's queue stranding parents elsewhere).
+  Cross-queue can be revisited later without breaking the model.
+- **Static shape.** The tree is declared at enqueue time. Children spawned
+  dynamically from inside a processor is the messiest corner of the systems
+  that have it, and is out of scope.
+
+### API
+
+```python
+from toro import FlowChild as c   # name, data, opts, children, on_fail
+
+parent: Job = await queue.add_flow(name, data, children=[...], **opts)
+```
+
+- `add_flow` enqueues the whole tree **atomically** (one Lua call) and returns
+  the parent `Job`. `await parent.result()` resolves when the flow does.
+- Children accept the same options as `Queue.add()` (priority, attempts,
+  backoff, delay, auto-removal), plus `on_fail` (below). Scheduler-style
+  options, custom ids and deduplication are not valid on flow nodes.
+- Inside the parent's processor, results are **pulled explicitly**:
+
+```python
+async def process(job):
+    if job.name == "report":
+        results = await job.children_results()   # {child_id: returnvalue}
+        failures = await job.failed_children()   # {child_id: failed_reason}
+```
+
+- Introspection: `await queue.get_flow(parent_id)` returns the tree
+  (`{job, children: [...]}`), for the dashboard and for users.
+
+### Failure semantics: two policies, no third
+
+Per-child `on_fail`, stored as a plain mutable field on the child's hash (so
+tooling *can* change it after enqueue - see lesson 7):
+
+| `on_fail`               | when this child terminally fails                       |
+| ----------------------- | ------------------------------------------------------ |
+| `"fail_parent"` (default) | the parent fails **immediately and eagerly**, in the same Lua script, recursively up ancestors that also default; no worker on the parent required |
+| `"continue"`            | the failure is recorded in the parent's failures hash, the dependency cleared; the parent runs once all children settle and inspects `failed_children()` |
+
+There is deliberately **no "wait indefinitely" option** (lesson 3). Retries
+still happen first: "terminally fails" means after the child's own `attempts`
+are exhausted (or it stalls past `max_stalled_count`). A child retried to
+success *after* its parent already failed does not resurrect the parent
+(documented, v1); retrying the parent re-arms its barrier instead.
+
+### Data model
+
+New job state `waiting-children` (a sixth `JobState`), backed by a per-queue
+ZSET (timestamp-scored, like `completed`/`failed`), surfaced in `counts()`,
+`get_jobs()`, `clean()` and the dashboard.
+
+Per parent job, three aux keys (joining `:lock` / `:logs`):
+
+| key             | type | content                                            |
+| --------------- | ---- | -------------------------------------------------- |
+| `{id}:deps`     | SET  | ids of children not yet settled (the barrier)      |
+| `{id}:results`  | HASH | child id → returnvalue JSON (completed children)   |
+| `{id}:cfail`    | HASH | child id → failed reason (`on_fail="continue"` children) |
+
+Plus two hash fields: `parentId` on every child; `children` (static JSON id
+list) on every parent - the deps set shrinks, the dashboard tree needs the
+full picture.
+
+A SET rather than a counter: it's idempotent under re-delivery, inspectable
+("which children is this parent still waiting on?"), and the empty-check
+(`SCARD == 0`) is the release condition. The counter-corruption bug class
+(lesson 1) is the argument against counters.
+
+### Mechanics
+
+- **`ADD_FLOW`** (new script): tree as JSON in ARGV, `cjson.decode`, walk
+  depth-first; leaves enqueue into `prioritized` (or `delayed`), interior
+  nodes land in `waiting-children` with their `:deps` set populated. Capped
+  (~1000 nodes per flow) to bound script time, like `PROMOTE_BATCH`. One
+  `added` increment of the node count and one announce (the root id) for
+  the whole tree.
+- **Release** lives inside the existing finish scripts, the extension point
+  `_LIB` reserved ("to add markers-with-delay or grouping later, we change
+  only these functions"):
+  - `MOVE_TO_COMPLETED` of a child: `HSET parent:results`, `SREM parent:deps`;
+    on empty, move the parent from `waiting-children` through the shared
+    `enqueue()` at its stored priority.
+  - `MOVE_TO_FAILED` terminal branch: apply `on_fail` - either record into
+    `:cfail` + `SREM` (+ release if last), or fail the parent now through
+    `recordFinished` (so `remove_on_fail` retention applies), publish the
+    event, and **loop upward** while the ancestor itself has a parent with
+    `fail_parent`.
+  - **The stalled path gets identical parent bookkeeping**: `MOVE_STALLED`'s
+    fail-branch runs the same settle logic (lesson 1: the crash path is where
+    barriers historically break). One pre-existing nuance: the stall-escalated
+    child itself is recorded into `failed` without `recordFinished`, so its
+    own `remove_on_fail` retention does not apply on the stall path - true
+    for all stalled jobs, not just flow children.
+- **Cleanup is structural** (lesson 5): `delJobs` and `REMOVE_JOB` know the
+  three aux keys, so every existing removal path (auto-removal
+  keepCount/keepAge, manual remove, `clean()`) deletes them for free.
+  Removing a parent removes its subtree; removing a child SREMs it from its
+  parent's deps (and releases the parent if it was the last). Settle writes
+  are guarded by a parent-exists check so a retention-trimmed parent can't
+  get orphan keys recreated by late siblings.
+- **Retry is flow-aware**: a failed parent with unsettled deps re-parks in
+  `waiting-children`; a retried child re-joins a parked parent's barrier and
+  clears its stale `:cfail` entry. `retry_all_failed()` therefore recovers a
+  whole flow in any order.
+
+### Edge cases pinned down (each has a test)
+
+- **Parent retries**: a released parent is a normal job; its own
+  `attempts`/`backoff` apply. Children are not re-run on parent retry -
+  results are already in `:results`.
+- **`remove_on_complete` on children**: allowed - the result is copied into
+  the parent's `:results` at completion, so the child hash is free to go.
+  Routine `clean("completed")` likewise never touches a pending parent's
+  collected results.
+- **Empty `children=[]`**: rejected (`ValueError`) - that's `add()`.
+- **Id rules**: flow nodes get server-side ids; custom ids on flow nodes are
+  not supported in v1 (id reuse inside trees is a documented zombie-job
+  hazard elsewhere).
+- **Progress display counts completions only** - a failed flow must never
+  read as 100% done.
+
+## Out of scope (v1)
+
+Cross-queue flows, DAGs/multi-parent, dynamic children from inside a
+processor, and child-retry-resurrects-failed-parent.