toro-queue 0.3.0__tar.gz → 0.5.0__tar.gz

{toro_queue-0.3.0 → toro_queue-0.5.0}/.gitignore

@@ -9,3 +9,5 @@ __pycache__/
 
 .coverage
 coverage.xml
+
+.DS_Store

{toro_queue-0.3.0 → toro_queue-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: toro-queue
-Version: 0.3.0
+Version: 0.5.0
 Summary: An async-first, Redis-backed job queue for Python.
 Project-URL: Homepage, https://github.com/ilovepixelart/toro
 Project-URL: Repository, https://github.com/ilovepixelart/toro
@@ -71,6 +71,7 @@ Pairs with **[matador](https://github.com/ilovepixelart/matador)**, a live web d
 | **Enqueue** | delayed jobs, global **priorities** (FIFO within a band) |
 | **Retries** | fixed or exponential **backoff**, capped attempts |
 | **Schedules** | repeatable **cron** and fixed-interval (`every`) jobs |
+| **Flows** | parent/child job trees: fan-out/fan-in, failure policies, flow-aware retry |
 | **Rate limiting** | queue-wide token bucket shared across all workers |
 | **Dedup** | custom (idempotent) job ids + a throttle window (`{id, ttl}`) |
 | **Auto-removal** | keep the last N and/or finished-within-age completed/failed |
@@ -113,6 +114,11 @@ await queue.add("charge", data, job_id="order-1234")
 # A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
 await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")
 
+# A flow: children run first (fan-out), the parent runs on their results (fan-in)
+from toro import FlowChild as c
+report = await queue.add_flow("report", {"q": 3},
+                              children=[c("fetch", {"shard": i}) for i in range(3)])
+
 # Queue-wide rate limit: at most 100 jobs / second across every worker
 worker = Worker("emails", process, rate_limit={"max": 100, "duration": 1000})
 
@@ -121,6 +127,15 @@ job = await queue.add("resize", {"src": "a.png"})
 print(await job.result(timeout=30))
 ```
 
+## Flows
+
+A flow enqueues a parent and its children as one atomic tree. The children run
+first (fan-out, nested arbitrarily); the parent parks until every child has
+settled, then runs and reads their results (fan-in). One primitive covers
+fan-out/fan-in and chained steps, with per-child failure policies and
+flow-aware retry that recovers a whole failed flow in one shot. Full guide:
+[docs/flows.md](docs/flows.md).
+
 ## Develop
 
 Managed with [uv](https://astral.sh/uv); the Astral toolchain throughout.

{toro_queue-0.3.0 → toro_queue-0.5.0}/README.md

@@ -44,6 +44,7 @@ Pairs with **[matador](https://github.com/ilovepixelart/matador)**, a live web d
 | **Enqueue** | delayed jobs, global **priorities** (FIFO within a band) |
 | **Retries** | fixed or exponential **backoff**, capped attempts |
 | **Schedules** | repeatable **cron** and fixed-interval (`every`) jobs |
+| **Flows** | parent/child job trees: fan-out/fan-in, failure policies, flow-aware retry |
 | **Rate limiting** | queue-wide token bucket shared across all workers |
 | **Dedup** | custom (idempotent) job ids + a throttle window (`{id, ttl}`) |
 | **Auto-removal** | keep the last N and/or finished-within-age completed/failed |
@@ -86,6 +87,11 @@ await queue.add("charge", data, job_id="order-1234")
 # A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
 await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")
 
+# A flow: children run first (fan-out), the parent runs on their results (fan-in)
+from toro import FlowChild as c
+report = await queue.add_flow("report", {"q": 3},
+                              children=[c("fetch", {"shard": i}) for i in range(3)])
+
 # Queue-wide rate limit: at most 100 jobs / second across every worker
 worker = Worker("emails", process, rate_limit={"max": 100, "duration": 1000})
 
@@ -94,6 +100,15 @@ job = await queue.add("resize", {"src": "a.png"})
 print(await job.result(timeout=30))
 ```
 
+## Flows
+
+A flow enqueues a parent and its children as one atomic tree. The children run
+first (fan-out, nested arbitrarily); the parent parks until every child has
+settled, then runs and reads their results (fan-in). One primitive covers
+fan-out/fan-in and chained steps, with per-child failure policies and
+flow-aware retry that recovers a whole failed flow in one shot. Full guide:
+[docs/flows.md](docs/flows.md).
+
 ## Develop
 
 Managed with [uv](https://astral.sh/uv); the Astral toolchain throughout.

{toro_queue-0.3.0 → toro_queue-0.5.0}/docs/architecture.md

@@ -9,7 +9,8 @@ every job is durable in Redis.
 ## Atomic state transitions via Lua
 
 Every state move (`wait→active`, `active→completed/failed/delayed`,
-`delayed→wait`) is a single Redis Lua script, run atomically, so multi-key
+`delayed→wait`, and the flow transitions around `waiting-children` -
+see [Flows](flows.md)) is a single Redis Lua script, run atomically, so multi-key
 "check-then-act" sequences can't interleave. That removes whole classes of race:
 
 - **pop-then-lock gap** - two workers claiming the same job: the claim pops from
@@ -88,6 +89,10 @@ due jobs into the prioritized set.
   `Worker.on(event, fn)` exposes in-process hooks. See [Concepts](concepts.md).
 - **Auto-removal** - `remove_on_complete` / `remove_on_fail` (bool / count /
   `{count, age}`) enforced inside the finish script, not by a separate sweeper.
+- **Flows** - `add_flow()` creates a parent/child tree atomically; children
+  settle into the parent's `:deps` barrier inside the same finish scripts that
+  commit their own transitions, so the fan-in resolves on the crash path too.
+  See [Flows](flows.md).
 
 ## The Lua scripts
 
@@ -104,20 +109,23 @@ The scripts share a small library of routines:
 | `acquireNext` | Pops the top prioritized job into `active` and locks it, honoring the rate limit. |
 | `tryRateLimit` | Token bucket: ms until a token frees, or 0 to proceed. |
 | `recordFinished` | Records a terminal job in `completed`/`failed` and applies auto-removal. |
+| `settleChildCompleted` / `settleChildFailed` / `releaseParent` | A finishing flow child settles into its parent's `:deps` barrier; the last one releases the parent - or fails it eagerly, per `on_fail`. |
+| `keepArgsFromOpts` | The Lua twin of `JobOptions.keep_args`, for eager parent failures that have no Python caller. |
 
 And the scripts themselves:
 
 | Script | Caller | Does |
 |---|---|---|
 | `ADD_JOB` | producer | Mint/accept an id, write the hash, enqueue or delay, dedup, publish `added`. |
+| `ADD_FLOW` | producer | Create a whole flow tree atomically: leaves enqueued/delayed, parents parked with their `:deps` barrier. |
 | `MOVE_TO_ACTIVE` | worker wakeup | Claim the next job: `ZPOPMIN prioritized` → `active` → lock + load. |
-| `MOVE_TO_COMPLETED` | worker finish | Commit the result and fetch-next in one round trip. |
-| `MOVE_TO_FAILED` | worker finish | Retry (to `wait`/`delayed`) or terminally fail, and fetch-next. |
+| `MOVE_TO_COMPLETED` | worker finish | Commit the result (settling a flow child into its parent) and fetch-next in one round trip. |
+| `MOVE_TO_FAILED` | worker finish | Retry (to `wait`/`delayed`) or terminally fail (applying a flow child's `on_fail`), and fetch-next. |
 | `EXTEND_LOCK` | renewer | Token-guarded lock renewal; clears the job from `stalled`. |
 | `MOVE_STALLED` | sweep | Mark-and-sweep recovery of jobs whose lock expired. |
 | `PROMOTE_DELAYED` | promote loop | Move up to `PROMOTE_BATCH` (1000) due delayed jobs to `prioritized`. |
 | `ADD_SCHEDULED` | scheduler | Enqueue a scheduler occurrence under a deterministic id (idempotent). |
-| `PROMOTE_JOB` / `RETRY_JOB` / `REMOVE_JOB` | dashboard | Run a delayed job now / re-enqueue a failed one / delete a job with its lock and logs. |
+| `PROMOTE_JOB` / `RETRY_JOB` / `REMOVE_JOB` | dashboard | Run a delayed job now / re-enqueue a failed one (flow-aware: a parent with unsettled children re-parks, a child re-joins the barrier) / delete a job with its lock, logs and flow aux keys (a flow parent takes its subtree). |
 
 ### Lua → Python return protocol
 
@@ -132,8 +140,9 @@ Scripts signal outcomes with sentinels the worker decodes:
   job terminally failed or will retry.
 
 Scores are packed under 2^53 (`PRIORITY_OFFSET = 2^20`, `SEQ_MOD = 2^32`) so ZSET
-double scores stay exact, and the scripts use only plain JSON and integer ARGV -
-no `cmsgpack` / `bit` / `cjson` - so they run on any Redis build.
+double scores stay exact, and the scripts use only plain JSON and integer ARGV with the built-in
+`cjson` for encode/decode - no `cmsgpack` / `bit` - so they run on any
+Redis build.
 
 ## Python-specific choices
 

{toro_queue-0.3.0 → toro_queue-0.5.0}/docs/concepts.md

@@ -37,15 +37,17 @@ type, `JobState`:
 | `wait` | Ready to run, waiting for a free worker. (Stored in the priority-ordered set, so "wait" and "prioritized" are the same place.) |
 | `delayed` | Scheduled for the future; not yet runnable. Promoted to `wait` when due. |
 | `active` | Claimed by a worker and currently running. |
+| `waiting-children` | A flow parent, parked until every child settles; released to `wait` by its last child. |
 | `completed` | Finished successfully; `returnvalue` holds the result. |
 | `failed` | Exhausted its retry attempts; `failed_reason` holds the error. |
 
 The normal path is `wait → active → completed`. A failure with retries left goes
 `active → wait` (or `active → delayed`, if a backoff delay applies) and tries
 again; only after the last attempt does it land in `failed`. A delayed or
-repeatable job starts in `delayed`. See [Job lifecycle](architecture.md) for the
-exact transitions and [Producing jobs](producing.md) for how delay and retries
-are configured.
+repeatable job starts in `delayed`; a flow parent starts in `waiting-children`
+(see [Flows](flows.md)). See [Job lifecycle](architecture.md) for the exact
+transitions and [Producing jobs](producing.md) for how delay and retries are
+configured.
 
 ## Workers vs. slots
 

{toro_queue-0.3.0 → toro_queue-0.5.0}/docs/data-model.md

@@ -29,6 +29,7 @@ Redis Cluster slot, which the multi-key Lua scripts require.
 | `delayed` | ZSET | Ids scored by their process-at timestamp (ms); promoted to `prioritized` when due. |
 | `completed` | ZSET | Successfully-finished ids, scored by finish time (for auto-removal + listing). |
 | `failed` | ZSET | Terminally-failed ids, scored by finish time. |
+| `waiting-children` | ZSET | Flow parents parked until their children settle, scored by enqueue time. |
 | `meta-paused` | string (flag) | Exists only while the queue is paused; workers stop claiming new jobs. |
 | `events` | pub/sub channel | Carries `added` / `progress` / `completed` / `failed`; drives `result()` and live dashboards. |
 | `limiter` | HASH | The queue-wide rate-limit token bucket (`{tokens, ts}`), shared by every worker. |
@@ -37,6 +38,8 @@ Redis Cluster slot, which the multi-key Lua scripts require.
 | `repeat` | ZSET | Scheduler id -> next-run timestamp. |
 | `workers` | ZSET | Live worker id -> last-heartbeat ms; stale entries pruned lazily on read. |
 | `departed` | LIST (capped) | Recent worker departures: graceful `stopped` or `lost` (crashed). |
+| `metrics:<minute>` | HASH | Per-minute counters (`added`/`completed`/`failed`/`ms`, per-name fields, histograms); self-expiring. |
+| `de:<dedupId>` | string (PX) | A live deduplication throttle window; holds the already-queued job's id. |
 
 ## Per-scheduler, per-worker, per-job keys
 
@@ -44,9 +47,12 @@ Redis Cluster slot, which the multi-key Lua scripts require.
 |---|---|---|
 | `repeat:<schedulerId>` | HASH | A scheduler's template: `name`, `every`/`cron`, `data`, `opts`. |
 | `worker:<workerId>` | HASH | A worker's presence record: host, pid, concurrency, current jobs, processed/failed counts, state. |
-| `<jobId>` | HASH | The job itself: `name`, `data`, `opts`, `state`, `attemptsMade`, timestamps, `returnvalue`/`failedReason`, `progress`, `stacktrace`, ... |
+| `<jobId>` | HASH | The job itself: `name`, `data`, `opts`, `state`, `attemptsMade`, timestamps, `returnvalue`/`failedReason`, `progress`, `stacktrace`, plus flow linkage on flow jobs: `parentId`/`onFail` (children), `children` (parents). |
 | `<jobId>:lock` | string (token, PX) | The per-job lock: the owning worker's token with an expiry. Only the holder may finish or renew it. |
 | `<jobId>:logs` | LIST | Log lines appended by `job.log(...)` from inside a processor. |
+| `<jobId>:deps` | SET | A flow parent's still-pending child ids - the fan-in barrier; the parent releases when it empties. |
+| `<jobId>:results` | HASH | Child id → returnvalue JSON, written as each child completes. |
+| `<jobId>:cfail` | HASH | Child id → failure reason for children failed under `on_fail="continue"`. |
 
 Note the job hash key is just `<prefix>:<name>:<jobId>` (no extra segment), so a job
 `5` on `toro:emails:` is the hash `toro:emails:5`, with `toro:emails:5:lock` and
@@ -54,9 +60,11 @@ Note the job hash key is just `<prefix>:<name>:<jobId>` (no extra segment), so a
 
 ## How the pieces connect
 
-- A job moves between `prioritized` / `active` / `delayed` / `completed` / `failed`
-  as its state changes; the move and the hash update happen in one Lua script. See
-  [Architecture](architecture.md).
+- A job moves between `prioritized` / `active` / `delayed` / `waiting-children` /
+  `completed` / `failed` as its state changes; the move and the hash update happen
+  in one Lua script. See [Architecture](architecture.md).
+- `:deps` + `:results` + `:cfail` are the flow fan-in machinery - children settle
+  into them as they finish. See [Flows](flows.md).
 - The `lock` + `stalled` keys are the at-least-once machinery. See
   [Reliability](reliability.md).
 - `repeat` + `repeat:<id>` drive [scheduling](scheduling.md); `workers` +

toro_queue-0.5.0/docs/flows-design.md

@@ -0,0 +1,196 @@
+# Flows: design notes
+
+> Status: SHIPPED. This documents the design decisions and the research behind
+> them; the user guide is [flows.md](flows.md).
+
+A flow is a parent job enqueued together with its children. Children run first
+(in parallel, nesting allowed); the parent becomes runnable only when every
+child has settled. This gives fan-out/fan-in ("fetch 10 parts, then summarize")
+and chains ("A, then B") with one primitive.
+
+## Why this design: lessons from the landscape
+
+Every mature queue eventually grew a dependency story, and each one left a
+public trail of what went wrong. Before building, we surveyed the dependency /
+workflow features of the established queues across the Node.js, Python and
+Postgres ecosystems - both how they model things and what their issue trackers
+say users keep hitting. The design below is shaped by that trail.
+
+What the survey says, distilled:
+
+1. **Fan-in barriers die at the crash boundary** unless the bookkeeping is
+   atomic with the child's settle. Counter- and callback-based barriers
+   coordinated from the client side have produced a decade of "the callback
+   never fired" bugs: a worker killed mid-job, a nested group counted wrong,
+   a result expiring before the barrier read it. toro already funnels every
+   settle through one Lua script - the barrier belongs inside it, crash path
+   included.
+2. **Implicit result injection ages badly.** Passing the previous job's return
+   value as a magic argument (first or last position, depending on the
+   framework) always grows an opt-out flag and confuses arity forever.
+   Explicit pull by handle is the shape that aged well - provided the
+   ergonomics are good; "fetch the dependency object and read `.result` off
+   it" is the unergonomic version users complain about.
+3. **"Wait forever" as a failure default is universally hated.** Two major
+   queues default to leaving the dependent/parent parked indefinitely when a
+   dependency fails terminally, silently; in both, it's the single most
+   complained-about behavior of the feature. Failure must propagate by
+   default.
+4. **Graph state lives in the datastore, not the message.** Serializing the
+   remaining workflow into message headers means bloat, no nesting, and no
+   visibility (one framework's own redesign RFC concedes exactly this). Keys
+   in the datastore are also the only way a dashboard can show a flow.
+5. **Cleanup must be structural.** The most mature implementation leaks its
+   children-results hashes (no TTL, surviving auto-removal - tens of
+   thousands of stale keys reported in production) because aux keys are
+   cleaned on a separate path from job removal. Cleanup has to ride the
+   removal paths that already exist.
+6. **Eager beats lazy on parent failure.** Failing the parent "lazily" -
+   marking it and waiting for a worker on the parent's queue to actually
+   transition it - surprises people whenever no such worker exists. The
+   parent should fail in the same atomic step as the child.
+7. **Failure policy frozen at enqueue time** (denormalized into the stored
+   job) is a recurring complaint: teams want to change their minds after the
+   flow exists. Storing the policy as a plain mutable field costs nothing.
+
+## Design
+
+### Scope and shape
+
+- **Trees, not DAGs.** A child has exactly one parent. Multi-parent is rare,
+  expensive, and even the most mature implementation declined it;
+  fan-out/fan-in plus nesting covers the real use cases.
+- **Single queue per flow (v1).** toro workers dispatch on `job.name` inside
+  one processor, so flow steps are just different names on one queue. This
+  removes the cross-queue hazard class outright (scripts reaching into other
+  queues' keys; a purged children's queue stranding parents elsewhere).
+  Cross-queue can be revisited later without breaking the model.
+- **Static shape.** The tree is declared at enqueue time. Children spawned
+  dynamically from inside a processor is the messiest corner of the systems
+  that have it, and is out of scope.
+
+### API
+
+```python
+from toro import FlowChild as c   # name, data, opts, children, on_fail
+
+parent: Job = await queue.add_flow(name, data, children=[...], **opts)
+```
+
+- `add_flow` enqueues the whole tree **atomically** (one Lua call) and returns
+  the parent `Job`. `await parent.result()` resolves when the flow does.
+- Children accept the same options as `Queue.add()` (priority, attempts,
+  backoff, delay, auto-removal), plus `on_fail` (below). Scheduler-style
+  options, custom ids and deduplication are not valid on flow nodes.
+- Inside the parent's processor, results are **pulled explicitly**:
+
+```python
+async def process(job):
+    if job.name == "report":
+        results = await job.children_results()   # {child_id: returnvalue}
+        failures = await job.failed_children()   # {child_id: failed_reason}
+```
+
+- Introspection: `await queue.get_flow(parent_id)` returns the tree
+  (`{job, children: [...]}`), for the dashboard and for users.
+
+### Failure semantics: two policies, no third
+
+Per-child `on_fail`, stored as a plain mutable field on the child's hash (so
+tooling *can* change it after enqueue - see lesson 7):
+
+| `on_fail`               | when this child terminally fails                       |
+| ----------------------- | ------------------------------------------------------ |
+| `"fail_parent"` (default) | the parent fails **immediately and eagerly**, in the same Lua script, recursively up ancestors that also default; no worker on the parent required |
+| `"continue"`            | the failure is recorded in the parent's failures hash, the dependency cleared; the parent runs once all children settle and inspects `failed_children()` |
+
+There is deliberately **no "wait indefinitely" option** (lesson 3). Retries
+still happen first: "terminally fails" means after the child's own `attempts`
+are exhausted (or it stalls past `max_stalled_count`). A child retried to
+success *after* its parent already failed does not resurrect the parent
+(documented, v1); retrying the parent re-arms its barrier and pulls its
+failed children along instead.
+
+### Data model
+
+New job state `waiting-children` (a sixth `JobState`), backed by a per-queue
+ZSET (timestamp-scored, like `completed`/`failed`), surfaced in `counts()`,
+`get_jobs()`, `clean()` and the dashboard.
+
+Per parent job, three aux keys (joining `:lock` / `:logs`):
+
+| key             | type | content                                            |
+| --------------- | ---- | -------------------------------------------------- |
+| `{id}:deps`     | SET  | ids of children not yet settled (the barrier)      |
+| `{id}:results`  | HASH | child id → returnvalue JSON (completed children)   |
+| `{id}:cfail`    | HASH | child id → failed reason (`on_fail="continue"` children) |
+
+Plus two hash fields: `parentId` on every child; `children` (static JSON id
+list) on every parent - the deps set shrinks, the dashboard tree needs the
+full picture.
+
+A SET rather than a counter: it's idempotent under re-delivery, inspectable
+("which children is this parent still waiting on?"), and the empty-check
+(`SCARD == 0`) is the release condition. The counter-corruption bug class
+(lesson 1) is the argument against counters.
+
+### Mechanics
+
+- **`ADD_FLOW`** (new script): tree as JSON in ARGV, `cjson.decode`, walk
+  depth-first; leaves enqueue into `prioritized` (or `delayed`), interior
+  nodes land in `waiting-children` with their `:deps` set populated. Capped
+  (~1000 nodes per flow) to bound script time, like `PROMOTE_BATCH`. One
+  `added` increment of the node count and one announce (the root id) for
+  the whole tree.
+- **Release** lives inside the existing finish scripts, the extension point
+  `_LIB` reserved ("to add markers-with-delay or grouping later, we change
+  only these functions"):
+  - `MOVE_TO_COMPLETED` of a child: `HSET parent:results`, `SREM parent:deps`;
+    on empty, move the parent from `waiting-children` through the shared
+    `enqueue()` at its stored priority.
+  - `MOVE_TO_FAILED` terminal branch: apply `on_fail` - either record into
+    `:cfail` + `SREM` (+ release if last), or fail the parent now through
+    `recordFinished` (so `remove_on_fail` retention applies), publish the
+    event, and **loop upward** while the ancestor itself has a parent with
+    `fail_parent`.
+  - **The stalled path gets identical parent bookkeeping**: `MOVE_STALLED`'s
+    fail-branch runs the same settle logic (lesson 1: the crash path is where
+    barriers historically break). One pre-existing nuance: the stall-escalated
+    child itself is recorded into `failed` without `recordFinished`, so its
+    own `remove_on_fail` retention does not apply on the stall path - true
+    for all stalled jobs, not just flow children.
+- **Cleanup is structural** (lesson 5): `delJobs` and `REMOVE_JOB` know the
+  three aux keys, so every existing removal path (auto-removal
+  keepCount/keepAge, manual remove, `clean()`) deletes them for free.
+  Removing a parent removes its subtree; removing a child SREMs it from its
+  parent's deps (and releases the parent if it was the last). Settle writes
+  are guarded by a parent-exists check so a retention-trimmed parent can't
+  get orphan keys recreated by late siblings.
+- **Retry is flow-aware**: at the script level a failed parent with unsettled
+  deps re-parks in `waiting-children`, and a retried child re-joins a parked
+  parent's barrier (clearing its stale `:cfail` entry). On top of that,
+  `retry_job(parent)` re-drives the whole failed subtree (delegating to
+  `retry_flow`), so `retry_all_failed()` and a single parent retry both recover
+  a whole flow in any order.
+
+### Edge cases pinned down (each has a test)
+
+- **Parent retries**: a released parent is a normal job; its own
+  `attempts`/`backoff` apply. `retry_job(parent)` re-drives the subtree's
+  *failed* children too (root-first); completed children are not re-run -
+  their results are already in `:results`.
+- **`remove_on_complete` on children**: allowed - the result is copied into
+  the parent's `:results` at completion, so the child hash is free to go.
+  Routine `clean("completed")` likewise never touches a pending parent's
+  collected results.
+- **Empty `children=[]`**: rejected (`ValueError`) - that's `add()`.
+- **Id rules**: flow nodes get server-side ids; custom ids on flow nodes are
+  not supported in v1 (id reuse inside trees is a documented zombie-job
+  hazard elsewhere).
+- **Progress display counts completions only** - a failed flow must never
+  read as 100% done.
+
+## Out of scope (v1)
+
+Cross-queue flows, DAGs/multi-parent, dynamic children from inside a
+processor, and child-retry-resurrects-failed-parent.

toro_queue-0.5.0/docs/flows.md

@@ -0,0 +1,190 @@
+# Flows
+
+A flow is a parent job enqueued atomically with its children. The children run
+first (in parallel, nested arbitrarily within the 1000-node cap); the parent
+is parked in the `waiting-children` state and runs only once every child has
+settled. One
+primitive covers fan-out/fan-in ("fetch 10 shards, then merge") and chains
+("build, then deploy").
+
+> Why the API and the failure defaults look the way they do - including the
+> failure modes other queues' dependency features taught us to avoid - is
+> documented in [the design notes](flows-design.md).
+
+## Enqueue a flow
+
+```python
+from toro import FlowChild as c
+
+parent = await queue.add_flow(
+    "report", {"period": "2026-06"},
+    children=[
+        c("fetch", {"shard": 1}),
+        c("fetch", {"shard": 2}),
+        c("summarize", {}, children=[c("fetch", {"shard": 3})]),  # nesting
+    ],
+    attempts=3,                # parent options: add()'s, minus job_id/deduplication
+)
+result = await parent.result()        # resolves when the WHOLE flow does
+```
+
+If the flow fails, `result()` raises `JobFailedError` naming the failing
+child (`"child 7 failed: ..."`). The default `timeout` is 30s - size it for
+the whole flow, not one job.
+
+`add_flow` inserts the entire tree in one atomic script - either the whole
+flow exists or none of it. Leaves go straight to `wait` (or `delayed`, if they
+carry a `delay`); every node with children parks in `waiting-children`.
+
+`FlowChild` takes the same options as `Queue.add()` (`priority` - clamped to
+the same range as `add()` - `attempts`, `backoff`, `delay`, auto-removal),
+plus `on_fail` (below). A few options are
+deliberately not valid on flow nodes: custom `job_id`s and `deduplication`
+(both invite id-reuse hazards inside trees), and `delay` on a node that has
+children (it runs when its children settle, not on a clock).
+
+## Reading children results
+
+The parent pulls its children's results explicitly - there is no implicit
+argument injection to reason backwards from:
+
+```python
+async def process(job):
+    if job.name == "report":
+        results = await job.children_results()   # {child_id: returnvalue}
+        failures = await job.failed_children()    # {child_id: reason}
+        return summarize(results, failures)
+```
+
+A child's result is copied into the parent at the moment the child completes,
+so children are free to use `remove_on_complete` - the parent's copy survives,
+and so does routine history cleanup (`clean("completed")`).
+
+## When a child fails
+
+Each child's `on_fail` says what its *terminal* failure (after its own
+retries) does to the parent:
+
+| `on_fail` | behavior |
+|---|---|
+| `"fail_parent"` (default) | The parent fails **immediately** - in the same atomic step, no worker needed - and the failure walks up through ancestors that also default. |
+| `"continue"` | The failure is recorded; the parent still runs once every child has settled and inspects `failed_children()`. |
+
+There is deliberately no "wait forever" option: a flow always settles. The
+crash path counts too - a child whose worker died and that exhausts the
+stalled-recovery limit settles its parent the same way.
+
+Eager parent failure does **not** cancel siblings: in-flight and still-queued
+children keep running, and their results are still collected into the parent
+(useful if you later retry it). To actually stop the remaining work, remove
+the parent - removal cascades the subtree.
+
+## Retrying a failed flow
+
+Retry is flow-aware:
+
+- Retrying a failed **parent** re-drives its whole failed subtree: it re-parks
+  on every non-completed child (completed children keep their collected results)
+  and re-queues the failed ones root-first, so the flow recovers in a single
+  call. It re-arms the barrier rather than running on partial results.
+- Retrying a failed **child** re-joins its parked parent's barrier and clears
+  the stale entry from the parent's failure report.
+
+So `retry_all_failed()` - or the dashboard's *retry all* - recovers an entire
+failed flow in one shot, in any order. Retrying just a **child** of an
+already-failed parent still won't resurrect the parent on its own; retry the
+parent (which now pulls its failed children along) or use retry-all.
+
+To recover *one* flow without touching the rest of the queue, use
+`retry_flow(parent_id)`: it retries every failed job in that subtree, root
+first, so a re-parked parent is ready before its children re-join the barrier -
+order-independent, like retry-all but scoped. It returns the number of jobs
+retried. The dashboard's *retry flow* button on a failed parent calls it.
+
+## Removing flow jobs
+
+- Removing a **parent** removes its whole subtree - children included, even
+  ones currently running. This is how you cancel a flow. (A running child's
+  processor coroutine is not interrupted; it finishes and its commit is then
+  discarded by the lock-token guard.)
+- Removing a pending **child** releases the parent if it was the last thing
+  being waited on. A removed completed child does *not* take its
+  already-collected result with it.
+- `clean("waiting-children")` therefore cancels every parked flow outright.
+
+All removal paths (manual, bulk, auto-removal) clean up the flow bookkeeping
+keys with the job - nothing is left behind to leak.
+
+## Introspection
+
+```python
+flow = await queue.get_flow(parent_id)        # {"job": Job, "children": [...]}
+results = await queue.children_results(parent_id)
+failures = await queue.failed_children(parent_id)
+```
+
+`get_flow` hydrates the tree breadth-first, one pipelined round trip per
+level, down to `depth` levels (default 10; `depth=0` returns just the root
+node) - pass a larger `depth` for deeper trees. `Job.parent_id` and
+`Job.children_ids` expose flow membership on any loaded job.
+
+For a dashboard detail view, `flow_view(parent_id)` folds those three reads
+into one `FlowView`:
+
+```python
+view = await queue.flow_view(parent_id)
+view.tree                       # same {"job": Job, "children": [...]} shape
+view.results, view.failures     # collected results + tolerated failures
+view.total, view.done, view.failed   # fan-in counts (completions only)
+view.live                       # True while any node is still non-terminal
+```
+
+It costs the tree's O(depth) round trips plus one pipelined read of the
+parent's result/failure hashes (not three separate calls), and `done`/`failed`
+count completions only - a failed flow never reads as done. Returns `None` if
+the job doesn't exist.
+
+To show fan-in progress for a whole *page* of parents without hydrating each
+tree, `flow_progress(parent_ids)` returns `{parent_id: (completed, failed)}` from
+cheap pipelined `HLEN` reads (just the counts):
+
+```python
+progress = await queue.flow_progress([p1, p2, p3])   # {p1: (2, 0), p2: (1, 1), ...}
+```
+
+[matador](https://github.com/ilovepixelart/matador) renders a flow as its root
+job moving through the normal state tabs (children hidden, shown only in the
+tree), with a recursive tree and fan-in progress on the job detail.
+
+## Metrics and events
+
+Enqueueing a flow increments the queue's `added` counter by the node count
+and publishes a single `added` event carrying the root's id. On failures,
+every terminally-failed child increments the per-job `failed` counter
+(tolerated `continue` failures included), and each eagerly-failed ancestor
+increments it again - so one leaf failure in a deep `fail_parent` chain
+produces several `failed` increments and events. That counter is about *jobs*.
+
+Whole flows are counted in their own right, one unit per **root** flow as it
+settles (nested sub-flows don't double count):
+
+```python
+points = await queue.flow_metrics(minutes=60)   # per-minute completed/failed
+pcts = await queue.flow_percentiles(minutes=60)  # end-to-end p50/p95/p99 (ms)
+```
+
+`flow_percentiles` measures the whole flow's wall clock - enqueue to the root
+finishing - which the per-job duration never captures (a flow that fans out
+wide finishes long after any single job's runtime). The dashboard charts this
+as a throughput strip on the active tab. Both reads zero-fill and share the
+8h metrics retention.
+
+## Limits
+
+- One queue per flow: steps are job names on the same queue, dispatched by
+  your processor. (Cross-queue flows are a possible future, not a v1 feature.)
+- Trees, not DAGs: a child has exactly one parent.
+- The shape is declared at enqueue time; processors can't append children to a
+  running flow.
+- At most 1000 nodes per flow (the atomic insert bounds how long Redis is
+  held, same idea as the delayed-promotion batch).

{toro_queue-0.3.0 → toro_queue-0.5.0}/docs/index.md

@@ -4,7 +4,7 @@ Reference docs for how toro works. The [README](../README.md) is the quick start
 
 ## Pages
 
-- **[Concepts](concepts.md)** - the mental model: queues, workers, jobs, the five
+- **[Concepts](concepts.md)** - the mental model: queues, workers, jobs, the six
   job states, and the difference between *workers* and *slots*.
 - **[Data model](data-model.md)** - the exact Redis keys a queue uses and what
   each one stores.
@@ -16,7 +16,14 @@ Reference docs for how toro works. The [README](../README.md) is the quick start
   rate limiting, and graceful shutdown.
 - **[Scheduling](scheduling.md)** - repeatable and cron jobs, and how each
   occurrence schedules the next.
+- **[Flows](flows.md)** - parent/child job trees: fan-out/fan-in, failure
+  policies, flow-aware retry and removal.
 - **[Architecture](architecture.md)** - the atomic-Lua core and the design
   decisions behind the queue.
 - **[Security](security.md)** - what toro guarantees (JSON-only, no dynamic
   dispatch, no string-built commands) and what you own (Redis access, secrets).
+
+## Design notes
+
+- **[Flows (design)](flows-design.md)** - the decisions and landscape lessons
+  behind the flows feature.

{toro_queue-0.3.0 → toro_queue-0.5.0}/docs/processing.md

@@ -34,6 +34,17 @@ async def handle(job):
 
 `update_progress` takes a number or any JSON value; dashboards render it live.
 
+A flow parent's processor pulls what its children produced (both helpers are
+processor-only and raise `RuntimeError` elsewhere - use the `Queue`-side
+equivalents outside a worker):
+
+```python
+results = await job.children_results()   # {child_id: returnvalue}
+failures = await job.failed_children()   # {child_id: reason} (on_fail="continue")
+```
+
+See [Flows](flows.md) for the full model.
+
 ## Concurrency
 
 `concurrency=N` runs N processing loops ("slots") as `asyncio` tasks on one