toro-queue 0.4.0__tar.gz → 0.5.0__tar.gz

{toro_queue-0.4.0 → toro_queue-0.5.0}/.gitignore

@@ -9,3 +9,5 @@ __pycache__/
 
 .coverage
 coverage.xml
+
+.DS_Store

{toro_queue-0.4.0 → toro_queue-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: toro-queue
-Version: 0.4.0
+Version: 0.5.0
 Summary: An async-first, Redis-backed job queue for Python.
 Project-URL: Homepage, https://github.com/ilovepixelart/toro
 Project-URL: Repository, https://github.com/ilovepixelart/toro

{toro_queue-0.4.0 → toro_queue-0.5.0}/docs/flows-design.md

@@ -108,7 +108,8 @@ There is deliberately **no "wait indefinitely" option** (lesson 3). Retries
 still happen first: "terminally fails" means after the child's own `attempts`
 are exhausted (or it stalls past `max_stalled_count`). A child retried to
 success *after* its parent already failed does not resurrect the parent
-(documented, v1); retrying the parent re-arms its barrier instead.
+(documented, v1); retrying the parent re-arms its barrier and pulls its
+failed children along instead.
 
 ### Data model
 
@@ -165,16 +166,19 @@ A SET rather than a counter: it's idempotent under re-delivery, inspectable
   parent's deps (and releases the parent if it was the last). Settle writes
   are guarded by a parent-exists check so a retention-trimmed parent can't
   get orphan keys recreated by late siblings.
-- **Retry is flow-aware**: a failed parent with unsettled deps re-parks in
-  `waiting-children`; a retried child re-joins a parked parent's barrier and
-  clears its stale `:cfail` entry. `retry_all_failed()` therefore recovers a
-  whole flow in any order.
+- **Retry is flow-aware**: at the script level a failed parent with unsettled
+  deps re-parks in `waiting-children`, and a retried child re-joins a parked
+  parent's barrier (clearing its stale `:cfail` entry). On top of that,
+  `retry_job(parent)` re-drives the whole failed subtree (delegating to
+  `retry_flow`), so `retry_all_failed()` and a single parent retry both recover
+  a whole flow in any order.
 
 ### Edge cases pinned down (each has a test)
 
 - **Parent retries**: a released parent is a normal job; its own
-  `attempts`/`backoff` apply. Children are not re-run on parent retry -
-  results are already in `:results`.
+  `attempts`/`backoff` apply. `retry_job(parent)` re-drives the subtree's
+  *failed* children too (root-first); completed children are not re-run -
+  their results are already in `:results`.
 - **`remove_on_complete` on children**: allowed - the result is copied into
   the parent's `:results` at completion, so the child hash is free to go.
   Routine `clean("completed")` likewise never touches a pending parent's

{toro_queue-0.4.0 → toro_queue-0.5.0}/docs/flows.md

@@ -83,16 +83,17 @@ the parent - removal cascades the subtree.
 
 Retry is flow-aware:
 
-- Retrying a failed **parent** re-arms its barrier: it goes back to
-  `waiting-children` until its unsettled children resolve (it will not run on
-  partial results).
+- Retrying a failed **parent** re-drives its whole failed subtree: it re-parks
+  on every non-completed child (completed children keep their collected results)
+  and re-queues the failed ones root-first, so the flow recovers in a single
+  call. It re-arms the barrier rather than running on partial results.
 - Retrying a failed **child** re-joins its parked parent's barrier and clears
   the stale entry from the parent's failure report.
 
 So `retry_all_failed()` - or the dashboard's *retry all* - recovers an entire
-failed flow in one shot, in any order. One pinned v1 edge: if the parent has
-already failed and you retry *only* the child, the child's later success does
-not resurrect the parent; retry the parent too (or use retry-all).
+failed flow in one shot, in any order. Retrying just a **child** of an
+already-failed parent still won't resurrect the parent on its own; retry the
+parent (which now pulls its failed children along) or use retry-all.
 
 To recover *one* flow without touching the rest of the queue, use
 `retry_flow(parent_id)`: it retries every failed job in that subtree, root

{toro_queue-0.4.0 → toro_queue-0.5.0}/docs/producing.md

@@ -91,7 +91,7 @@ waiting; only the terminal outcome resolves the call.
 
 | Call | Does |
 |---|---|
-| `await queue.retry_job(job_id)` | Move one failed job back to the queue. Flow-aware: a failed flow parent re-parks until its children settle; a retried child re-joins its parked parent's barrier ([Flows](flows.md)). |
+| `await queue.retry_job(job_id)` | Move one failed job back to the queue. Flow-aware: retrying a flow parent re-drives its whole failed subtree (failed children pulled along, completed ones kept); a retried child re-joins its parked parent's barrier ([Flows](flows.md)). |
 | `await queue.retry_all_failed(limit=1000)` | Re-queue every failed job (pipelined, one round trip per batch); returns how many were retried. |
 | `await queue.promote_job(job_id)` | Run a delayed job now. |
 | `await queue.remove_job(job_id)` | Delete a job from every state, with its lock, logs and flow keys. Removing a flow parent removes its whole subtree - children included, even running ones. |

{toro_queue-0.4.0 → toro_queue-0.5.0}/pyproject.toml

@@ -2,7 +2,7 @@
 # PyPI distribution name - plain `toro` is taken (abandoned). The import package
 # stays `toro` (e.g. `pip install toro-queue` then `import toro`).
 name = "toro-queue"
-version = "0.4.0"
+version = "0.5.0"
 description = "An async-first, Redis-backed job queue for Python."
 readme = "README.md"
 requires-python = ">=3.10"

toro_queue-0.5.0/tests/integration/test_concurrency_stress.py

@@ -0,0 +1,99 @@
+"""Concurrency stress: many workers (each its own connection, like separate
+processes) hammering one Redis must process every job EXACTLY once - the atomic
+BLMOVE+Lua-lock claim is what makes running N processes against one queue safe -
+and flows must still settle under that contention. The stalled sweep runs the
+whole time on a short interval, so this also proves healthy jobs aren't falsely
+recovered (which would double-process).
+"""
+
+import asyncio
+import collections
+import contextlib
+
+from toro import FlowChild as c  # noqa: N813
+from toro import Worker
+
+PREFIX = "torotest"
+
+
+def _completed(q, n):
+    async def check():
+        return (await q.counts())["completed"] >= n
+
+    return check
+
+
+@contextlib.asynccontextmanager
+async def _fleet(name, proc, *, n: int, concurrency: int):
+    """n independent workers (own connections), all sweeping on a short interval;
+    lock_duration comfortably exceeds a job so a healthy job is never false-swept."""
+    workers = [
+        Worker(
+            name,
+            proc,
+            prefix=PREFIX,
+            concurrency=concurrency,
+            stalled_interval=100,
+            lock_duration=3000,
+            block_timeout=0.2,
+        )
+        for _ in range(n)
+    ]
+    tasks = [asyncio.create_task(w.run()) for w in workers]
+    try:
+        yield
+    finally:
+        for w in workers:
+            await w.stop(grace_period=2)
+        for t in tasks:
+            t.cancel()
+            with contextlib.suppress(asyncio.CancelledError):
+                await t
+
+
+async def test_every_job_processed_exactly_once_under_contention(q, run_until):
+    total, n_workers, concurrency = 200, 4, 4
+    runs: collections.Counter[str] = collections.Counter()
+
+    async def proc(job):
+        await asyncio.sleep(0.002)  # hold the slot so claims genuinely contend
+        runs[job.id] += 1  # single event loop, no await between r-m-w: safe
+
+    expected = {(await q.add(f"j{i}", {"i": i})).id for i in range(total)}
+
+    async with _fleet(q.name, proc, n=n_workers, concurrency=concurrency):
+        assert await run_until(_completed(q, total), timeout=30), "fleet did not drain"
+
+    counts = await q.counts()
+    assert counts["completed"] == total and counts["failed"] == 0
+    assert set(runs) == expected  # nothing missed, nothing spurious
+    dupes = [jid for jid, runcount in runs.items() if runcount != 1]
+    assert not dupes, f"processed more than once: {dupes}"
+
+
+async def test_flows_settle_under_contention(q, run_until):
+    n_flows, kids, n_workers = 30, 4, 4
+    parents = []
+
+    async def proc(job):
+        if job.name.startswith("flow"):
+            return sorted((await job.children_results()).values())
+        return job.data["i"]
+
+    for f in range(n_flows):
+        p = await q.add_flow(
+            f"flow{f}", {}, children=[c(f"c{f}-{i}", {"i": i}) for i in range(kids)]
+        )
+        parents.append(p.id)
+
+    nodes = n_flows * (kids + 1)
+    async with _fleet(q.name, proc, n=n_workers, concurrency=4):
+        assert await run_until(_completed(q, nodes), timeout=30), "flows did not drain"
+
+    # every parent ran exactly once on its children's results; nothing stuck
+    assert (await q.counts())["completed"] == nodes
+    assert (await q.counts())["waiting-children"] == 0
+    for pid in parents:
+        job = await q.get_job(pid)
+        assert job is not None and job.state == "completed"
+        assert len(job.returnvalue) == kids  # the fan-in collected every child

toro_queue-0.5.0/tests/integration/test_fault_injection.py

@@ -0,0 +1,163 @@
+"""Fault injection / chaos: drive toro through faults the happy-path tests never
+see and assert the at-least-once machinery still holds - a dropped Redis call
+mid-commit, and two workers racing the same recovery or promotion.
+
+These target the gaps left by test_reliability/test_worker_resilience: those cover
+a worker that *dies*; here the worker *survives* a transient failure mid-finish
+(the lock renewer is cancelled, the lock lapses, the stalled sweep recovers the
+job) and two workers hit the same atomic guard at once.
+"""
+
+import asyncio
+
+from toro import Queue, Worker
+
+PREFIX = "torotest"
+
+
+async def _noop(job):
+    return None
+
+
+def _completed(q: Queue, n: int = 1):
+    async def check():
+        return (await q.counts())["completed"] >= n
+
+    return check
+
+
+def _failed(q: Queue, n: int = 1):
+    async def check():
+        return (await q.counts())["failed"] >= n
+
+    return check
+
+
+# ---- a dropped Redis call mid-commit recovers, exactly one terminal state ----------
+
+
+async def test_dropped_commit_recovers_and_completes_once(q, run_worker, run_until):
+    runs: list[str] = []
+
+    async def proc(job):
+        runs.append(job.id)
+
+    # short lock so it lapses fast once the renewer is cancelled; sweep is driven
+    # by hand for determinism (stalled_interval=0 disables the worker's own loop).
+    async with run_worker(
+        q, proc, concurrency=1, stalled_interval=0, lock_duration=150, block_timeout=0.2
+    ) as w:
+        orig = w._finish_completed
+        hits = {"n": 0}
+
+        async def flaky(job, result):
+            hits["n"] += 1
+            if hits["n"] == 1:
+                raise ConnectionError("redis dropped mid-commit")  # the first commit never lands
+            return await orig(job, result)
+
+        w._finish_completed = flaky
+
+        await q.add("j", {})
+        # claim + process happen, then the commit is dropped: the job is stranded
+        # on `active`, the renewer was cancelled, so the lock will lapse.
+        assert await run_until(lambda: hits["n"] >= 1, timeout=10), "commit never attempted"
+        await asyncio.sleep(0.3)  # > lock_duration: the lock is now dead
+        await w.check_stalled(throttle_ms=0)  # pass 1: mark
+        await w.check_stalled(throttle_ms=0)  # pass 2: recover -> wait + wakeup marker
+        # the freed slot re-claims and this time the commit lands
+        assert await run_until(_completed(q, 1), timeout=10), "never recovered"
+
+    assert (await q.counts())["completed"] == 1  # exactly one terminal completion
+    assert hits["n"] >= 2  # the commit was genuinely retried after recovery
+    assert len(runs) >= 1  # at-least-once: the processor may have run twice
+
+
+async def test_dropped_fail_commit_recovers_and_fails_once(q, run_worker, run_until):
+    async def proc(job):
+        raise RuntimeError("boom")  # always fails
+
+    async with run_worker(
+        q, proc, concurrency=1, stalled_interval=0, lock_duration=150, block_timeout=0.2
+    ) as w:
+        orig = w._finish_failed
+        hits = {"n": 0}
+
+        async def flaky(job, exc):
+            hits["n"] += 1
+            if hits["n"] == 1:
+                raise ConnectionError("redis dropped mid-fail-commit")
+            return await orig(job, exc)
+
+        w._finish_failed = flaky
+
+        await q.add("j", {}, attempts=1)
+        assert await run_until(lambda: hits["n"] >= 1, timeout=10), "fail-commit never attempted"
+        await asyncio.sleep(0.3)
+        await w.check_stalled(throttle_ms=0)
+        await w.check_stalled(throttle_ms=0)
+        assert await run_until(_failed(q, 1), timeout=10), "never recovered to failed"
+
+    assert (await q.counts())["failed"] == 1  # exactly one terminal failure
+    assert hits["n"] >= 2
+
+
+# ---- two workers hit the same atomic guard at once ---------------------------------
+
+
+async def test_concurrent_stalled_sweeps_recover_exactly_once(q):
+    # A dead job swept by two workers at the same instant: the atomic LREM-from-active
+    # guard inside MOVE_STALLED means exactly one recovers it - no double re-enqueue,
+    # no double counter bump.
+    job = await q.add("x", {}, attempts=5)
+    jid = job.id
+    w1 = Worker(q.name, _noop, prefix=PREFIX, max_stalled_count=3, connection=q.redis)
+    w2 = Worker(q.name, _noop, prefix=PREFIX, max_stalled_count=3, connection=q.redis)
+
+    # a worker grabbed it and died: on `active`, no lock
+    await q.redis.zrem(q.keys.prioritized, jid)
+    await q.redis.rpush(q.keys.active, jid)
+
+    await w1.check_stalled(throttle_ms=0)  # pass 1: mark (single, sequential)
+    assert await q.redis.sismember(q.keys.stalled, jid)
+    # pass 2: BOTH workers try to recover the now-marked job at the same instant
+    results = await asyncio.gather(w1.check_stalled(throttle_ms=0), w2.check_stalled(throttle_ms=0))
+
+    recovered = [r for _failed_ids, rec in results for r in rec]
+    assert recovered.count(jid) == 1, results  # exactly one sweep won
+    assert await q.redis.zscore(q.keys.prioritized, jid) is not None  # back in the queue once
+    assert jid not in await q.redis.lrange(q.keys.active, 0, -1)  # off active
+    assert await q.redis.hget(q.keys.job(jid), "stalledCounter") == "1"  # bumped once, not twice
+
+
+async def test_concurrent_delayed_promotion_promotes_each_once(q):
+    # Two workers promote the same due batch at once. Lua runs serially, so the
+    # first promotes the whole batch and the second finds an empty range: every
+    # due job lands in `prioritized` exactly once, none lost or duplicated.
+    from toro import scripts
+
+    now = await _server_now_ms(q)
+    ids = [f"d{i}" for i in range(5)]
+    for jid in ids:
+        await q.redis.hset(
+            q.keys.job(jid),
+            mapping={"id": jid, "name": jid, "data": "{}", "opts": "{}", "state": "delayed"},
+        )
+        await q.redis.zadd(q.keys.delayed, {jid: now - 1000})  # already due
+
+    w1 = Worker(q.name, _noop, prefix=PREFIX, connection=q.redis)
+    w2 = Worker(q.name, _noop, prefix=PREFIX, connection=q.redis)
+    keys = [q.keys.delayed, q.keys.prioritized, q.keys.marker, q.keys.base, q.keys.pc]
+    await asyncio.gather(
+        w1._promote_delayed(keys=keys, args=[now, scripts.PROMOTE_BATCH]),
+        w2._promote_delayed(keys=keys, args=[now, scripts.PROMOTE_BATCH]),
+    )
+
+    promoted = sorted(await q.redis.zrange(q.keys.prioritized, 0, -1))
+    assert promoted == sorted(ids)  # each due job promoted exactly once, none duplicated
+    assert await q.redis.zcard(q.keys.delayed) == 0  # nothing left behind
+
+
+async def _server_now_ms(q: Queue) -> int:
+    secs, micros = await q.redis.time()
+    return int(secs) * 1000 + int(micros) // 1000

{toro_queue-0.4.0 → toro_queue-0.5.0}/tests/integration/test_flows_edges.py

@@ -172,30 +172,28 @@ async def test_retry_failed_child_after_parent_failed(q, run_worker, run_until):
     assert (await q.get_job(parent.id)).state == "failed"
 
 
-async def test_retry_reparks_failed_parent_until_children_settle(q, run_worker, run_until):
-    fail = True
+async def test_retry_parent_reparks_and_redrives_its_failed_child(q, run_worker, run_until):
+    # retrying a parent re-arms the barrier AND re-drives its failed child, but the
+    # parent must still wait for that child - never run on partial results.
+    async def boom(job):
+        raise RuntimeError("boom")
 
-    async def proc(job):
-        if job.name == "bad" and fail:
-            raise RuntimeError("boom")
-        return "fixed" if job.name == "bad" else "report-ran"
+    parent = await q.add_flow("report", {}, children=[c("bad", {})])
+    async with run_worker(q, boom):  # fail the flow, then the worker exits
+        assert await run_until(_count_is(q, "failed", 2))  # child + parent
 
-    async with run_worker(q, proc):
-        parent = await q.add_flow("report", {}, children=[c("bad", {})])
-        assert await run_until(_count_is(q, "failed", 2))
+    # retry the PARENT only; with no worker running we can inspect the re-armed state
+    assert await q.retry_job(parent.id)
+    cid = (await q.get_flow(parent.id))["children"][0]["job"].id
+    assert (await q.get_job(parent.id)).state == "waiting-children"  # re-parked, not run
+    assert (await q.get_job(cid)).state == "wait"  # the failed child was re-driven too
+    assert await _count(q, "completed") == 0  # no partial run
 
-        # retrying the parent re-arms the barrier - it must NOT run with
-        # partial results while its failed child is unsettled
-        assert await q.retry_job(parent.id)
-        assert (await q.get_job(parent.id)).state == "waiting-children"
-        assert await _count(q, "completed") == 0
+    async def fixed(job):
+        return "fixed" if job.name == "bad" else "report-ran"
 
-        # retrying the child completes it, which releases the re-parked parent
-        fail = False
-        cid = (await q.get_flow(parent.id))["children"][0]["job"].id
-        assert await q.retry_job(cid)
+    async with run_worker(q, fixed):  # the re-driven child completes, releasing the parent
         assert await run_until(_count_is(q, "completed", 2))
-
     assert (await q.get_job(parent.id)).returnvalue == "report-ran"
 
 
@@ -620,6 +618,31 @@ async def test_retry_flow_recovers_a_whole_failed_flow(q, run_worker, run_until)
     assert (await q.get_job(parent.id)).returnvalue == ["fixed", "fixed"]
 
 
+async def test_retry_parent_recovers_the_whole_flow(q, run_worker, run_until):
+    # retrying a PARENT directly re-drives its failed children too, so a flow that
+    # failed on a child recovers in one call instead of stranding on it.
+    fail = True
+
+    async def proc(job):
+        if job.name == "bad":
+            if fail:
+                raise RuntimeError("boom")
+            return "fixed"
+        if job.name == "ok":
+            return "ok-ran"
+        return "report-ran"  # the parent
+
+    async with run_worker(q, proc, concurrency=2):
+        parent = await q.add_flow("report", {}, children=[c("ok", {}), c("bad", {})])
+        assert await run_until(_count_is(q, "failed", 2))  # the bad child + the parent
+
+        fail = False
+        assert await q.retry_job(parent.id)  # parent only - the child rides along
+        assert await run_until(_count_is(q, "completed", 3))  # both children + parent
+
+    assert (await q.get_job(parent.id)).state == "completed"
+
+
 async def test_retry_flow_leaves_completed_children_untouched(q, run_worker, run_until):
     fail_bad = True
 

toro_queue-0.5.0/tests/integration/test_invariants.py

@@ -0,0 +1,197 @@
+"""Property-based invariants: drive the queue through long randomized operation
+sequences and assert the global invariants hold after every step, then that every
+flow settles once the queue is drained.
+
+A deterministic seeded fuzzer (not Hypothesis): processing is driven by hand via
+_acquire + _finish_* so each step is atomic with nothing else mutating - no live
+worker, no timing races - and a failing case reproduces exactly from its seed.
+
+Invariants per step:
+  1. counts() equals the real set cardinalities.
+  2. no job id is in two state sets at once (atomic moves never duplicate).
+  3. no orphan aux key (:deps/:results/:cfail/:lock) outlives its job hash.
+Final invariant after a full drain:
+  4. flows always settle - waiting-children is empty (no parent stranded).
+"""
+
+import random
+
+import pytest
+
+from toro import FlowChild as c  # noqa: N813
+from toro import Queue, Worker
+from toro.job import Job
+
+PREFIX = "torotest"
+
+_STATE_SETS = ("prioritized", "active", "delayed", "completed", "failed", "waiting_children")
+# counts() key -> the keys.py property backing it. They line up one-to-one except
+# `wait`: the waiting set is the priority-ordered zset, so its key is `prioritized`
+# (the state is named for what it means; the key for what it is).
+_COUNT_TO_SET = {
+    "wait": "prioritized",
+    "active": "active",
+    "delayed": "delayed",
+    "completed": "completed",
+    "failed": "failed",
+    "waiting-children": "waiting_children",
+}
+
+
+async def _members(q: Queue) -> dict[str, set[str]]:
+    """Current id membership of every state set (active is a LIST, rest ZSETs)."""
+    out: dict[str, set[str]] = {}
+    for name in _STATE_SETS:
+        key = getattr(q.keys, name)
+        ids = (
+            await q.redis.lrange(key, 0, -1)
+            if name == "active"
+            else await q.redis.zrange(key, 0, -1)
+        )
+        out[name] = set(ids)
+    return out
+
+
+async def _check_invariants(q: Queue) -> None:
+    members = await _members(q)
+
+    # 1. counts() agrees with the real cardinalities
+    counts = await q.counts()
+    for cname, sname in _COUNT_TO_SET.items():
+        assert counts[cname] == len(members[sname]), f"{cname} count {counts[cname]} != {sname}"
+
+    # 2. no id lives in two state sets at once
+    seen: dict[str, str] = {}
+    for sname, ids in members.items():
+        for jid in ids:
+            assert jid not in seen, f"{jid} in both {seen[jid]} and {sname}"
+            seen[jid] = sname
+
+    # 3. no orphan aux keys (a removed job leaves nothing behind)
+    base = q.keys.base
+    for suffix in (":deps", ":results", ":cfail", ":lock"):
+        for key in await q.redis.keys(f"{base}*{suffix}"):
+            jid = key[len(base) : -len(suffix)]
+            assert await q.redis.exists(q.keys.job(jid)), f"orphan {suffix} for {jid}"
+
+
+async def _process_one(w: Worker, *, succeed: bool) -> bool:
+    """Claim one runnable job and finish it (atomic, no fetch-next chaining since
+    the worker isn't run()). Returns False when nothing was claimable."""
+    loaded = await w._acquire()
+    if loaded is None:
+        return False
+    job_id, fields = loaded
+    job = Job.from_hash(job_id, fields)
+    if succeed:
+        await w._finish_completed(job, {"ok": 1})
+    else:
+        await w._finish_failed(job, RuntimeError("boom"))
+    return True
+
+
+async def _rand_member(q: Queue, rng: random.Random, *names: str) -> str | None:
+    members = await _members(q)
+    pool = sorted(jid for n in names for jid in members[n])
+    return rng.choice(pool) if pool else None
+
+
+# ---- the random operations (module level, so the test body stays simple) -----------
+
+
+async def _op_add(q, w, rng, ctr):
+    ctr["j"] += 1
+    kw = {"priority": rng.randint(0, 5)}
+    if rng.random() < 0.3:
+        kw["delay"] = rng.randint(1, 50)
+    if rng.random() < 0.3:
+        kw["attempts"] = rng.randint(1, 3)
+    await q.add(f"job{ctr['j']}", {"i": ctr["j"]}, **kw)
+
+
+async def _op_add_flow(q, w, rng, ctr):
+    ctr["j"] += 1
+    kids = [
+        c(f"child{ctr['j']}-{i}", {}, on_fail=rng.choice(["fail_parent", "continue"]))
+        for i in range(rng.randint(1, 3))
+    ]
+    await q.add_flow(f"flow{ctr['j']}", {}, children=kids)
+
+
+async def _op_process(q, w, rng, ctr):
+    await _process_one(w, succeed=rng.random() < 0.7)
+
+
+async def _op_retry(q, w, rng, ctr):
+    jid = await _rand_member(q, rng, "failed")
+    if jid:
+        await q.retry_job(jid)
+
+
+async def _op_remove(q, w, rng, ctr):
+    jid = await _rand_member(q, rng, "prioritized", "active", "delayed", "completed", "failed")
+    if jid:
+        await q.remove_job(jid)
+
+
+async def _op_clean(q, w, rng, ctr):
+    await q.clean(rng.choice(["wait", "delayed", "completed", "failed"]))
+
+
+async def _op_promote(q, w, rng, ctr):
+    jid = await _rand_member(q, rng, "delayed")
+    if jid:
+        await q.promote_job(jid)
+
+
+_OPS = [
+    _op_add,
+    _op_add_flow,
+    _op_process,
+    _op_process,  # process twice as often as the admin ops
+    _op_retry,
+    _op_remove,
+    _op_clean,
+    _op_promote,
+]
+
+
+async def _recover_and_drain(q: Queue, w: Worker) -> None:
+    """Fully recover the queue: retry every failed job (re-arming any parked flow -
+    the documented retry-all path), promote the delayed, process the runnable as
+    success, to quiescence. A flow stranded on a failed child only settles because
+    the child is retried too, which is exactly the guarantee under test."""
+    for _ in range(5000):
+        progressed = bool(await q.retry_all_failed())
+        for jid in await q.redis.zrange(q.keys.delayed, 0, -1):
+            progressed = await q.promote_job(jid) or progressed
+        progressed = await _process_one(w, succeed=True) or progressed
+        if not progressed:
+            return
+
+
+async def _settle_diagnostic(q: Queue) -> str:
+    parts = []
+    for pid in await q.redis.zrange(q.keys.waiting_children, 0, -1):
+        info = {}
+        for d in await q.redis.smembers(q.keys.deps(pid)):
+            exists = await q.redis.exists(q.keys.job(d))
+            info[d] = await q.redis.hget(q.keys.job(d), "state") if exists else "GONE"
+        parts.append(f"parent {pid}: deps={info}")
+    return " | ".join(parts)
+
+
+@pytest.mark.parametrize("seed", range(12))
+async def test_invariants_hold_under_random_ops(q, seed):
+    rng = random.Random(seed)  # noqa: S311 - a reproducible test fuzzer, not crypto
+    w = Worker(q.name, lambda j: None, prefix=PREFIX, connection=q.redis)
+    ctr = {"j": 0}
+
+    for _ in range(40):
+        await rng.choice(_OPS)(q, w, rng, ctr)
+        await _check_invariants(q)
+
+    await _recover_and_drain(q, w)
+    await _check_invariants(q)
+    settled = (await q.counts())["waiting-children"] == 0
+    assert settled, "a flow never settled -> " + await _settle_diagnostic(q)

toro_queue-0.5.0/tests/integration/test_leaks.py

@@ -0,0 +1,151 @@
+"""Resource hygiene: a job (or a whole flow) that runs its course must leave
+NOTHING behind - no hash, lock, logs, or flow aux keys, and no stray entry in the
+children index or the roots scratch - and a worker that stops must leak no
+background tasks. The fuzzer checks orphans mid-run; this pins the end state.
+"""
+
+import asyncio
+import contextlib
+
+from toro import FlowChild as c  # noqa: N813
+from toro import Queue, Worker
+from toro.job import Job
+
+PREFIX = "torotest"
+
+# Keys that legitimately outlive any single job: counters, the wakeup marker, the
+# state sets themselves, and the self-expiring metrics buckets / presence records.
+_INFRA = {
+    "id", "pc", "marker", "prioritized", "active", "delayed", "completed", "failed",
+    "waiting-children", "children", "stalled", "stalled-check", "meta-paused",
+    "limiter", "repeat", "workers", "departed", "roots-scratch", "events",
+}  # fmt: skip
+
+
+async def _noop(job):
+    return None
+
+
+async def _leaked_keys(q: Queue) -> list[str]:
+    """Per-job/aux keys left under the queue's namespace (infra keys excluded)."""
+    base = q.keys.base
+    leaked = []
+    for key in await q.redis.keys(base + "*"):
+        suffix = key[len(base) :]
+        if suffix in _INFRA or suffix.split(":")[0] in ("metrics", "worker", "repeat"):
+            continue
+        leaked.append(suffix)
+    return leaked
+
+
+async def _process(w: Worker, *, succeed: bool = True) -> bool:
+    loaded = await w._acquire()
+    if loaded is None:
+        return False
+    job = Job.from_hash(loaded[0], loaded[1])
+    if succeed:
+        await w._finish_completed(job, {"ok": 1})
+    else:
+        await w._finish_failed(job, RuntimeError("boom"))
+    return True
+
+
+# ---- a finished job leaves nothing behind ------------------------------------------
+
+
+async def test_completed_then_autoremoved_leaves_no_keys(q):
+    w = Worker(q.name, _noop, prefix=PREFIX, connection=q.redis)
+    job = await q.add("j", {"x": 1}, remove_on_complete=True)
+    await q.redis.rpush(q.keys.logs(job.id), "a log line")  # the job logged something
+
+    assert await _process(w, succeed=True)
+
+    assert await q.redis.exists(q.keys.job(job.id)) == 0
+    assert await q.redis.exists(q.keys.logs(job.id)) == 0  # logs cleaned too
+    assert await q.redis.exists(q.keys.lock(job.id)) == 0
+    assert await _leaked_keys(q) == []
+
+
+async def test_failed_then_autoremoved_leaves_no_keys(q):
+    w = Worker(q.name, _noop, prefix=PREFIX, connection=q.redis)
+    await q.add("j", {}, attempts=1, remove_on_fail=True)
+
+    assert await _process(w, succeed=False)
+
+    assert await _leaked_keys(q) == []
+
+
+# ---- a whole flow leaves nothing behind --------------------------------------------
+
+
+async def test_completed_flow_leaves_no_aux_keys_or_children_index(q):
+    w = Worker(q.name, _noop, prefix=PREFIX, connection=q.redis)
+    # every node auto-removes (a retained child legitimately stays in the index,
+    # so to prove the flow leaves NOTHING behind, remove the children too)
+    parent = await q.add_flow(
+        "p",
+        {},
+        children=[c("a", {}, remove_on_complete=True), c("b", {}, remove_on_complete=True)],
+        remove_on_complete=True,
+    )
+    # children index is populated while the flow exists
+    assert await q.redis.zcard(q.keys.children) == 2
+
+    # process both children, then the released parent
+    for _ in range(5):
+        if not await _process(w, succeed=True):
+            break
+
+    assert await q.redis.exists(q.keys.deps(parent.id)) == 0
+    assert await q.redis.exists(q.keys.results(parent.id)) == 0
+    assert await q.redis.exists(q.keys.cfail(parent.id)) == 0
+    assert await q.redis.zcard(q.keys.children) == 0  # index pruned with the nodes
+    assert await _leaked_keys(q) == []
+
+
+async def test_removed_flow_leaves_no_keys(q):
+    parent = await q.add_flow("p", {}, children=[c("a", {}), c("b", {}, children=[c("d", {})])])
+    assert await q.redis.zcard(q.keys.children) == 3  # a, b, d are all children
+
+    await q.remove_job(parent.id)  # cascades the whole subtree
+
+    assert await q.redis.zcard(q.keys.children) == 0
+    assert await _leaked_keys(q) == []
+
+
+# ---- the roots scratch key never persists ------------------------------------------
+
+
+async def test_roots_queries_leave_no_scratch_key(q):
+    await q.add_flow("p", {}, children=[c("a", {}), c("b", {})])
+    await q.add("solo", {})
+
+    for state in ("wait", "completed", "failed", "waiting-children", "active"):
+        await q.get_jobs_roots(state, 0, 50)
+    await q.roots_counts()
+
+    # ZDIFFSTORE writes the scratch then DELs it inside the same atomic script
+    assert await q.redis.exists(q.keys.roots_scratch) == 0
+
+
+# ---- a stopped worker leaks no background tasks ------------------------------------
+
+
+async def test_stopped_worker_leaks_no_tasks(q, run_until):
+    done = []
+
+    async def proc(job):
+        done.append(job.id)
+
+    worker = Worker(q.name, proc, prefix=PREFIX, connection=q.redis, heartbeat_interval=50)
+    task = asyncio.create_task(worker.run())
+    await q.add("j", {})
+    assert await run_until(lambda: len(done) >= 1, timeout=10)
+    await worker.stop()
+    task.cancel()
+    with contextlib.suppress(asyncio.CancelledError):
+        await task
+
+    # every background loop (stalled / heartbeat / promote) is done, none lingering
+    assert all(t.done() for t in worker._tasks), [t for t in worker._tasks if not t.done()]
+    await worker.stop()  # idempotent: a second stop must not raise

{toro_queue-0.4.0 → toro_queue-0.5.0}/toro/__init__.py

@@ -26,4 +26,4 @@ __all__ = [
     "ToroError",
     "Worker",
 ]
-__version__ = "0.4.0"
+__version__ = "0.5.0"

{toro_queue-0.4.0 → toro_queue-0.5.0}/toro/queue.py

@@ -932,11 +932,24 @@ class Queue:
     async def retry_job(self, job_id: str) -> bool:
         """Move a failed job back to the queue for another attempt.
 
-        Flow-aware: a failed flow parent whose children haven't all settled
-        re-parks in `waiting-children` instead of running with partial results;
-        a retried child re-joins its parked parent's barrier. Retrying parent
-        and children in any order (retry_all_failed does) recovers the flow.
+        Flow-aware: retrying a flow PARENT re-drives its whole failed subtree, not
+        the parent alone. The parent re-parks on every non-completed child
+        (completed children keep their collected results; failed and still-pending
+        ones stay in the barrier) and each failed descendant is re-queued root-first
+        - so a parent that failed because a child failed recovers in one call
+        instead of stranding on that still-failed child. A retried child re-joins
+        its parked parent's barrier. (retry_all_failed and retry_flow drive the
+        per-job script directly, so this convenience does not change them.)
+
+        The parent path keys off being a flow parent, not off being failed: called
+        on a parent that is not itself failed (e.g. still in-flight in
+        `waiting-children` with a `continue`-failed child), it re-drives the
+        subtree's failed nodes instead of being a no-op as it is for a non-failed
+        plain job. Pass a leaf child id to retry just that one job.
         """
+        job = await self.get_job(job_id)
+        if job is not None and job.children_ids:  # a flow parent: recover the subtree
+            return await self.retry_flow(job_id) > 0
         res = await self._retry_job(keys=self._retry_job_keys(job_id), args=[job_id, _now_ms()])
         return bool(res)
 

{toro_queue-0.4.0 → toro_queue-0.5.0}/uv.lock

@@ -424,7 +424,7 @@ wheels = [
 
 [[package]]
 name = "toro-queue"
-version = "0.4.0"
+version = "0.5.0"
 source = { editable = "." }
 dependencies = [
     { name = "croniter" },