toro-queue 0.1.0__tar.gz

toro_queue-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,41 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+    services:
+      redis:
+        image: redis:7-alpine
+        ports:
+          - 6379:6379
+        options: >-
+          --health-cmd "redis-cli ping"
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Sync dependencies
+        run: uv sync
+
+      - name: Lint (ruff)
+        run: uv run ruff check .
+
+      - name: Format (ruff)
+        run: uv run ruff format --check .
+
+      - name: Type check (ty)
+        run: uv run ty check
+
+      - name: Tests (unit + integration)
+        run: uv run pytest -m "unit or integration"

toro_queue-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,37 @@
+name: Release
+
+# A version tag (v*) builds and publishes to PyPI via trusted publishing (OIDC)
+# — no API tokens stored. The publish job runs in the `pypi` environment, which
+# must match the trusted publisher registered on PyPI.
+on:
+  push:
+    tags: ["v*"]
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+      - name: Build sdist + wheel
+        run: uv build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

toro_queue-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+.venv/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+*.pyc
+/dist/
+/build/
+
+.coverage

toro_queue-0.1.0/.pre-commit-config.yaml

@@ -0,0 +1,18 @@
+# Astral toolchain on every commit: ruff (lint + format) and ty (type check).
+# Install once with:  uvx pre-commit install
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.15
+    hooks:
+      - id: ruff-check
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: local
+    hooks:
+      - id: ty
+        name: ty (type check)
+        entry: uv run ty check
+        language: system
+        types: [python]
+        pass_filenames: false

toro_queue-0.1.0/DESIGN.md

@@ -0,0 +1,114 @@
+# toro — design & architecture
+
+`toro` is an async-first, Redis-backed job queue for Python. This doc describes
+how it works and the reasoning behind the core choices.
+
+> Prior art: the design draws on the proven Redis-queue patterns popularized by
+> the Node.js ecosystem (atomic Lua transitions, reliable blocking pop, lock +
+> stalled recovery). Credit where due — but the rest of this doc describes
+> toro's own design.
+
+## Core principles
+
+### 1. Atomic state transitions via Lua
+Every state move (`wait→active`, `active→completed/failed/delayed`,
+`delayed→wait`) is a single Redis Lua script. Redis runs each script atomically,
+so multi-key "check-then-act" sequences can't interleave. This kills three
+classes of race:
+- **pop-then-lock gap** — two workers both grabbing the same job.
+- **finish-after-steal** — a worker committing a result for a job a stalled-sweep
+  already re-queued (guarded with a token check + `LREM active` returning 0).
+- **priority insertion vs concurrent consume** — `LINSERT` position computed and
+  used in one snapshot.
+
+Scripts live in `scripts.py`, registered via `redis.asyncio`'s `register_script`.
+The Python side only assembles KEYS/ARGV — the guarantees live in the Lua.
+
+### 2. Reliable fetch: blocking pop on a dedicated connection
+The worker pops `wait→active` with `BLMOVE` (the non-deprecated `BRPOPLPUSH`) so
+the job is on `active` *before* the worker starts. If the worker dies mid-job,
+the id is still on `active` and the stalled sweep (principle 3) recovers it. The
+blocking call should use a **separate** Redis connection, since a blocked
+connection can't issue other commands.
+
+→ **toro:** uses `BLMOVE wait active RIGHT LEFT`. TODO: isolate the blocking pop
+on its own connection.
+
+### 3. Locks + token + stalled recovery — the at-least-once guarantee
+- On move-to-active, set `<q>:<id>:lock = <token>  PX lockDuration` (default 30s).
+  The token is a per-worker UUID — only the owner can renew.
+- A renewer extends the lock every `lockDuration/2` (token-guarded `GET==token`
+  then re-`SET PX`); a successful renew also `SREM`s the job from the `stalled`
+  set.
+- A sweep runs every `stalledInterval` (~30s), throttled cluster-wide by a
+  `stalled-check` PX key. It is **mark-and-sweep**: sweep the `stalled` set (any
+  member whose `:lock` is gone → `LREM active`, push back to `wait`, or fail if
+  `stalledCounter > maxStalledCount`, default 1), then re-`SADD` the current
+  `active` list into `stalled`. Live workers renew and remove themselves before
+  the next sweep; only genuinely dead jobs remain.
+
+**Guarantee:** at-least-once. A job is never lost while Redis persists, but its
+handler may run more than once (bounded by `maxStalledCount`). Exactly-once
+*result commit* is enforced by the token-guarded lock at finish time, not by
+preventing duplicate handler execution.
+
+→ **toro:** NOT YET IMPLEMENTED. Top priority after the core. Plan: per-job lock
+key + token, an `asyncio` lock-renewal task per in-flight job, and a background
+`_stalled_loop` running a mark-and-sweep Lua script.
+
+### 4. One delay timer, pub/sub-woken
+Delayed jobs live in a ZSET scored `(ts << 12) | (counter & 0xFFF)` — low 12 bits
+keep FIFO order among same-ms jobs. Arm a single timer for the next due job
+(capped by a guard interval) and re-arm it instantly when a sooner job is added,
+via a pub/sub message on the `delayed` channel. A promotion script moves all due
+jobs to `wait`.
+
+→ **toro:** currently a 1s poll in `_promote_loop` (simple, correct, slightly
+wasteful). Upgrade path: single `asyncio` timer re-armed via a pub/sub
+subscriber. Also adopt the packed score for correct same-ms ordering.
+
+### 5. Fetch-next-in-finish  ✅ done
+The finish script commits the current job and pops + locks the next one in the
+same round trip — skipped when shutting down (fetch flag), so the queue drains
+cleanly.
+
+→ **toro:** implemented. All job acquisition funnels through one shared Lua
+routine (`lockAndLoad` / `acquireNext` in `scripts.py`), used by both the
+blocking-wakeup path (`MOVE_TO_ACTIVE`) and the fetch-next tail of
+`MOVE_TO_COMPLETED` / `MOVE_TO_FAILED`. This routine is the seed of a future
+`moveToActive`: to add priorities/markers we change only *which* job
+`acquireNext` picks — `lockAndLoad` and every caller stay untouched.
+
+**Measured:** process throughput went ~6,080 → ~13,900 jobs/s (~2.3×) at
+concurrency 20. Notably cmds/job barely changed (13 → 12) — the win is fewer
+*round trips* (the old `BLMOVE` + `MOVE_TO_ACTIVE` + `HGETALL` per job collapse
+into the finish call), not less server work.
+
+## Higher-level features (roadmap)
+- **Priorities:** ✅ done — and we deliberately *diverge* from the common approach. The usual design
+  inserts into `wait` with `LINSERT` (O(N)) and keeps a separate `wait`
+  fast-lane that strictly beats `prioritized` (priority-0 jobs can starve
+  prioritized ones). toro instead puts **every** job in one `prioritized` ZSET
+  scored `(PRIORITY_OFFSET - priority) * 2^32 + seq` — a single GLOBAL order,
+  higher priority = more urgent, FIFO within a level, no fast lane. Because a
+  single ZSET can't be `BLMOVE`'d, this brings in the base marker: producers
+  `ZADD marker 0 "0"` (idempotent), idle workers `BZPOPMIN marker`, and the
+  atomic `MOVE_TO_ACTIVE` does `ZPOPMIN prioritized` → active → lock. The 1s
+  delay poll still promotes delayed → prioritized (delay-marker is future work).
+- **Repeatable/cron:** a `repeat` ZSET of schedule entries; each occurrence
+  schedules its successor as a *delayed* job with a deterministic id. Port with
+  `croniter`.
+- **Rate limiting:** a `PSETEX`/`INCR` counter per window (optionally per group);
+  limited jobs parked in `delayed`. Disables fetch-next.
+- **Events:** Redis pub/sub or Streams. Streams give replay + consumer groups,
+  which suits an async dashboard.
+- **Auto-removal:** `removeOnComplete`/`removeOnFail` (bool/count/{count,age})
+  enforced inside the finish script, not by a sweeper.
+
+## Python-specific choices
+- **async-first**: `redis.asyncio`, `async def` processors, one event loop.
+  Concurrency = N `asyncio` tasks sharing the loop.
+- **Cluster:** use a `{braces}` hash-tag prefix so all keys for a queue share a
+  slot (multi-key Lua needs one slot).
+- **Lua deps:** plain JSON, integers passed as ARGV — keeps scripts portable
+  across Redis builds (no `cmsgpack`/`bit`/`cjson` requirement).

toro_queue-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 ilovepixelart
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

toro_queue-0.1.0/PKG-INFO

@@ -0,0 +1,127 @@
+Metadata-Version: 2.4
+Name: toro-queue
+Version: 0.1.0
+Summary: An async-first, Redis-backed job queue for Python.
+Project-URL: Homepage, https://github.com/ilovepixelart/toro
+Project-URL: Repository, https://github.com/ilovepixelart/toro
+Project-URL: Issues, https://github.com/ilovepixelart/toro/issues
+Author: ilovepixelart
+License-Expression: MIT
+License-File: LICENSE
+Keywords: async,asyncio,jobs,queue,redis,scheduler,task-queue,worker
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: croniter>=2.0
+Requires-Dist: redis>=5.0
+Description-Content-Type: text/markdown
+
+# toro 🐂
+
+An **async-first**, Redis-backed job queue for Python. Every state transition is
+an atomic Lua script; producing and processing are `asyncio` end to end.
+
+```bash
+pip install toro-queue      # the import name is `toro`
+```
+
+> Installed as **`toro-queue`** on PyPI (the name `toro` was taken), but you
+> `import toro`. See [DESIGN.md](https://github.com/ilovepixelart/toro/blob/main/DESIGN.md) for the architecture and the
+> at-least-once reliability model.
+
+## Why toro
+
+- **Async-native.** Enqueue and process with `async`/`await` — no thread pools,
+  no sync bridge. A natural fit for FastAPI, aiohttp, or any asyncio app.
+- **Atomic by construction.** Claims, retries, promotions and finishes are Lua
+  scripts, so a job can't be lost or double-committed between two round trips.
+- **At-least-once delivery.** Per-job locks + a background mark-and-sweep recover
+  jobs from workers that crashed — without the visibility-timeout double-delivery
+  trap of some other queues.
+- **Typed.** Ships `py.typed`; the public API is fully annotated.
+
+## Features
+
+| | |
+|---|---|
+| **Enqueue** | delayed jobs, global **priorities** (FIFO within a band) |
+| **Retries** | fixed or exponential **backoff**, capped attempts |
+| **Schedules** | repeatable **cron** and fixed-interval (`every`) jobs |
+| **Rate limiting** | queue-wide token bucket shared across all workers |
+| **Dedup** | custom (idempotent) job ids + a throttle window (`{id, ttl}`) |
+| **Auto-removal** | keep the last N and/or finished-within-age completed/failed |
+| **Reliability** | per-job locks, lock renewal, stalled-job recovery |
+| **Observability** | progress, per-job logs, lifecycle events, `await result()` |
+| **Lifecycle** | pause / resume, graceful shutdown that drains in-flight jobs |
+| **Dashboard** | [matador](https://github.com/ilovepixelart/matador) — a live web UI |
+
+## Quick start
+
+```python
+import asyncio
+from toro import Queue, Worker
+
+async def main():
+    queue = Queue("emails")
+    await queue.add("welcome", {"to": "ada@example.com"})
+
+    async def process(job):
+        print("sending", job.data)
+        return {"ok": True}
+
+    worker = Worker("emails", process, concurrency=8)
+    worker.on("completed", lambda job, result: print("done", job.id))
+    await worker.run()
+
+asyncio.run(main())
+```
+
+## A taste of the options
+
+```python
+# Priorities, delay, and retry-with-backoff
+await queue.add("report", data, priority=10, delay=5000,
+                attempts=5, backoff={"type": "exponential", "delay": 1000})
+
+# Idempotent custom id (a second add with the same id is ignored)
+await queue.add("charge", data, job_id="order-1234")
+
+# A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
+await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")
+
+# Queue-wide rate limit: at most 100 jobs / second across every worker
+worker = Worker("emails", process, rate_limit={"max": 100, "duration": 1000})
+
+# Wait for a result from the producer side
+job = await queue.add("resize", {"src": "a.png"})
+print(await job.result(timeout=30))
+```
+
+## Develop
+
+Managed with [uv](https://astral.sh/uv); the Astral toolchain throughout.
+
+```bash
+uv sync                          # venv + deps + dev group
+uv run ruff check .              # lint  (strict: select = ALL)
+uv run ruff format .             # format
+uv run ty check                  # type check
+uv run pytest -m "unit or integration"   # tests (integration needs Redis on :6379)
+uv run python examples/basic.py
+```
+
+The suite is a pyramid — `-m unit` (fast, no Redis), `-m integration` (Redis),
+and `-m load` (the open-loop benchmark harness in `tests/load/`).
+
+## License
+
+[MIT](https://github.com/ilovepixelart/toro/blob/main/LICENSE)

toro_queue-0.1.0/README.md

@@ -0,0 +1,100 @@
+# toro 🐂
+
+An **async-first**, Redis-backed job queue for Python. Every state transition is
+an atomic Lua script; producing and processing are `asyncio` end to end.
+
+```bash
+pip install toro-queue      # the import name is `toro`
+```
+
+> Installed as **`toro-queue`** on PyPI (the name `toro` was taken), but you
+> `import toro`. See [DESIGN.md](https://github.com/ilovepixelart/toro/blob/main/DESIGN.md) for the architecture and the
+> at-least-once reliability model.
+
+## Why toro
+
+- **Async-native.** Enqueue and process with `async`/`await` — no thread pools,
+  no sync bridge. A natural fit for FastAPI, aiohttp, or any asyncio app.
+- **Atomic by construction.** Claims, retries, promotions and finishes are Lua
+  scripts, so a job can't be lost or double-committed between two round trips.
+- **At-least-once delivery.** Per-job locks + a background mark-and-sweep recover
+  jobs from workers that crashed — without the visibility-timeout double-delivery
+  trap of some other queues.
+- **Typed.** Ships `py.typed`; the public API is fully annotated.
+
+## Features
+
+| | |
+|---|---|
+| **Enqueue** | delayed jobs, global **priorities** (FIFO within a band) |
+| **Retries** | fixed or exponential **backoff**, capped attempts |
+| **Schedules** | repeatable **cron** and fixed-interval (`every`) jobs |
+| **Rate limiting** | queue-wide token bucket shared across all workers |
+| **Dedup** | custom (idempotent) job ids + a throttle window (`{id, ttl}`) |
+| **Auto-removal** | keep the last N and/or finished-within-age completed/failed |
+| **Reliability** | per-job locks, lock renewal, stalled-job recovery |
+| **Observability** | progress, per-job logs, lifecycle events, `await result()` |
+| **Lifecycle** | pause / resume, graceful shutdown that drains in-flight jobs |
+| **Dashboard** | [matador](https://github.com/ilovepixelart/matador) — a live web UI |
+
+## Quick start
+
+```python
+import asyncio
+from toro import Queue, Worker
+
+async def main():
+    queue = Queue("emails")
+    await queue.add("welcome", {"to": "ada@example.com"})
+
+    async def process(job):
+        print("sending", job.data)
+        return {"ok": True}
+
+    worker = Worker("emails", process, concurrency=8)
+    worker.on("completed", lambda job, result: print("done", job.id))
+    await worker.run()
+
+asyncio.run(main())
+```
+
+## A taste of the options
+
+```python
+# Priorities, delay, and retry-with-backoff
+await queue.add("report", data, priority=10, delay=5000,
+                attempts=5, backoff={"type": "exponential", "delay": 1000})
+
+# Idempotent custom id (a second add with the same id is ignored)
+await queue.add("charge", data, job_id="order-1234")
+
+# A repeatable schedule (cron or every-N-ms); "run now" with trigger_scheduler
+await queue.add_scheduler("nightly-rollup", cron="0 0 * * *")
+
+# Queue-wide rate limit: at most 100 jobs / second across every worker
+worker = Worker("emails", process, rate_limit={"max": 100, "duration": 1000})
+
+# Wait for a result from the producer side
+job = await queue.add("resize", {"src": "a.png"})
+print(await job.result(timeout=30))
+```
+
+## Develop
+
+Managed with [uv](https://astral.sh/uv); the Astral toolchain throughout.
+
+```bash
+uv sync                          # venv + deps + dev group
+uv run ruff check .              # lint  (strict: select = ALL)
+uv run ruff format .             # format
+uv run ty check                  # type check
+uv run pytest -m "unit or integration"   # tests (integration needs Redis on :6379)
+uv run python examples/basic.py
+```
+
+The suite is a pyramid — `-m unit` (fast, no Redis), `-m integration` (Redis),
+and `-m load` (the open-loop benchmark harness in `tests/load/`).
+
+## License
+
+[MIT](https://github.com/ilovepixelart/toro/blob/main/LICENSE)

toro_queue-0.1.0/bench/bench.py

@@ -0,0 +1,73 @@
+"""Measure toro's hot path: enqueue + process throughput and Redis cmds/job.
+
+Compares against the pre-fetch-next baseline we recorded (~6,080 jobs/s,
+13 cmds/job) to show the round-trip savings.
+
+Usage:  uv run python bench/bench.py [N] [concurrency]
+"""
+
+import asyncio
+import sys
+import time
+
+import redis.asyncio as aioredis
+
+from toro import Queue, Worker
+
+URL = "redis://localhost:6379"
+N = int(sys.argv[1]) if len(sys.argv) > 1 else 5000
+CONC = int(sys.argv[2]) if len(sys.argv) > 2 else 20
+
+
+async def _clear(r, base):
+    keys = await r.keys(base + "*")
+    if keys:
+        await r.delete(*keys)
+
+
+async def _cmds(r):
+    return (await r.info("stats"))["total_commands_processed"]
+
+
+async def main():
+    r = aioredis.from_url(URL, decode_responses=True)
+    await _clear(r, "bench:b:")
+    q = Queue("b", url=URL, prefix="bench")
+
+    c0 = await _cmds(r)
+    t0 = time.perf_counter()
+    for i in range(N):
+        await q.add("bench", {"i": i})
+    enq = time.perf_counter() - t0
+    enq_cmds = await _cmds(r) - c0
+
+    done = asyncio.Event()
+    count = 0
+
+    async def handler(job):
+        nonlocal count
+        count += 1
+        if count >= N:
+            done.set()
+
+    worker = Worker("b", handler, url=URL, prefix="bench", concurrency=CONC)
+    c1 = await _cmds(r)
+    t1 = time.perf_counter()
+    task = asyncio.create_task(worker.run())
+    await asyncio.wait_for(done.wait(), timeout=120)
+    proc = time.perf_counter() - t1
+    proc_cmds = await _cmds(r) - c1
+
+    await worker.stop()
+    task.cancel()
+    await q.close()
+    await r.aclose()
+
+    print(f"N={N}, concurrency={CONC}\n")
+    print(f"enqueue:  {N / enq:>8,.0f} jobs/s   {enq_cmds / N:>5.1f} cmds/job")
+    print(f"process:  {count / proc:>8,.0f} jobs/s   {proc_cmds / count:>5.1f} cmds/job")
+    print("\nbaseline (pre fetch-next): ~6,080 jobs/s, 13.0 cmds/job")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

toro_queue-0.1.0/examples/basic.py

@@ -0,0 +1,42 @@
+"""Minimal end-to-end demo.
+
+Run a Redis on localhost:6379, then:  python examples/basic.py
+"""
+
+import asyncio
+
+from toro import Queue, Worker
+
+
+async def main():
+    queue = Queue("emails")
+
+    # Producer: enqueue a few jobs, one delayed, one that will fail-and-retry.
+    await queue.add("welcome", {"to": "ada@example.com"})
+    await queue.add("welcome", {"to": "alan@example.com"}, delay=2000)
+    await queue.add("flaky", {"n": 1}, attempts=3, backoff={"type": "exponential", "delay": 500})
+
+    # Consumer.
+    async def process(job):
+        if job.name == "flaky" and job.attempts_made < 3:
+            raise RuntimeError(f"transient failure (try {job.attempts_made})")
+        print(f"  processed {job.name} #{job.id} -> {job.data}")
+        return {"ok": True}
+
+    worker = Worker("emails", process, concurrency=4)
+    worker.on("completed", lambda job, res: print(f"  ✓ completed {job.name} #{job.id}"))
+    worker.on("retrying", lambda job, exc: print(f"  ↻ retrying #{job.id}: {exc}"))
+    worker.on("failed", lambda job, exc: print(f"  ✗ failed #{job.id}: {exc}"))
+
+    runner = asyncio.create_task(worker.run())
+
+    await asyncio.sleep(5)  # let the delayed + retried jobs flush
+    print("counts:", await queue.counts())
+
+    await worker.stop()
+    runner.cancel()
+    await queue.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

toro_queue-0.1.0/examples/stalled.py

@@ -0,0 +1,57 @@
+"""Demo: a worker dies mid-job and another recovers it — exactly one completion.
+
+Run a Redis on localhost:6379, then:  python examples/stalled.py
+"""
+
+import asyncio
+
+from toro import Queue, Worker
+
+
+async def main():
+    queue = Queue("recovery_demo")
+    # clean slate for the demo
+    keys = await queue.redis.keys(queue.keys.base + "*")
+    if keys:
+        await queue.redis.delete(*keys)
+
+    await queue.add("report", {"id": 42})
+
+    async def hangs(job):
+        print(f"  [zombie] picked up #{job.id}, then hangs forever...")
+        await asyncio.sleep(30)  # never finishes in time
+        return {"by": "zombie"}
+
+    async def works(job):
+        print(f"  [healthy] recovered #{job.id}, processing")
+        return {"by": "healthy"}
+
+    # A zombie: short lock, never renews, doesn't run stalled checks.
+    zombie = Worker(
+        "recovery_demo", hangs, lock_duration=500, renew_locks=False, stalled_interval=0
+    )
+    # A healthy worker that sweeps for stalled jobs every 500ms.
+    healthy = Worker("recovery_demo", works, stalled_interval=500, max_stalled_count=3)
+    healthy.on("stalled", lambda jid: print(f"  [healthy] detected #{jid} stalled -> requeued"))
+    healthy.on("completed", lambda j, r: print(f"  ✓ #{j.id} completed by {r['by']}"))
+    zombie.on(
+        "lock-lost",
+        lambda jid: print(f"  [zombie] woke up, but #{jid} was taken — finish rejected"),
+    )
+
+    zt = asyncio.create_task(zombie.run())
+    await asyncio.sleep(0.4)  # let the zombie grab the job
+    ht = asyncio.create_task(healthy.run())
+
+    await asyncio.sleep(3)
+    print("counts:", await queue.counts())
+
+    await zombie.stop()
+    await healthy.stop()
+    zt.cancel()
+    ht.cancel()
+    await queue.close()
+
+
+if __name__ == "__main__":
+    asyncio.run(main())