rrq 0.2.5__tar.gz → 0.3.6__tar.gz

{rrq-0.2.5 → rrq-0.3.6}/.gitignore

@@ -1,3 +1,4 @@
 .venv/
 __pycache__/
-.git/
+.git/
+.vscode/

rrq-0.3.6/FUTURE.md

@@ -0,0 +1,33 @@
+# RRQ Multi-Worker Safety
+
+The core coordination primitives in RRQ are already designed for safe fan-out over multiple worker processes:
+
+- Jobs live in a Redis ZSET, with workers atomically acquiring a per-job lock (`SET NX PX`) before removing (`ZREM`) the job ID and executing it
+- The lock ensures that even if two workers race on the same job, only one proceeds past the `SET NX`
+- Once removed from the ZSET, the job can't re-appear until a retry or DLQ–requeue
+- Heartbeats and health keys are namespaced by worker ID (PID+UUID), so many workers can register themselves independently
+
+## Areas to Watch in Large Worker Fleets
+
+### 1. Two-step Lock+Pop Isn't Fully Atomic
+- If a worker acquires the lock but crashes before `ZREM`, the job stays in the queue until the lock TTL expires, and then another worker can grab it
+- In practice it's rare (must crash in that tiny window), but you can eliminate it by bundling "pop from ZSET + set lock" in a single Lua script
+
+### 2. Lock TTL vs. Job Duration
+- We set the lock TTL = `job_timeout + default_lock_timeout_extension_seconds`. If your handlers sometimes exceed that window, the lock can expire mid-run (though the job isn't in the queue anymore)
+- Consider increasing the extension, or implementing a "lock refresher" for very long tasks
+
+### 3. Graceful Shutdown & Task Drain
+- Workers will stop polling in burst mode or on a shutdown signal, then "drain" in-flight tasks up to `worker_shutdown_grace_period_seconds`
+- Beyond that they cancel. Make sure your handlers handle `CancelledError` gracefully
+
+### 4. Logging & Observability
+- If you're tailing stdout/stderr from many workers, add the worker ID (and queue list) to your log formatter to keep things straight
+
+### 5. Health-Check TTLs
+- The heartbeat loop writes a Redis key with a buffer TTL
+- If your network is flaky or your workers get paused (e.g. in a debugger), you may see transient "missing" health keys
+
+## Summary
+
+With these caveats in mind, you can absolutely spin up dozens (or hundreds) of RRQ worker processes against the same Redis instance, each pulling from the same or different queue names. The locking, queueing, and retry/DLQ logic will keep them from stepping on each other.

{rrq-0.2.5 → rrq-0.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rrq
-Version: 0.2.5
+Version: 0.3.6
 Summary: RRQ is a Python library for creating reliable job queues using Redis and asyncio
 Project-URL: Homepage, https://github.com/getresq/rrq
 Project-URL: Bug Tracker, https://github.com/getresq/rrq/issues
@@ -21,17 +21,12 @@ Requires-Dist: redis[hiredis]<6,>=4.2.0
 Requires-Dist: watchfiles>=0.19.0
 Provides-Extra: dev
 Requires-Dist: pytest-asyncio>=0.26.0; extra == 'dev'
+Requires-Dist: pytest-cov>=6.0.0; extra == 'dev'
 Requires-Dist: pytest>=8.3.5; extra == 'dev'
 Description-Content-Type: text/markdown
 
 # RRQ: Reliable Redis Queue
 
- ____    ____     ___
-|  _ \  |  _ \   / _ \
-| |_) | | |_) | | | | |
-|  _ <  |  _ <  | |_| |
-|_| \_\ |_| \_\  \__\_\
-
 RRQ is a Python library for creating reliable job queues using Redis and `asyncio`, inspired by [ARQ (Async Redis Queue)](https://github.com/samuelcolvin/arq). It focuses on providing at-least-once job processing semantics with features like automatic retries, job timeouts, dead-letter queues, and graceful worker shutdown.
 
 ## Core Components
@@ -58,10 +53,20 @@ RRQ is a Python library for creating reliable job queues using Redis and `asynci
 *   **Worker Health Checks**: Workers periodically update a health key in Redis with a TTL, allowing monitoring systems to track active workers.
 *   **Deferred Execution**: Jobs can be scheduled to run at a future time using `_defer_by` or `_defer_until`.
     *Note: Using deferral with a specific `_job_id` will effectively reschedule the job associated with that ID to the new time, overwriting its previous definition and score. It does not create multiple distinct scheduled jobs with the same ID.*
+    *To batch multiple enqueue calls into a single deferred job (and prevent duplicates within the defer window), combine `_unique_key` with `_defer_by`. For example:*
+
+      ```python
+      await client.enqueue(
+          "process_updates",
+          item_id=123,
+          _unique_key="update:123",
+          _defer_by=10,
+      )
+      ```
 
 ## Basic Usage
 
-*(See [`rrq_example.py`](examples/rrq_example.py) in the project root for a runnable example)*
+*(See [`rrq_example.py`](https://github.com/GetResQ/rrq/tree/master/example) in the project root for a runnable example)*
 
 **1. Define Handlers:**
 
@@ -165,10 +170,9 @@ rrq <command> [options]
 
 - **`worker run`**: Run an RRQ worker process to process jobs from queues.
   ```bash
-  rrq worker run [--burst] [--detach] --settings <settings_path>
+  rrq worker run [--burst] --settings <settings_path>
   ```
   - `--burst`: Run in burst mode (process one job/batch then exit).
-  - `--detach`: Run the worker in the background.
   - `--settings`: Python settings path for application worker settings (e.g., `myapp.worker_config.rrq_settings`).
 
 - **`worker watch`**: Run an RRQ worker with auto-restart on file changes in a specified directory.

{rrq-0.2.5 → rrq-0.3.6}/README.md

@@ -1,11 +1,5 @@
 # RRQ: Reliable Redis Queue
 
- ____    ____     ___
-|  _ \  |  _ \   / _ \
-| |_) | | |_) | | | | |
-|  _ <  |  _ <  | |_| |
-|_| \_\ |_| \_\  \__\_\
-
 RRQ is a Python library for creating reliable job queues using Redis and `asyncio`, inspired by [ARQ (Async Redis Queue)](https://github.com/samuelcolvin/arq). It focuses on providing at-least-once job processing semantics with features like automatic retries, job timeouts, dead-letter queues, and graceful worker shutdown.
 
 ## Core Components
@@ -32,10 +26,20 @@ RRQ is a Python library for creating reliable job queues using Redis and `asynci
 *   **Worker Health Checks**: Workers periodically update a health key in Redis with a TTL, allowing monitoring systems to track active workers.
 *   **Deferred Execution**: Jobs can be scheduled to run at a future time using `_defer_by` or `_defer_until`.
     *Note: Using deferral with a specific `_job_id` will effectively reschedule the job associated with that ID to the new time, overwriting its previous definition and score. It does not create multiple distinct scheduled jobs with the same ID.*
+    *To batch multiple enqueue calls into a single deferred job (and prevent duplicates within the defer window), combine `_unique_key` with `_defer_by`. For example:*
+
+      ```python
+      await client.enqueue(
+          "process_updates",
+          item_id=123,
+          _unique_key="update:123",
+          _defer_by=10,
+      )
+      ```
 
 ## Basic Usage
 
-*(See [`rrq_example.py`](examples/rrq_example.py) in the project root for a runnable example)*
+*(See [`rrq_example.py`](https://github.com/GetResQ/rrq/tree/master/example) in the project root for a runnable example)*
 
 **1. Define Handlers:**
 
@@ -139,10 +143,9 @@ rrq <command> [options]
 
 - **`worker run`**: Run an RRQ worker process to process jobs from queues.
   ```bash
-  rrq worker run [--burst] [--detach] --settings <settings_path>
+  rrq worker run [--burst] --settings <settings_path>
   ```
   - `--burst`: Run in burst mode (process one job/batch then exit).
-  - `--detach`: Run the worker in the background.
   - `--settings`: Python settings path for application worker settings (e.g., `myapp.worker_config.rrq_settings`).
 
 - **`worker watch`**: Run an RRQ worker with auto-restart on file changes in a specified directory.

{rrq-0.2.5 → rrq-0.3.6}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "rrq"
-version = "0.2.5"
+version = "0.3.6"
 authors = [
     { name = "Mazdak Rezvani", email = "mazdak@me.com" },
 ]
@@ -33,6 +33,7 @@ dependencies = [
 dev = [
     "pytest>=8.3.5",
     "pytest-asyncio>=0.26.0",
+    "pytest-cov>=6.0.0",
 ]
 
 [project.urls]
@@ -40,7 +41,7 @@ dev = [
 "Bug Tracker" = "https://github.com/getresq/rrq/issues"
 
 [project.scripts]
-rrq = "rrq.rrq:rrq" # Assumes rrq.py is at the root of your source directory
+rrq = "rrq.cli:rrq"
 
 [tool.pytest.ini_options]
-asyncio_default_fixture_loop_scope = "function"
+asyncio_default_fixture_loop_scope = "function"