vtrr-queue 0.1.0__tar.gz → 0.2.1__tar.gz

{vtrr_queue-0.1.0 → vtrr_queue-0.2.1}/CLAUDE.md

@@ -20,22 +20,27 @@ src/vtrr_queue/
   queue.py          # all logic lives here — single file by design
   py.typed          # PEP 561 marker
   scripts/
+    common.lua      # prelude prepended to every script: vtrr_enqueue (scoring) + vtrr_reclaim
     enqueue.lua     # bulk enqueue, called by VTRRTask.queue()
-    dequeue.lua     # pops one task, called by each @vtrr.task celery wrapper
-tests/              # empty — needs fakeredis-based tests
+    dequeue.lua     # reclaims, then leases one task; called by each @vtrr.task celery wrapper
+    reclaim.lua     # public VTRRQueue.reclaim()
+    ack.lua         # fenced ack of a lease, VTRRQueue.ack()
+    renew.lua       # fenced re-lease used by retry runs, VTRRQueue._renew_lease()
+tests/              # fakeredis-based tests
 ```
 
 ## Key design constraints
 
 - **Single source file** (`queue.py`) — do not split into multiple modules unless the file becomes unmanageable.
 - **No Django dependency** — the library must work with plain Celery + redis-py. Django integration is the consumer's responsibility (`get_redis_connection`, settings wiring, etc.).
-- **No enqueue.lua modification without updating dequeue.lua** — the two scripts share assumptions about key layout and must stay consistent. Both are atomic Lua scripts; never replace them with multi-command Python.
+- **No script modification without checking the others** — the scripts share `common.lua` and assumptions about key layout and must stay consistent. Both are atomic Lua scripts; never replace them with multi-command Python.
 - **`@vtrr.task` IS the Celery task** — the decorator wraps the user function in a Celery task whose body calls `vtrr._dequeue()` then invokes the function directly. One broker hop total; no intermediate dispatcher task.
+- **One task per `VTRRQueue`** — `@vtrr.task` raises `ValueError` for a second function. Triggers are interchangeable and run whatever item they dequeue, so a mixed queue would run items under the wrong task's Celery options. `vtrr._task` is the single registered task; the item payload's `task_name` is checked against it.
 - **`celery_queue` is passed explicitly** in every `apply_async` call, bypassing the consumer's `task_routes`.
 
 ## Redis key contract
 
-Each `VTRRQueue` instance namespaces its four keys under `vtrr:{name}:*`. Do not rename without updating both Lua scripts and the README.
+Each `VTRRQueue` instance namespaces its keys under `vtrr:{name}:*`. Do not rename without updating the Lua scripts and the README. Every script receives the same `KEYS` list (order defined in `common.lua`).
 
 | Key | Structure |
 |---|---|
@@ -46,7 +51,7 @@ Each `VTRRQueue` instance namespaces its four keys under `vtrr:{name}:*`. Do not
 
 JSON payload shape: `{"task_name": str, "args": list, "kwargs": dict}`
 
-All four keys are deleted/reset when the queue drains (handled in `dequeue.lua`).
+The first four keys are deleted/reset when the queue drains (handled in `dequeue.lua`). Lease state (`meta`, `inflight`, `inflight_task`, `lease`, `timeouts`; see README) is untouched by a drain. Dequeue grants a lease rather than deleting the item; reclaim rescores expired items through the same `vtrr_enqueue` function a fresh enqueue uses — never duplicate that logic.
 
 ## Task weights
 
@@ -56,6 +61,10 @@ Each task dict passed to `.queue()` may include a `"weight"` key (int, default `
 
 `_schedule_workers(num_tasks)` schedules `min(max_concurrency - workers_scheduled, num_tasks)` Celery tasks after every `.queue()` call. `_get_workers_scheduled()` counts broker list lengths across Celery's four Redis priority suffixes (`""`, `":3"`, `":6"`, `":9"`). It returns `0` on any error so scheduling may over-fire slightly — this is intentional and safe.
 
+## Periodic reclaim
+
+Recovery of work from dead workers must not depend on Celery's `acks_late` / `reject_on_worker_lost` (they stay allowed task options but the library never requires them). Trigger messages carry no payload, so a lost trigger loses nothing; `reclaim_and_dispatch()` (reclaim expired leases, then `_schedule_workers(zcard(queue))` using any registered task as the trigger) is what wakes workers back up. With `reclaim_interval` set, the constructor registers the `vtrr_queue.reclaim.{name}` Celery task and merges its entry into `celery_app.conf.beat_schedule`; `reclaim_beat_schedule()` returns the entry for apps that assign the schedule wholesale.
+
 ## Development setup
 
 ```

{vtrr_queue-0.1.0 → vtrr_queue-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: vtrr-queue
-Version: 0.1.0
+Version: 0.2.1
 Summary: Virtual-time round-robin fair queue on Redis
 Author-email: Nicolas Johnson <nicolas.johnson@c4ads.org>
 License-Expression: Apache-2.0
@@ -36,6 +36,7 @@ Integrates with an existing Celery + Redis setup. The library owns the dequeue l
 - Python >= 3.12
 - Redis (used for both the VTRR queue state and as the Celery broker)
 - Celery >= 5.0
+- Optional but recommended: Celery Beat (or any scheduler that can call `vtrr.reclaim_and_dispatch()`). Queueing and running tasks work without it, but recovering work from a dead worker when the queue is otherwise idle does not. See [Periodic reclaim](#periodic-reclaim-with-celery-beat)
 
 
 ## Installation
@@ -107,6 +108,9 @@ Use a dedicated Redis DB index for the VTRR queue so its keys don't collide with
 | `name` | `str` | required | Unique name for this queue instance; namespaces all Redis keys as `vtrr:{name}:*` |
 | `celery_queue` | `str` | `"vtrr_queue"` | Celery queue name for the dispatcher task |
 | `max_concurrency` | `int` | `4` | Max dequeue workers held in the broker at once |
+| `lease_seconds` | `int` | `3900` | How long a dequeued item stays leased before it is reclaimed. **Must exceed the task's hard `time_limit`.** See [Leasing](#leasing-and-at-least-once-delivery) |
+| `reclaim_interval` | `float \| None` | `None` | Seconds between periodic reclaims run by Celery Beat. `None` disables it. See [Periodic reclaim](#periodic-reclaim-with-celery-beat) |
+| `reclaim_queue` | `str \| None` | `None` | Celery queue the reclaim task is sent to. `None` uses your default queue. See [Periodic reclaim](#periodic-reclaim-with-celery-beat) |
 
 
 ### 2. Register a dedicated Celery worker
@@ -213,8 +217,10 @@ Each task dict may also include an `"id"` key to supply a stable task ID; otherw
 | `retry_kwargs` | Dict of kwargs passed to `self.retry()`, e.g. `{"countdown": 10}` |
 | `soft_time_limit` | Seconds before a `SoftTimeLimitExceeded` is raised in the worker |
 | `time_limit` | Hard kill timeout in seconds |
+| `acks_late` | Ack the Celery trigger after the task finishes. Optional: leasing does not depend on it |
+| `reject_on_worker_lost` | Requeue the trigger if the worker process is killed mid-task. Optional: leasing does not depend on it |
 
-Since each `@vtrr.task` becomes its own Celery task, different task functions can have different retry policies and time limits.
+A `VTRRQueue` runs exactly one task: decorating a second function on the same instance raises `ValueError`. Every Celery trigger on a queue dequeues whatever item is next, so the Celery options (time limits, retries, `base`) must belong to the one task that every item runs. For several task types, create one `VTRRQueue` per task, each with its own `name` and `celery_queue`.
 
 
 ## Django-specific setup
@@ -312,20 +318,119 @@ class UploadView(View):
 ```
 
 
+## Leasing and at-least-once delivery
+
+Dequeuing an item does not delete it; it **leases** it. The item moves to an in-flight set with an expiry of `redis_time + lease_seconds` (Redis `TIME`, so worker clock skew is irrelevant) and its lease token is stored. The `@vtrr.task` wrapper acks the lease when your function returns. If the worker dies first, the lease expires and the next dequeue (or a call to `vtrr.reclaim()`, or the [periodic reclaim](#periodic-reclaim-with-celery-beat)) puts the item back in the queue, **rescored** under VTRR as if freshly enqueued; it does not keep its old position. Each reclaim adds one to the item's **timeouts** count, which is never reset.
+
+Delivery is therefore **at-least-once: task bodies must be idempotent.** Inside a task, `self.request.vtrr_timeouts` is how many times this item's lease has expired and been reclaimed: 0 on the first run, +1 each time it is run again after a timeout. The library does not cap timeouts; your task decides when to give up (see [Timeout limits](#timeout-limits)).
+
+A task that raises an exception is not redelivered: the item is acked and finished, as in Celery's default behavior. Only a time limit or a dead worker causes redelivery (see [Retries and timeouts](#retries-and-timeouts)). If you use the low-level API directly:
+
+```python
+task_name, args, kwargs, task_id, token, timeouts = vtrr._dequeue()
+try:
+    ...
+finally:
+    vtrr.ack(task_id, token)   # False if the lease already expired and was re-granted; harmless
+```
+
+`ack` is fenced: it only succeeds when the token matches the current lease, so a slow worker whose lease was reclaimed cannot ack the new holder's lease.
+
+### Retries and timeouts
+
+| What happens in the task | Result | Counts as a timeout (`vtrr_timeouts`) |
+|---|---|---|
+| Returns normally | Done | No |
+| Raises an exception | Done (failed); not run again | No |
+| Calls `self.retry()`, or raises an exception in `autoretry_for`, with retries left | Run again after the retry delay | No |
+| Raises an `autoretry_for` exception after `max_retries` is used up | Done (failed); the exception is raised | No |
+| Hits `soft_time_limit` (`SoftTimeLimitExceeded`) | Put back in the queue once its lease expires | Yes |
+| Hits the hard `time_limit`, or the worker dies | Put back in the queue once its lease expires | Yes |
+
+- **Retries** (`self.retry()` and `autoretry_for`, with `retry_kwargs`, `retry_backoff`, etc.) behave as in plain Celery, and the item keeps its lease while it waits. `self.request.retries` counts them. The retry delay should stay well under `lease_seconds`: if the lease expires first, the retry is dropped and the item is run again as a timeout instead.
+- **Timeouts** are what `self.request.vtrr_timeouts` counts. A timed-out item is re-enqueued with a fresh position in the queue, and the count is never reset.
+- `SoftTimeLimitExceeded` is always treated as a timeout, even if it is listed in `autoretry_for`. It is never counted by `max_retries`.
+
+### Celery configuration and timing
+
+Recovery does not depend on Celery's delivery guarantees. The Celery message that wakes a worker (the "trigger") carries no payload; the work item lives in Redis under its lease. If a worker dies, the item comes back when its lease expires, and [periodic reclaim](#periodic-reclaim-with-celery-beat) wakes a worker for it. You do not need `acks_late`, `reject_on_worker_lost`, or a raised `visibility_timeout` for this. (They remain available task options. `reject_on_worker_lost` in particular does not speed recovery: the requeued trigger runs while the lease is still held, so it cannot reclaim the item.)
+
+Required order:
+
+```
+soft_time_limit < time_limit (hard) < lease_seconds
+```
+
+For an hour-long task: soft 3600, hard 3720, lease 3900.
+
+```python
+vtrr = VTRRQueue(..., lease_seconds=3900, reclaim_interval=600)
+
+@vtrr.task(soft_time_limit=3600, time_limit=3720)
+def process_file(self, ...):
+    ...
+```
+
+- **Hard `time_limit` is required.** `soft_time_limit` only raises an exception, which blocking C/IO code can delay or swallow. The hard limit kills the process, so by the time a lease expires the original holder is gone and reclaim rarely causes concurrent executions. A task that hits either limit is put back in the queue after `lease_seconds` and counted as a timeout.
+- **`lease_seconds` > hard limit**, for the reason above.
+- A recovered item waits for its lease to expire, so worst-case recovery time is about `lease_seconds + reclaim_interval`.
+- If you do enable `acks_late`, the Redis transport's `visibility_timeout` (default 3600s, broker-wide) applies to it: a trigger held unacked longer is redelivered while still running. The lease token fences the duplicate, but keep `visibility_timeout` above your longest task.
+
+### Periodic reclaim with Celery Beat
+
+Reclaim also runs at the start of every dequeue, but that is not enough on its own: if the queue is idle, or every trigger for waiting items has been lost (a worker died between receiving its trigger and dequeuing), no dequeue happens and the item sits there. `vtrr.reclaim_and_dispatch()` reclaims expired leases and then schedules dequeue workers for whatever is waiting (up to `max_concurrency` minus the triggers already in the broker). It is cheap: when nothing has expired and the queue is empty it is one Redis call that returns immediately.
+
+Pass `reclaim_interval` to run it on Celery Beat:
+
+```python
+vtrr = VTRRQueue(..., reclaim_interval=600, reclaim_queue="celery")
+```
+
+At construction the library registers a task named `vtrr_queue.reclaim.{name}` and merges its entry into `celery_app.conf.beat_schedule`. You still have to run Beat (`celery -A myapp beat`), and the process that executes the task must have imported your `@vtrr.task` functions, which it uses as triggers.
+
+- **`reclaim_queue`**: leave it unset to use your default Celery queue, or name a queue that some worker consumes. Don't send it to `celery_queue` if those workers are usually busy with long tasks; it would wait behind them.
+- **If your app assigns `beat_schedule` wholesale after creating the `VTRRQueue`**, it overwrites the entry. Merge `vtrr.reclaim_beat_schedule()` back in:
+
+  ```python
+  app.conf.beat_schedule = {**my_entries, **vtrr.reclaim_beat_schedule()}
+  ```
+- **Without Beat**, call `vtrr.reclaim_and_dispatch()` from whatever scheduler you have (cron, a management command, etc.).
+
+### Timeout limits
+
+The library never gives up on an item: every expired lease is reclaimed and re-enqueued. To cap redeliveries, check the timeouts count in your task and return early; the wrapper then acks the item and it is dropped.
+
+```python
+MAX_TIMEOUTS = 3
+
+@vtrr.task(soft_time_limit=3600, time_limit=3720)
+def process_file(self, ...):
+    if self.request.vtrr_timeouts >= MAX_TIMEOUTS:
+        log_poison_item(...)   # your own dead-letter handling
+        return
+    ...
+```
+
+
 ## Redis keys
 
-Each `VTRRQueue` instance manages four keys namespaced under `vtrr:{name}:*`.
+Each `VTRRQueue` instance manages these keys, namespaced under `vtrr:{name}:*`.
 
 | Key | Type | Description |
 |---|---|---|
 | `vtrr:{name}:queue` | Sorted Set | Task IDs scored by virtual time; lowest score = highest priority |
 | `vtrr:{name}:current_virtual_time` | String | Virtual time of the last dequeued task; resets to 0 when queue drains |
 | `vtrr:{name}:partition_virtual_time` | Hash | Per-partition virtual time counter; deleted when queue drains |
-| `vtrr:{name}:task` | Hash | `task_id → JSON payload`; deleted when queue drains |
+| `vtrr:{name}:task` | Hash | `task_id → JSON payload` for queued items; deleted when queue drains |
+| `vtrr:{name}:meta` | Hash | `task_id → "weight:partition_key"`, kept until ack so reclaim can rescore |
+| `vtrr:{name}:inflight` | Sorted Set | Leased task IDs scored by lease expiry (Redis time) |
+| `vtrr:{name}:inflight_task` | Hash | `task_id → JSON payload` for leased items |
+| `vtrr:{name}:lease` | Hash | `task_id → lease token` |
+| `vtrr:{name}:timeouts` | Hash | `task_id → times its lease expired and was reclaimed`; cleared on ack |
 
 Multiple `VTRRQueue` instances in the same application can safely share a Redis database as long as they have distinct `name` values.
 
-All enqueue and dequeue operations are atomic Lua scripts, so concurrent writers and workers are safe.
+All enqueue, dequeue, ack and reclaim operations are atomic Lua scripts, so concurrent writers and workers are safe.
 
 # Development
 

{vtrr_queue-0.1.0 → vtrr_queue-0.2.1}/README.md

@@ -10,6 +10,7 @@ Integrates with an existing Celery + Redis setup. The library owns the dequeue l
 - Python >= 3.12
 - Redis (used for both the VTRR queue state and as the Celery broker)
 - Celery >= 5.0
+- Optional but recommended: Celery Beat (or any scheduler that can call `vtrr.reclaim_and_dispatch()`). Queueing and running tasks work without it, but recovering work from a dead worker when the queue is otherwise idle does not. See [Periodic reclaim](#periodic-reclaim-with-celery-beat)
 
 
 ## Installation
@@ -81,6 +82,9 @@ Use a dedicated Redis DB index for the VTRR queue so its keys don't collide with
 | `name` | `str` | required | Unique name for this queue instance; namespaces all Redis keys as `vtrr:{name}:*` |
 | `celery_queue` | `str` | `"vtrr_queue"` | Celery queue name for the dispatcher task |
 | `max_concurrency` | `int` | `4` | Max dequeue workers held in the broker at once |
+| `lease_seconds` | `int` | `3900` | How long a dequeued item stays leased before it is reclaimed. **Must exceed the task's hard `time_limit`.** See [Leasing](#leasing-and-at-least-once-delivery) |
+| `reclaim_interval` | `float \| None` | `None` | Seconds between periodic reclaims run by Celery Beat. `None` disables it. See [Periodic reclaim](#periodic-reclaim-with-celery-beat) |
+| `reclaim_queue` | `str \| None` | `None` | Celery queue the reclaim task is sent to. `None` uses your default queue. See [Periodic reclaim](#periodic-reclaim-with-celery-beat) |
 
 
 ### 2. Register a dedicated Celery worker
@@ -187,8 +191,10 @@ Each task dict may also include an `"id"` key to supply a stable task ID; otherw
 | `retry_kwargs` | Dict of kwargs passed to `self.retry()`, e.g. `{"countdown": 10}` |
 | `soft_time_limit` | Seconds before a `SoftTimeLimitExceeded` is raised in the worker |
 | `time_limit` | Hard kill timeout in seconds |
+| `acks_late` | Ack the Celery trigger after the task finishes. Optional: leasing does not depend on it |
+| `reject_on_worker_lost` | Requeue the trigger if the worker process is killed mid-task. Optional: leasing does not depend on it |
 
-Since each `@vtrr.task` becomes its own Celery task, different task functions can have different retry policies and time limits.
+A `VTRRQueue` runs exactly one task: decorating a second function on the same instance raises `ValueError`. Every Celery trigger on a queue dequeues whatever item is next, so the Celery options (time limits, retries, `base`) must belong to the one task that every item runs. For several task types, create one `VTRRQueue` per task, each with its own `name` and `celery_queue`.
 
 
 ## Django-specific setup
@@ -286,20 +292,119 @@ class UploadView(View):
 ```
 
 
+## Leasing and at-least-once delivery
+
+Dequeuing an item does not delete it; it **leases** it. The item moves to an in-flight set with an expiry of `redis_time + lease_seconds` (Redis `TIME`, so worker clock skew is irrelevant) and its lease token is stored. The `@vtrr.task` wrapper acks the lease when your function returns. If the worker dies first, the lease expires and the next dequeue (or a call to `vtrr.reclaim()`, or the [periodic reclaim](#periodic-reclaim-with-celery-beat)) puts the item back in the queue, **rescored** under VTRR as if freshly enqueued; it does not keep its old position. Each reclaim adds one to the item's **timeouts** count, which is never reset.
+
+Delivery is therefore **at-least-once: task bodies must be idempotent.** Inside a task, `self.request.vtrr_timeouts` is how many times this item's lease has expired and been reclaimed: 0 on the first run, +1 each time it is run again after a timeout. The library does not cap timeouts; your task decides when to give up (see [Timeout limits](#timeout-limits)).
+
+A task that raises an exception is not redelivered: the item is acked and finished, as in Celery's default behavior. Only a time limit or a dead worker causes redelivery (see [Retries and timeouts](#retries-and-timeouts)). If you use the low-level API directly:
+
+```python
+task_name, args, kwargs, task_id, token, timeouts = vtrr._dequeue()
+try:
+    ...
+finally:
+    vtrr.ack(task_id, token)   # False if the lease already expired and was re-granted; harmless
+```
+
+`ack` is fenced: it only succeeds when the token matches the current lease, so a slow worker whose lease was reclaimed cannot ack the new holder's lease.
+
+### Retries and timeouts
+
+| What happens in the task | Result | Counts as a timeout (`vtrr_timeouts`) |
+|---|---|---|
+| Returns normally | Done | No |
+| Raises an exception | Done (failed); not run again | No |
+| Calls `self.retry()`, or raises an exception in `autoretry_for`, with retries left | Run again after the retry delay | No |
+| Raises an `autoretry_for` exception after `max_retries` is used up | Done (failed); the exception is raised | No |
+| Hits `soft_time_limit` (`SoftTimeLimitExceeded`) | Put back in the queue once its lease expires | Yes |
+| Hits the hard `time_limit`, or the worker dies | Put back in the queue once its lease expires | Yes |
+
+- **Retries** (`self.retry()` and `autoretry_for`, with `retry_kwargs`, `retry_backoff`, etc.) behave as in plain Celery, and the item keeps its lease while it waits. `self.request.retries` counts them. The retry delay should stay well under `lease_seconds`: if the lease expires first, the retry is dropped and the item is run again as a timeout instead.
+- **Timeouts** are what `self.request.vtrr_timeouts` counts. A timed-out item is re-enqueued with a fresh position in the queue, and the count is never reset.
+- `SoftTimeLimitExceeded` is always treated as a timeout, even if it is listed in `autoretry_for`. It is never counted by `max_retries`.
+
+### Celery configuration and timing
+
+Recovery does not depend on Celery's delivery guarantees. The Celery message that wakes a worker (the "trigger") carries no payload; the work item lives in Redis under its lease. If a worker dies, the item comes back when its lease expires, and [periodic reclaim](#periodic-reclaim-with-celery-beat) wakes a worker for it. You do not need `acks_late`, `reject_on_worker_lost`, or a raised `visibility_timeout` for this. (They remain available task options. `reject_on_worker_lost` in particular does not speed recovery: the requeued trigger runs while the lease is still held, so it cannot reclaim the item.)
+
+Required order:
+
+```
+soft_time_limit < time_limit (hard) < lease_seconds
+```
+
+For an hour-long task: soft 3600, hard 3720, lease 3900.
+
+```python
+vtrr = VTRRQueue(..., lease_seconds=3900, reclaim_interval=600)
+
+@vtrr.task(soft_time_limit=3600, time_limit=3720)
+def process_file(self, ...):
+    ...
+```
+
+- **Hard `time_limit` is required.** `soft_time_limit` only raises an exception, which blocking C/IO code can delay or swallow. The hard limit kills the process, so by the time a lease expires the original holder is gone and reclaim rarely causes concurrent executions. A task that hits either limit is put back in the queue after `lease_seconds` and counted as a timeout.
+- **`lease_seconds` > hard limit**, for the reason above.
+- A recovered item waits for its lease to expire, so worst-case recovery time is about `lease_seconds + reclaim_interval`.
+- If you do enable `acks_late`, the Redis transport's `visibility_timeout` (default 3600s, broker-wide) applies to it: a trigger held unacked longer is redelivered while still running. The lease token fences the duplicate, but keep `visibility_timeout` above your longest task.
+
+### Periodic reclaim with Celery Beat
+
+Reclaim also runs at the start of every dequeue, but that is not enough on its own: if the queue is idle, or every trigger for waiting items has been lost (a worker died between receiving its trigger and dequeuing), no dequeue happens and the item sits there. `vtrr.reclaim_and_dispatch()` reclaims expired leases and then schedules dequeue workers for whatever is waiting (up to `max_concurrency` minus the triggers already in the broker). It is cheap: when nothing has expired and the queue is empty it is one Redis call that returns immediately.
+
+Pass `reclaim_interval` to run it on Celery Beat:
+
+```python
+vtrr = VTRRQueue(..., reclaim_interval=600, reclaim_queue="celery")
+```
+
+At construction the library registers a task named `vtrr_queue.reclaim.{name}` and merges its entry into `celery_app.conf.beat_schedule`. You still have to run Beat (`celery -A myapp beat`), and the process that executes the task must have imported your `@vtrr.task` functions, which it uses as triggers.
+
+- **`reclaim_queue`**: leave it unset to use your default Celery queue, or name a queue that some worker consumes. Don't send it to `celery_queue` if those workers are usually busy with long tasks; it would wait behind them.
+- **If your app assigns `beat_schedule` wholesale after creating the `VTRRQueue`**, it overwrites the entry. Merge `vtrr.reclaim_beat_schedule()` back in:
+
+  ```python
+  app.conf.beat_schedule = {**my_entries, **vtrr.reclaim_beat_schedule()}
+  ```
+- **Without Beat**, call `vtrr.reclaim_and_dispatch()` from whatever scheduler you have (cron, a management command, etc.).
+
+### Timeout limits
+
+The library never gives up on an item: every expired lease is reclaimed and re-enqueued. To cap redeliveries, check the timeouts count in your task and return early; the wrapper then acks the item and it is dropped.
+
+```python
+MAX_TIMEOUTS = 3
+
+@vtrr.task(soft_time_limit=3600, time_limit=3720)
+def process_file(self, ...):
+    if self.request.vtrr_timeouts >= MAX_TIMEOUTS:
+        log_poison_item(...)   # your own dead-letter handling
+        return
+    ...
+```
+
+
 ## Redis keys
 
-Each `VTRRQueue` instance manages four keys namespaced under `vtrr:{name}:*`.
+Each `VTRRQueue` instance manages these keys, namespaced under `vtrr:{name}:*`.
 
 | Key | Type | Description |
 |---|---|---|
 | `vtrr:{name}:queue` | Sorted Set | Task IDs scored by virtual time; lowest score = highest priority |
 | `vtrr:{name}:current_virtual_time` | String | Virtual time of the last dequeued task; resets to 0 when queue drains |
 | `vtrr:{name}:partition_virtual_time` | Hash | Per-partition virtual time counter; deleted when queue drains |
-| `vtrr:{name}:task` | Hash | `task_id → JSON payload`; deleted when queue drains |
+| `vtrr:{name}:task` | Hash | `task_id → JSON payload` for queued items; deleted when queue drains |
+| `vtrr:{name}:meta` | Hash | `task_id → "weight:partition_key"`, kept until ack so reclaim can rescore |
+| `vtrr:{name}:inflight` | Sorted Set | Leased task IDs scored by lease expiry (Redis time) |
+| `vtrr:{name}:inflight_task` | Hash | `task_id → JSON payload` for leased items |
+| `vtrr:{name}:lease` | Hash | `task_id → lease token` |
+| `vtrr:{name}:timeouts` | Hash | `task_id → times its lease expired and was reclaimed`; cleared on ack |
 
 Multiple `VTRRQueue` instances in the same application can safely share a Redis database as long as they have distinct `name` values.
 
-All enqueue and dequeue operations are atomic Lua scripts, so concurrent writers and workers are safe.
+All enqueue, dequeue, ack and reclaim operations are atomic Lua scripts, so concurrent writers and workers are safe.
 
 # Development
 

{vtrr_queue-0.1.0 → vtrr_queue-0.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "vtrr-queue"
-version = "0.1.0"
+version = "0.2.1"
 description = "Virtual-time round-robin fair queue on Redis"
 readme = "README.md"
 authors = [{ name = "Nicolas Johnson", email = "nicolas.johnson@c4ads.org" }]