vtrr-queue 0.2.0__tar.gz → 0.2.1__tar.gz

{vtrr_queue-0.2.0 → vtrr_queue-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: vtrr-queue
-Version: 0.2.0
+Version: 0.2.1
 Summary: Virtual-time round-robin fair queue on Redis
 Author-email: Nicolas Johnson <nicolas.johnson@c4ads.org>
 License-Expression: Apache-2.0
@@ -324,7 +324,7 @@ Dequeuing an item does not delete it; it **leases** it. The item moves to an in-
 
 Delivery is therefore **at-least-once: task bodies must be idempotent.** Inside a task, `self.request.vtrr_timeouts` is how many times this item's lease has expired and been reclaimed: 0 on the first run, +1 each time it is run again after a timeout. The library does not cap timeouts; your task decides when to give up (see [Timeout limits](#timeout-limits)).
 
-The wrapper acks in a `finally`, so a Python exception does not redeliver forever; only process death or lease expiry does. This matches Celery's default `task_acks_on_failure_or_timeout=True`. A non-`Exception` such as `SystemExit` is treated as process death and is *not* acked: billiard raises `SystemExit` in the task when the hard time limit sends SIGTERM (SIGKILL follows ~0.1s later), so a task killed by its hard `time_limit` keeps its lease, is reclaimed, and counts as a timeout. If you use the low-level API directly:
+A task that raises an exception is not redelivered: the item is acked and finished, as in Celery's default behavior. Only a time limit or a dead worker causes redelivery (see [Retries and timeouts](#retries-and-timeouts)). If you use the low-level API directly:
 
 ```python
 task_name, args, kwargs, task_id, token, timeouts = vtrr._dequeue()
@@ -334,10 +334,23 @@ finally:
     vtrr.ack(task_id, token)   # False if the lease already expired and was re-granted; harmless
 ```
 
-`self.retry()` / `autoretry_for` keep the lease: the wrapper does not ack when a task retries, and the retry run re-leases the item (expiry reset to `redis_time + lease_seconds`) and acks when it finishes. If the lease was already reclaimed during the retry delay (for example a `countdown` longer than `lease_seconds`), the item belongs to a newer delivery and the retry run is dropped instead of executing twice.
-
 `ack` is fenced: it only succeeds when the token matches the current lease, so a slow worker whose lease was reclaimed cannot ack the new holder's lease.
 
+### Retries and timeouts
+
+| What happens in the task | Result | Counts as a timeout (`vtrr_timeouts`) |
+|---|---|---|
+| Returns normally | Done | No |
+| Raises an exception | Done (failed); not run again | No |
+| Calls `self.retry()`, or raises an exception in `autoretry_for`, with retries left | Run again after the retry delay | No |
+| Raises an `autoretry_for` exception after `max_retries` is used up | Done (failed); the exception is raised | No |
+| Hits `soft_time_limit` (`SoftTimeLimitExceeded`) | Put back in the queue once its lease expires | Yes |
+| Hits the hard `time_limit`, or the worker dies | Put back in the queue once its lease expires | Yes |
+
+- **Retries** (`self.retry()` and `autoretry_for`, with `retry_kwargs`, `retry_backoff`, etc.) behave as in plain Celery, and the item keeps its lease while it waits. `self.request.retries` counts them. The retry delay should stay well under `lease_seconds`: if the lease expires first, the retry is dropped and the item is run again as a timeout instead.
+- **Timeouts** are what `self.request.vtrr_timeouts` counts. A timed-out item is re-enqueued with a fresh position in the queue, and the count is never reset.
+- `SoftTimeLimitExceeded` is always treated as a timeout, even if it is listed in `autoretry_for`. It is never counted by `max_retries`.
+
 ### Celery configuration and timing
 
 Recovery does not depend on Celery's delivery guarantees. The Celery message that wakes a worker (the "trigger") carries no payload; the work item lives in Redis under its lease. If a worker dies, the item comes back when its lease expires, and [periodic reclaim](#periodic-reclaim-with-celery-beat) wakes a worker for it. You do not need `acks_late`, `reject_on_worker_lost`, or a raised `visibility_timeout` for this. (They remain available task options. `reject_on_worker_lost` in particular does not speed recovery: the requeued trigger runs while the lease is still held, so it cannot reclaim the item.)
@@ -358,7 +371,7 @@ def process_file(self, ...):
     ...
 ```
 
-- **Hard `time_limit` is required.** `soft_time_limit` only raises an exception, which blocking C/IO code can delay or swallow. The hard limit kills the process, so by the time a lease expires the original holder is gone and reclaim rarely causes concurrent executions. Hitting the hard limit is itself recovered the same way: the lease is not acked, so the item is reclaimed after `lease_seconds` and its `timeouts` count goes up.
+- **Hard `time_limit` is required.** `soft_time_limit` only raises an exception, which blocking C/IO code can delay or swallow. The hard limit kills the process, so by the time a lease expires the original holder is gone and reclaim rarely causes concurrent executions. A task that hits either limit is put back in the queue after `lease_seconds` and counted as a timeout.
 - **`lease_seconds` > hard limit**, for the reason above.
 - A recovered item waits for its lease to expire, so worst-case recovery time is about `lease_seconds + reclaim_interval`.
 - If you do enable `acks_late`, the Redis transport's `visibility_timeout` (default 3600s, broker-wide) applies to it: a trigger held unacked longer is redelivered while still running. The lease token fences the duplicate, but keep `visibility_timeout` above your longest task.

{vtrr_queue-0.2.0 → vtrr_queue-0.2.1}/README.md

@@ -298,7 +298,7 @@ Dequeuing an item does not delete it; it **leases** it. The item moves to an in-
 
 Delivery is therefore **at-least-once: task bodies must be idempotent.** Inside a task, `self.request.vtrr_timeouts` is how many times this item's lease has expired and been reclaimed: 0 on the first run, +1 each time it is run again after a timeout. The library does not cap timeouts; your task decides when to give up (see [Timeout limits](#timeout-limits)).
 
-The wrapper acks in a `finally`, so a Python exception does not redeliver forever; only process death or lease expiry does. This matches Celery's default `task_acks_on_failure_or_timeout=True`. A non-`Exception` such as `SystemExit` is treated as process death and is *not* acked: billiard raises `SystemExit` in the task when the hard time limit sends SIGTERM (SIGKILL follows ~0.1s later), so a task killed by its hard `time_limit` keeps its lease, is reclaimed, and counts as a timeout. If you use the low-level API directly:
+A task that raises an exception is not redelivered: the item is acked and finished, as in Celery's default behavior. Only a time limit or a dead worker causes redelivery (see [Retries and timeouts](#retries-and-timeouts)). If you use the low-level API directly:
 
 ```python
 task_name, args, kwargs, task_id, token, timeouts = vtrr._dequeue()
@@ -308,10 +308,23 @@ finally:
     vtrr.ack(task_id, token)   # False if the lease already expired and was re-granted; harmless
 ```
 
-`self.retry()` / `autoretry_for` keep the lease: the wrapper does not ack when a task retries, and the retry run re-leases the item (expiry reset to `redis_time + lease_seconds`) and acks when it finishes. If the lease was already reclaimed during the retry delay (for example a `countdown` longer than `lease_seconds`), the item belongs to a newer delivery and the retry run is dropped instead of executing twice.
-
 `ack` is fenced: it only succeeds when the token matches the current lease, so a slow worker whose lease was reclaimed cannot ack the new holder's lease.
 
+### Retries and timeouts
+
+| What happens in the task | Result | Counts as a timeout (`vtrr_timeouts`) |
+|---|---|---|
+| Returns normally | Done | No |
+| Raises an exception | Done (failed); not run again | No |
+| Calls `self.retry()`, or raises an exception in `autoretry_for`, with retries left | Run again after the retry delay | No |
+| Raises an `autoretry_for` exception after `max_retries` is used up | Done (failed); the exception is raised | No |
+| Hits `soft_time_limit` (`SoftTimeLimitExceeded`) | Put back in the queue once its lease expires | Yes |
+| Hits the hard `time_limit`, or the worker dies | Put back in the queue once its lease expires | Yes |
+
+- **Retries** (`self.retry()` and `autoretry_for`, with `retry_kwargs`, `retry_backoff`, etc.) behave as in plain Celery, and the item keeps its lease while it waits. `self.request.retries` counts them. The retry delay should stay well under `lease_seconds`: if the lease expires first, the retry is dropped and the item is run again as a timeout instead.
+- **Timeouts** are what `self.request.vtrr_timeouts` counts. A timed-out item is re-enqueued with a fresh position in the queue, and the count is never reset.
+- `SoftTimeLimitExceeded` is always treated as a timeout, even if it is listed in `autoretry_for`. It is never counted by `max_retries`.
+
 ### Celery configuration and timing
 
 Recovery does not depend on Celery's delivery guarantees. The Celery message that wakes a worker (the "trigger") carries no payload; the work item lives in Redis under its lease. If a worker dies, the item comes back when its lease expires, and [periodic reclaim](#periodic-reclaim-with-celery-beat) wakes a worker for it. You do not need `acks_late`, `reject_on_worker_lost`, or a raised `visibility_timeout` for this. (They remain available task options. `reject_on_worker_lost` in particular does not speed recovery: the requeued trigger runs while the lease is still held, so it cannot reclaim the item.)
@@ -332,7 +345,7 @@ def process_file(self, ...):
     ...
 ```
 
-- **Hard `time_limit` is required.** `soft_time_limit` only raises an exception, which blocking C/IO code can delay or swallow. The hard limit kills the process, so by the time a lease expires the original holder is gone and reclaim rarely causes concurrent executions. Hitting the hard limit is itself recovered the same way: the lease is not acked, so the item is reclaimed after `lease_seconds` and its `timeouts` count goes up.
+- **Hard `time_limit` is required.** `soft_time_limit` only raises an exception, which blocking C/IO code can delay or swallow. The hard limit kills the process, so by the time a lease expires the original holder is gone and reclaim rarely causes concurrent executions. A task that hits either limit is put back in the queue after `lease_seconds` and counted as a timeout.
 - **`lease_seconds` > hard limit**, for the reason above.
 - A recovered item waits for its lease to expire, so worst-case recovery time is about `lease_seconds + reclaim_interval`.
 - If you do enable `acks_late`, the Redis transport's `visibility_timeout` (default 3600s, broker-wide) applies to it: a trigger held unacked longer is redelivered while still running. The lease token fences the duplicate, but keep `visibility_timeout` above your longest task.

{vtrr_queue-0.2.0 → vtrr_queue-0.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "vtrr-queue"
-version = "0.2.0"
+version = "0.2.1"
 description = "Virtual-time round-robin fair queue on Redis"
 readme = "README.md"
 authors = [{ name = "Nicolas Johnson", email = "nicolas.johnson@c4ads.org" }]

{vtrr_queue-0.2.0 → vtrr_queue-0.2.1}/src/vtrr_queue/queue.py

@@ -4,7 +4,8 @@ import uuid
 from pathlib import Path
 from typing import Any, Callable
 from celery import Task
-from celery.exceptions import Retry
+from celery.exceptions import Retry, SoftTimeLimitExceeded
+from celery.utils.time import get_exponential_backoff_interval
 
 
 import redis as redis_lib
@@ -219,6 +220,13 @@ class VTRRQueue:
             options = dict(task_options)
             options.pop("bind", None)
             base = _resolve_base(options.pop("base", None))
+            # Autoretry is re-implemented by celery_wrapper instead of using Celery's autoretry_for wrapper
+            # This is to support retries along with vtrr_queue timeouts/message leasing
+            autoretry_for = tuple(options.pop("autoretry_for", ()))
+            retry_backoff = float(options.pop("retry_backoff", False))
+            retry_backoff_max = int(options.pop("retry_backoff_max", 600))
+            retry_jitter = options.pop("retry_jitter", True)
+            retry_kwargs = options.pop("retry_kwargs", {})
             task_name = f"{f.__module__}.{f.__name__}"
 
             @self._celery_app.task(name=task_name, bind=True, base=base, **options)
@@ -281,11 +289,35 @@ class VTRRQueue:
                             registered.name if registered else None,
                         )
                         return
-                    registered._fn(celery_task, *args, **kwargs)
+                    try:
+                        registered._fn(celery_task, *args, **kwargs)
+                    except Retry:
+                        # This exception is only raised by celery if self.retry()
+                        # was already called by the inner function, so assume the retry task is already scheduled.
+                        raise
+                    except SoftTimeLimitExceeded:
+                        # Treated as a vtrr timeout. Re-raise so terminated=True and message goes through the lease lifecycle.
+                        raise
+                    except autoretry_for as exc:
+                        # Raises Retry (lease kept), or re-raises exc once retries are exhausted.
+                        auto_retry_kwargs = dict(retry_kwargs)
+                        if retry_backoff:
+                            auto_retry_kwargs["countdown"] = (
+                                get_exponential_backoff_interval(
+                                    factor=int(max(1.0, retry_backoff)),
+                                    retries=celery_task.request.retries,
+                                    maximum=retry_backoff_max,
+                                    full_jitter=retry_jitter,
+                                )
+                            )
+                        celery_task.retry(exc=exc, **auto_retry_kwargs)
                 except Retry:
                     # Keep the lease: the retry run re-leases it (renew) and acks when done.
                     retrying = True
                     raise
+                except SoftTimeLimitExceeded:
+                    terminated = True
+                    raise
                 except BaseException as err:
                     # A non-Exception (SystemExit, KeyboardInterrupt, ...) means the process is being
                     # terminated, e.g. billiard's SIGTERM at the hard time limit (it raises SystemExit

{vtrr_queue-0.2.0 → vtrr_queue-0.2.1}/tests/test_vtrr.py

@@ -5,6 +5,7 @@ from unittest.mock import MagicMock
 
 import fakeredis
 import pytest
+from celery.exceptions import SoftTimeLimitExceeded
 
 import vtrr_queue.queue as vtrr_module
 from vtrr_queue.queue import VTRRQueue
@@ -578,6 +579,23 @@ class TestLeases:
         assert vtrr.reclaim() == ["t1"]
         assert vtrr._redis.hget(vtrr._timeouts_key, "t1") == b"1"
 
+    def test_wrapper_does_not_ack_on_soft_time_limit(self, vtrr, monkeypatch):
+        """SoftTimeLimitExceeded is treated like the hard limit: the lease is kept, so it
+        expires, the item is reclaimed, and the reclaim counts as a timeout."""
+
+        @vtrr.task
+        def work(self):
+            raise SoftTimeLimitExceeded
+
+        monkeypatch.setattr(vtrr, "_schedule_workers", lambda *a, **k: None)
+        work.queue(partition_key="A", tasks=[{"id": "t1"}])
+        with pytest.raises(SoftTimeLimitExceeded):
+            kick(work)
+        assert vtrr._redis.zscore(vtrr._inflight_key, "t1") is not None  # still leased
+        expire(vtrr, "t1")
+        assert vtrr.reclaim() == ["t1"]
+        assert vtrr._redis.hget(vtrr._timeouts_key, "t1") == b"1"
+
     def test_timeouts_is_visible_to_task(self, vtrr, monkeypatch):
         """A task run from a first dequeue sees self.request.vtrr_timeouts == 0."""
         seen = []
@@ -632,6 +650,74 @@ class TestRetryReleasesLease:
         assert in_flight == [1, 1]  # leased during the original run and the retry
         assert vtrr._redis.zcard(vtrr._inflight_key) == 0  # acked once, at the end
 
+    def test_autoretry_for_holds_lease_and_acks_at_end(self, vtrr, monkeypatch):
+        """An exception in autoretry_for keeps the lease through the retry, like self.retry()."""
+        in_flight = []
+
+        @vtrr.task(
+            autoretry_for=(ConnectionRefusedError,),
+            max_retries=2,
+            retry_kwargs={"countdown": 0},
+        )
+        def work(self):
+            in_flight.append(vtrr._redis.zcard(vtrr._inflight_key))
+            if len(in_flight) == 1:
+                raise ConnectionRefusedError
+
+        monkeypatch.setattr(vtrr, "_schedule_workers", lambda *a, **k: None)
+        work.queue(partition_key="A", tasks=[{"id": "t1"}])
+        work._celery_task.apply(kwargs={"is_start": True}, throw=False)
+        # Work able to run twice and so it was able to re-acquire the lease after the exception.
+        assert in_flight == [1, 1]
+        assert vtrr._redis.zcard(vtrr._inflight_key) == 0
+
+    def test_autoretry_for_acks_when_retries_exhausted(self, vtrr, monkeypatch):
+        """Once max_retries is used up the original exception propagates and the item is acked."""
+        runs = []
+
+        @vtrr.task(
+            autoretry_for=(ConnectionRefusedError,),
+            max_retries=1,
+            retry_kwargs={"countdown": 0},
+        )
+        def work(self):
+            runs.append(1)
+            raise ConnectionRefusedError
+
+        monkeypatch.setattr(vtrr, "_schedule_workers", lambda *a, **k: None)
+        work.queue(partition_key="A", tasks=[{"id": "t1"}])
+        # eager mode runs the retry inline, so the final exception surfaces through apply()
+        with pytest.raises(ConnectionRefusedError):
+            work._celery_task.apply(kwargs={"is_start": True}, throw=False)
+        assert len(runs) == 2  # original + 1 retry
+        assert vtrr._redis.zcard(vtrr._inflight_key) == 0
+        assert vtrr._redis.hget(vtrr._timeouts_key, "t1") is None
+
+    def test_soft_time_limit_in_autoretry_for_goes_through_lease_expiry(
+        self, vtrr, monkeypatch
+    ):
+        """SoftTimeLimitExceeded listed in autoretry_for is not Celery-retried: the lease is
+        kept until it expires, then the reclaim counts it as a timeout."""
+        runs = []
+
+        @vtrr.task(
+            autoretry_for=(SoftTimeLimitExceeded,),
+            max_retries=3,
+            retry_kwargs={"countdown": 0},
+        )
+        def work(self):
+            runs.append(1)
+            raise SoftTimeLimitExceeded
+
+        monkeypatch.setattr(vtrr, "_schedule_workers", lambda *a, **k: None)
+        work.queue(partition_key="A", tasks=[{"id": "t1"}])
+        work._celery_task.apply(kwargs={"is_start": True}, throw=False)
+        assert len(runs) == 1
+        assert vtrr._redis.zscore(vtrr._inflight_key, "t1") is not None
+        expire(vtrr, "t1")
+        assert vtrr.reclaim() == ["t1"]
+        assert vtrr._redis.hget(vtrr._timeouts_key, "t1") == b"1"
+
     def test_retry_dropped_when_lease_was_lost(self, vtrr):
         """A retry run whose token no longer matches any lease (reclaimed, re-granted or
         acked) is dropped without."""