redis-message-queue 8.2.7__tar.gz → 8.2.9__tar.gz

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 8.2.7
+Version: 8.2.9
 Summary: Python message queuing with Redis and message deduplication
 Project-URL: Homepage, https://github.com/Elijas/redis-message-queue
 Project-URL: Repository, https://github.com/Elijas/redis-message-queue
@@ -34,7 +34,7 @@ Description-Content-Type: text/markdown
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=8.2.7,<9.0.0"
+pip install "redis-message-queue>=8.2.9,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
@@ -129,9 +129,12 @@ All features are optional and can be enabled or disabled as needed.
 | Configuration | Delivery guarantee |
 |---|---|
 | Default (`visibility_timeout_seconds=300`) | **At-least-once** — expired messages are reclaimed and redelivered |
-| With `visibility_timeout_seconds=None` | **At-most-once** — a consumer crash loses the in-flight message |
+| With `visibility_timeout_seconds=None, max_delivery_count=None` | **At-most-once** — a consumer crash loses the in-flight message |
 
 See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-timeout) for details and tradeoffs.
+Because delivery-count limits depend on visibility-timeout reclaim, disabling
+lease-based crash recovery requires setting both `visibility_timeout_seconds=None`
+and `max_delivery_count=None`.
 
 > **Important:** Handler exceptions are terminal. This library is a payload
 > queue, not a task framework: raising inside `process_message()` does not
@@ -290,7 +293,10 @@ queue = RedisMessageQueue(
 
 The callback is **advisory** — it may fire briefly after a successful `process_message` exit when a final renewal coincided with the success path. Use it for metrics or alerting, not as a correctness signal. For the async queue (`redis_message_queue.asyncio`), the callback may also be `async def`.
 
-Without a visibility timeout, messages already moved to `processing` remain there indefinitely after a consumer crash and are not redelivered, even if the crash happened before your handler started running.
+With `visibility_timeout_seconds=None, max_delivery_count=None`, messages
+already moved to `processing` remain there indefinitely after a consumer crash
+and are not redelivered, even if the crash happened before your handler
+started running.
 
 Visibility deadlines use Redis server time (`TIME`), not Python process time.
 A forward step in the Redis server clock can make a live lease appear expired
@@ -473,10 +479,10 @@ without `aclose()`, or sync processes killed mid-handler, can leave the message
 and its processing/lease metadata in Redis until a later consumer claim path
 triggers visibility-timeout reclaim. With visibility timeouts enabled, this is
 the designed at-least-once recovery path: the message is delayed by the lease,
-not lost. With `visibility_timeout_seconds=None`, there is no automatic reclaim
-path. For low-visibility-timeout workloads, prefer an explicit `drain()` /
-`aclose()` during shutdown so local pending claim IDs are recovered before
-process exit.
+not lost. With `visibility_timeout_seconds=None, max_delivery_count=None`,
+there is no automatic reclaim path. For low-visibility-timeout workloads,
+prefer an explicit `drain()` / `aclose()` during shutdown so local pending claim
+IDs are recovered before process exit.
 
 `drain()` / `aclose()` timeouts are measured with Python monotonic clocks, but
 any lease deadlines they recover were created from Redis server time. The same
@@ -842,6 +848,11 @@ Pre-commit and mid-flight exceptions:
   exceptions. Under an ambiguous lost response, Redis may have committed
   despite the exception. Treat them as "operation did not succeed from the
   caller's perspective", not "Redis did not commit".
+- Visibility-timeout claim-store write failures raise
+  `ClaimStoreFailedError` and emit `claim/failure`. When the compensating
+  return-to-pending write succeeds, the payload is back in pending; if
+  return-to-pending also fails, the payload remains in processing so there is
+  still a live queue copy.
 
 #### Drain events
 
@@ -869,11 +880,6 @@ The following operations have no `on_event` surface by design:
   preserves queue safety on Cluster `CROSSSLOT` rejection but cannot be
   observed through `on_event`. Operators watching key-TTL behavior or Redis
   slow logs can detect orphans.
-- **VT claim-store OOM compensation:** if the visibility-timeout Lua script
-  cannot store the claim result, it removes the message from processing, pushes
-  it back to pending, and returns `false`. Python translates that into
-  `claim_empty/skipped`, the same shape as an empty poll. This is intentional
-  fail-safe behavior; the message is not lost.
 - **`drop_oldest` evictions:** when publish backpressure uses
   `pending_overload_policy="drop_oldest"`, the oldest pending message is
   discarded before the new message is enqueued. The successful enqueue emits
@@ -884,14 +890,22 @@ The following operations have no `on_event` surface by design:
   terminal operation's failure event. There is no per-attempt event for those
   paths.
 
-The public exception hierarchy is rooted at `RedisMessageQueueError`.
-Configuration value/combinations raise `ConfigurationError` (also a
-`ValueError`), custom gateway contract violations raise `GatewayContractError`
-(also a `TypeError`), and Lua `redis.error_reply(...)` failures raise
-`LuaScriptError` (also a redis-py `ResponseError`). Publish overload raises
-`QueueBackpressureError`; publish after explicit drain raises
-`QueueDrainedError`. `CleanupFailedError` and `RetryBudgetExhaustedError` are
-reserved categories for cleanup and retry surfaces.
+The public exception hierarchy is rooted at `RedisMessageQueueError`. The
+current exported queue-owned exception classes are:
+
+- `RedisMessageQueueError` (base)
+  - `ClaimStoreFailedError` - visibility-timeout claim metadata could not be stored
+  - `ConfigurationError` - invalid constructor args; also a `ValueError`
+  - `DrainFailedError` - drain pending-claim recovery failed
+  - `GatewayContractError` - custom gateway protocol violation; also a `TypeError`
+  - `LuaScriptError` - Lua `redis.error_reply(...)`; also a redis-py `ResponseError`
+  - `MalformedStoredMessageError` - stored value is not a valid RMQ envelope
+  - `PayloadTooLargeError` - serialized payload exceeds `max_payload_bytes`; also a `ValueError`
+  - `PayloadTooDeepError` - payload nesting exceeds `max_payload_depth`; also a `ValueError`
+  - `QueueBackpressureError` - `pending_overload_policy="raise"` rejected enqueue
+  - `QueueDrainedError` - `publish()` called after explicit drain/aclose
+  - `CleanupFailedError` - cleanup after handler completion failed
+  - `RetryBudgetExhaustedError` - Redis retry budget exhausted; also a redis-py `RedisError`
 
 ## Known limitations
 

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/README.md

@@ -11,7 +11,7 @@
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=8.2.7,<9.0.0"
+pip install "redis-message-queue>=8.2.9,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
@@ -106,9 +106,12 @@ All features are optional and can be enabled or disabled as needed.
 | Configuration | Delivery guarantee |
 |---|---|
 | Default (`visibility_timeout_seconds=300`) | **At-least-once** — expired messages are reclaimed and redelivered |
-| With `visibility_timeout_seconds=None` | **At-most-once** — a consumer crash loses the in-flight message |
+| With `visibility_timeout_seconds=None, max_delivery_count=None` | **At-most-once** — a consumer crash loses the in-flight message |
 
 See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-timeout) for details and tradeoffs.
+Because delivery-count limits depend on visibility-timeout reclaim, disabling
+lease-based crash recovery requires setting both `visibility_timeout_seconds=None`
+and `max_delivery_count=None`.
 
 > **Important:** Handler exceptions are terminal. This library is a payload
 > queue, not a task framework: raising inside `process_message()` does not
@@ -267,7 +270,10 @@ queue = RedisMessageQueue(
 
 The callback is **advisory** — it may fire briefly after a successful `process_message` exit when a final renewal coincided with the success path. Use it for metrics or alerting, not as a correctness signal. For the async queue (`redis_message_queue.asyncio`), the callback may also be `async def`.
 
-Without a visibility timeout, messages already moved to `processing` remain there indefinitely after a consumer crash and are not redelivered, even if the crash happened before your handler started running.
+With `visibility_timeout_seconds=None, max_delivery_count=None`, messages
+already moved to `processing` remain there indefinitely after a consumer crash
+and are not redelivered, even if the crash happened before your handler
+started running.
 
 Visibility deadlines use Redis server time (`TIME`), not Python process time.
 A forward step in the Redis server clock can make a live lease appear expired
@@ -450,10 +456,10 @@ without `aclose()`, or sync processes killed mid-handler, can leave the message
 and its processing/lease metadata in Redis until a later consumer claim path
 triggers visibility-timeout reclaim. With visibility timeouts enabled, this is
 the designed at-least-once recovery path: the message is delayed by the lease,
-not lost. With `visibility_timeout_seconds=None`, there is no automatic reclaim
-path. For low-visibility-timeout workloads, prefer an explicit `drain()` /
-`aclose()` during shutdown so local pending claim IDs are recovered before
-process exit.
+not lost. With `visibility_timeout_seconds=None, max_delivery_count=None`,
+there is no automatic reclaim path. For low-visibility-timeout workloads,
+prefer an explicit `drain()` / `aclose()` during shutdown so local pending claim
+IDs are recovered before process exit.
 
 `drain()` / `aclose()` timeouts are measured with Python monotonic clocks, but
 any lease deadlines they recover were created from Redis server time. The same
@@ -819,6 +825,11 @@ Pre-commit and mid-flight exceptions:
   exceptions. Under an ambiguous lost response, Redis may have committed
   despite the exception. Treat them as "operation did not succeed from the
   caller's perspective", not "Redis did not commit".
+- Visibility-timeout claim-store write failures raise
+  `ClaimStoreFailedError` and emit `claim/failure`. When the compensating
+  return-to-pending write succeeds, the payload is back in pending; if
+  return-to-pending also fails, the payload remains in processing so there is
+  still a live queue copy.
 
 #### Drain events
 
@@ -846,11 +857,6 @@ The following operations have no `on_event` surface by design:
   preserves queue safety on Cluster `CROSSSLOT` rejection but cannot be
   observed through `on_event`. Operators watching key-TTL behavior or Redis
   slow logs can detect orphans.
-- **VT claim-store OOM compensation:** if the visibility-timeout Lua script
-  cannot store the claim result, it removes the message from processing, pushes
-  it back to pending, and returns `false`. Python translates that into
-  `claim_empty/skipped`, the same shape as an empty poll. This is intentional
-  fail-safe behavior; the message is not lost.
 - **`drop_oldest` evictions:** when publish backpressure uses
   `pending_overload_policy="drop_oldest"`, the oldest pending message is
   discarded before the new message is enqueued. The successful enqueue emits
@@ -861,14 +867,22 @@ The following operations have no `on_event` surface by design:
   terminal operation's failure event. There is no per-attempt event for those
   paths.
 
-The public exception hierarchy is rooted at `RedisMessageQueueError`.
-Configuration value/combinations raise `ConfigurationError` (also a
-`ValueError`), custom gateway contract violations raise `GatewayContractError`
-(also a `TypeError`), and Lua `redis.error_reply(...)` failures raise
-`LuaScriptError` (also a redis-py `ResponseError`). Publish overload raises
-`QueueBackpressureError`; publish after explicit drain raises
-`QueueDrainedError`. `CleanupFailedError` and `RetryBudgetExhaustedError` are
-reserved categories for cleanup and retry surfaces.
+The public exception hierarchy is rooted at `RedisMessageQueueError`. The
+current exported queue-owned exception classes are:
+
+- `RedisMessageQueueError` (base)
+  - `ClaimStoreFailedError` - visibility-timeout claim metadata could not be stored
+  - `ConfigurationError` - invalid constructor args; also a `ValueError`
+  - `DrainFailedError` - drain pending-claim recovery failed
+  - `GatewayContractError` - custom gateway protocol violation; also a `TypeError`
+  - `LuaScriptError` - Lua `redis.error_reply(...)`; also a redis-py `ResponseError`
+  - `MalformedStoredMessageError` - stored value is not a valid RMQ envelope
+  - `PayloadTooLargeError` - serialized payload exceeds `max_payload_bytes`; also a `ValueError`
+  - `PayloadTooDeepError` - payload nesting exceeds `max_payload_depth`; also a `ValueError`
+  - `QueueBackpressureError` - `pending_overload_policy="raise"` rejected enqueue
+  - `QueueDrainedError` - `publish()` called after explicit drain/aclose
+  - `CleanupFailedError` - cleanup after handler completion failed
+  - `RetryBudgetExhaustedError` - Redis retry budget exhausted; also a redis-py `RedisError`
 
 ## Known limitations
 

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "redis-message-queue"
-version = "8.2.7"
+version = "8.2.9"
 description = "Python message queuing with Redis and message deduplication"
 authors = [{ name = "Elijas", email = "4084885+Elijas@users.noreply.github.com" }]
 readme = "README.md"
@@ -48,7 +48,7 @@ default-groups = ["dev", "test"]
 ##############################
 
 [tool.bumpversion]
-current_version = "8.2.7"
+current_version = "8.2.9"
 parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
 serialize = ["{major}.{minor}.{patch}"]
 search = "{current_version}"

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/redis_message_queue/_config.py

@@ -891,10 +891,11 @@ local dead_lettered_events = {}
 local claim_store_failed_sentinel = string.char(0) .. '__rmq_claim_store_failed__'
 
 local function store_claim_and_return(stored)
-    -- pcall guards against OOM mid-write: compensate by returning message to pending
+    -- pcall guards against OOM mid-write: fail fast while preserving a live payload copy.
+    local lease_token = nil
     local ok, result = pcall(function()
         redis.call('INCR', KEYS[5])
-        local lease_token = redis.call('GET', KEYS[5])
+        lease_token = redis.call('GET', KEYS[5])
         local claim_payload = cjson.encode({stored, lease_token})
         redis.call('ZADD', KEYS[3], now_ms + tonumber(ARGV[1]), stored)
         redis.call('HSET', KEYS[4], stored, lease_token)
@@ -906,8 +907,20 @@ local function store_claim_and_return(stored)
     end)
     if not ok then
         redis.call('HINCRBY', KEYS[6], stored, -1)
+        local return_result = redis.pcall('RPUSH', KEYS[1], stored)
+        if type(return_result) == 'table' and return_result['err'] then
+            local failure = tostring(result) .. '; return-to-pending failed: ' .. tostring(return_result['err'])
+            return {claim_store_failed_sentinel, failure, stored}
+        end
         redis.call('LREM', KEYS[2], 1, stored)
-        redis.pcall('RPUSH', KEYS[1], stored)
+        redis.call('ZREM', KEYS[3], stored)
+        redis.call('HDEL', KEYS[4], stored)
+        redis.call('DEL', KEYS[8])
+        redis.call('HDEL', KEYS[10], ARGV[4])
+        if lease_token then
+            redis.call('HDEL', KEYS[9], lease_token)
+            redis.call('HDEL', KEYS[11], lease_token)
+        end
         return {claim_store_failed_sentinel, tostring(result), stored}
     end
     return result

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/redis_message_queue/_exceptions.py

@@ -51,8 +51,9 @@ class CleanupFailedError(RedisMessageQueueError):
 class ClaimStoreFailedError(RedisMessageQueueError):
     """Raised when the VT-claim Lua store_claim_and_return pcall failed.
 
-    The script decremented the speculative delivery_count increment and
-    compensated by returning the message to pending before surfacing this error.
+    The script decremented the speculative delivery_count increment. It returns
+    the message to pending when that write succeeds; if that write fails, it
+    leaves the message in processing so the payload still has a live queue copy.
     """
 
 

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/redis_message_queue/_redis_gateway.py

@@ -987,7 +987,8 @@ class RedisGateway(AbstractRedisGateway):
             stored_message = result[2] if len(result) > 2 else None
             message_id = extract_stored_message_id(stored_message) if isinstance(stored_message, (str, bytes)) else None
             raise ClaimStoreFailedError(
-                f"VT claim store failed after delivery_count rollback: {_decode_lua_error(result[1])}",
+                f"VT claim store failed after delivery_count rollback and payload preservation: "
+                f"{_decode_lua_error(result[1])}",
                 queue=from_queue,
                 message_id=message_id,
                 operation="claim",

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/redis_message_queue/asyncio/_redis_gateway.py

@@ -967,7 +967,8 @@ class RedisGateway(AbstractRedisGateway):
             stored_message = result[2] if len(result) > 2 else None
             message_id = extract_stored_message_id(stored_message) if isinstance(stored_message, (str, bytes)) else None
             raise ClaimStoreFailedError(
-                f"VT claim store failed after delivery_count rollback: {_decode_lua_error(result[1])}",
+                f"VT claim store failed after delivery_count rollback and payload preservation: "
+                f"{_decode_lua_error(result[1])}",
                 queue=from_queue,
                 message_id=message_id,
                 operation="claim",

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/redis_message_queue/asyncio/redis_message_queue.py

@@ -87,6 +87,12 @@ def _warning_exception_name(exc: BaseException) -> str:
     return type(exc).__name__
 
 
+def _warn_runtime_warning(message: str, *, stacklevel: int) -> None:
+    with warnings.catch_warnings():
+        warnings.simplefilter("always", RuntimeWarning)
+        warnings.warn(message, RuntimeWarning, stacklevel=stacklevel + 1)
+
+
 def _find_non_string_dict_keys(value: object) -> list[object]:
     non_str_keys: list[object] = []
     seen: set[int] = set()
@@ -607,8 +613,9 @@ class RedisMessageQueue:
     ):
         """Create a queue bound to an async Redis client or custom gateway.
 
-        ``visibility_timeout_seconds`` defaults to 300. Set it to ``None`` to
-        disable lease-based crash recovery; messages left in ``processing`` by a
+        ``visibility_timeout_seconds`` defaults to 300. To disable
+        lease-based crash recovery, set both ``visibility_timeout_seconds=None``
+        and ``max_delivery_count=None``; messages left in ``processing`` by a
         crashed worker are then not reclaimed automatically.
 
         ``visibility_timeout_seconds`` is a Redis server-time lease, not a
@@ -1272,7 +1279,7 @@ class RedisMessageQueue:
                         message_id=message_id,
                         lease_token_hash=lease_token_hash,
                     )
-                    warnings.warn(_STALE_LEASE_NACK_WARNING, RuntimeWarning, stacklevel=2)
+                    _warn_runtime_warning(_STALE_LEASE_NACK_WARNING, stacklevel=2)
             except BaseException as cleanup_exc:
                 # The handler exception is the user-visible failure; cleanup failure is secondary.
                 logger.exception("Failed to clean up message from processing queue")
@@ -1285,10 +1292,9 @@ class RedisMessageQueue:
                     error=cleanup_exc,
                     duration_ms=_duration_ms(cleanup_started_at),
                 )
-                warnings.warn(
+                _warn_runtime_warning(
                     f"Cleanup raised after handler exception ({_warning_exception_name(cleanup_exc)}); "
                     "see logs for both tracebacks",
-                    RuntimeWarning,
                     stacklevel=2,
                 )
             raise
@@ -1348,7 +1354,7 @@ class RedisMessageQueue:
                     message_id=message_id,
                     lease_token_hash=lease_token_hash,
                 )
-                warnings.warn(_STALE_LEASE_ACK_WARNING, RuntimeWarning, stacklevel=2)
+                _warn_runtime_warning(_STALE_LEASE_ACK_WARNING, stacklevel=2)
             finished_without_error = True
         finally:
             if lease_heartbeat is not None:
@@ -1437,9 +1443,8 @@ class RedisMessageQueue:
                     exception_type=type(exc).__name__,
                     error=exc,
                 )
-                warnings.warn(
+                _warn_runtime_warning(
                     f"Failed to trim queue {destination_queue} ({type(exc).__name__}); list may exceed max_*_length",
-                    RuntimeWarning,
                     stacklevel=3,
                 )
 

{redis_message_queue-8.2.7 → redis_message_queue-8.2.9}/redis_message_queue/redis_message_queue.py

@@ -100,6 +100,12 @@ def _warning_exception_name(exc: BaseException) -> str:
     return type(exc).__name__
 
 
+def _warn_runtime_warning(message: str, *, stacklevel: int) -> None:
+    with warnings.catch_warnings():
+        warnings.simplefilter("always", RuntimeWarning)
+        warnings.warn(message, RuntimeWarning, stacklevel=stacklevel + 1)
+
+
 def _find_non_string_dict_keys(value: object) -> list[object]:
     non_str_keys: list[object] = []
     seen: set[int] = set()
@@ -339,12 +345,16 @@ def _close_or_cancel_awaitable(awaitable: object) -> None:
         try:
             close()
             return
+        except asyncio.CancelledError:
+            pass
         except Exception:
             pass
     cancel = getattr(awaitable, "cancel", None)
     if callable(cancel):
         try:
             cancel()
+        except asyncio.CancelledError:
+            pass
         except Exception:
             pass
 
@@ -566,8 +576,9 @@ class RedisMessageQueue:
     ):
         """Create a queue bound to a Redis client or custom gateway.
 
-        ``visibility_timeout_seconds`` defaults to 300. Set it to ``None`` to
-        disable lease-based crash recovery; messages left in ``processing`` by a
+        ``visibility_timeout_seconds`` defaults to 300. To disable
+        lease-based crash recovery, set both ``visibility_timeout_seconds=None``
+        and ``max_delivery_count=None``; messages left in ``processing`` by a
         crashed worker are then not reclaimed automatically.
 
         ``visibility_timeout_seconds`` is a Redis server-time lease, not a
@@ -1234,7 +1245,7 @@ class RedisMessageQueue:
                         message_id=message_id,
                         lease_token_hash=lease_token_hash,
                     )
-                    warnings.warn(_STALE_LEASE_NACK_WARNING, RuntimeWarning, stacklevel=2)
+                    _warn_runtime_warning(_STALE_LEASE_NACK_WARNING, stacklevel=2)
             except BaseException as cleanup_exc:
                 # The handler exception is the user-visible failure; cleanup failure is secondary.
                 logger.exception("Failed to clean up message from processing queue")
@@ -1247,10 +1258,9 @@ class RedisMessageQueue:
                     error=cleanup_exc,
                     duration_ms=_duration_ms(cleanup_started_at),
                 )
-                warnings.warn(
+                _warn_runtime_warning(
                     f"Cleanup raised after handler exception ({_warning_exception_name(cleanup_exc)}); "
                     "see logs for both tracebacks",
-                    RuntimeWarning,
                     stacklevel=2,
                 )
             raise
@@ -1309,7 +1319,7 @@ class RedisMessageQueue:
                     message_id=message_id,
                     lease_token_hash=lease_token_hash,
                 )
-                warnings.warn(_STALE_LEASE_ACK_WARNING, RuntimeWarning, stacklevel=2)
+                _warn_runtime_warning(_STALE_LEASE_ACK_WARNING, stacklevel=2)
         finally:
             if lease_heartbeat is not None:
                 lease_heartbeat.stop()
@@ -1399,9 +1409,8 @@ class RedisMessageQueue:
                     exception_type=type(exc).__name__,
                     error=exc,
                 )
-                warnings.warn(
+                _warn_runtime_warning(
                     f"Failed to trim queue {destination_queue} ({type(exc).__name__}); list may exceed max_*_length",
-                    RuntimeWarning,
                     stacklevel=3,
                 )