redis-message-queue 2.0.0__tar.gz → 2.1.0__tar.gz

{redis_message_queue-2.0.0 → redis_message_queue-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 2.0.0
+Version: 2.1.0
 Summary: Python message queuing with Redis and message deduplication
 License-File: LICENSE
 Author: Elijas
@@ -16,7 +16,7 @@ Description-Content-Type: text/markdown
 
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v2.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v2.1.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -78,7 +78,7 @@ while True:
 | **Dead-letter queue** | Poison messages that exceed a configurable delivery count are automatically routed to a dead-letter queue instead of being redelivered indefinitely |
 | **Graceful shutdown** | Built-in interrupt handler lets consumers finish current work before stopping |
 | **Lease heartbeats** | Optional background lease renewal keeps long-running handlers from being redelivered prematurely |
-| **Connection retries** | Exponential backoff with jitter for Redis operations (deduplicated publish, ack, lease renewal). Message-claim paths use idempotent Lua claim IDs so retryable errors can recover the original claim safely, including on the next call from the same gateway instance if the original wait call had to give up before Redis became reachable again. Non-deduplicated publish is not retried — the exception propagates so the caller can decide whether to retry (accepting potential duplicates) |
+| **Connection retries** | Exponential backoff with jitter for Redis operations (deduplicated publish, ack, lease renewal). Publish and cleanup paths use replay markers so retryable connection drops preserve the original result within the same call. Message-claim paths use idempotent Lua claim IDs plus persisted claim metadata so retryable errors can recover the original claim safely, either in the same wait call or on the next call from the same gateway instance if the original wait had to give up before Redis became reachable again. Active waits keep their in-flight claim IDs private until they exit, so a concurrent caller on the same gateway instance cannot recover the same claim twice. Timed waits also stay bounded: once the configured wait window expires, the queue only replays persisted state for that same claim attempt and will not claim fresh work after the deadline. If a graceful interrupt arrives during claim recovery, the wait call stops instead of taking fresh work. Non-deduplicated publish is not retried — the exception propagates so the caller can decide whether to retry (accepting potential duplicates) |
 | **Async support** | Drop-in async variant with identical API |
 
 All features are optional and can be enabled or disabled as needed.
@@ -155,6 +155,19 @@ Tradeoffs:
 - heartbeats add background renewal work for active messages
 - if a heartbeat fails (network error or stale lease), the heartbeat stops silently; the consumer continues processing but may find at ack time that the message was reclaimed by another consumer
 
+Pass `on_heartbeat_failure` to receive a best-effort callback when the heartbeat stops because renewal failed:
+
+```python
+queue = RedisMessageQueue(
+    "q", client=client,
+    visibility_timeout_seconds=300,
+    heartbeat_interval_seconds=60,
+    on_heartbeat_failure=lambda: log.warning("heartbeat failed; lease may be stale"),
+)
+```
+
+The callback is **advisory** — it may fire briefly after a successful `process_message` exit when a final renewal coincided with the success path. Use it for metrics or alerting, not as a correctness signal. For the async queue (`redis_message_queue.asyncio`), the callback may also be `async def`.
+
 Without a visibility timeout, messages already moved to `processing` remain there indefinitely after a consumer crash and are not redelivered, even if the crash happened before your handler started running.
 
 ### Dead-letter queue
@@ -176,6 +189,7 @@ Notes:
 - the delivery count increments when Redis grants the claim/lease, not when your handler begins running. If a process exits after Redis claims a message, that claim still counts toward `max_delivery_count`
 - `max_delivery_count=1` means the message is delivered once; any reclaim routes it to the dead-letter queue
 - without `max_delivery_count`, messages are redelivered indefinitely (existing behavior)
+- dead-lettered messages contain the **raw payload** only — the internal envelope (which carries a per-delivery UUID) is stripped before pushing to the DLQ, consistent with how completed/failed queues store messages. Two identical payloads dead-lettered separately are indistinguishable in the DLQ
 
 ### Graceful shutdown
 
@@ -196,8 +210,9 @@ while not interrupt.is_interrupted():
 > its signals (default: SIGINT, SIGTERM, SIGHUP), but only when those signals are
 > still using Python's default disposition. If another handler is already installed,
 > or if another `GracefulInterruptHandler` already owns the signal, construction raises
-> `ValueError`. If you need multiple shutdown hooks, use a single handler and fan out
-> in your own code.
+> `ValueError`. A repeated owned signal falls back to the default behavior
+> (for example, a second Ctrl+C raises `KeyboardInterrupt`). If you need multiple
+> shutdown hooks, use a single handler and fan out in your own code.
 
 ### Custom gateway
 
@@ -220,6 +235,20 @@ If your custom gateway uses visibility timeouts, it must expose a public
 `wait_for_message_and_move()`. The queue now fails closed if a lease-capable
 gateway returns plain `str`/`bytes`, because cleanup without a lease token can
 ack a message that has already been reclaimed by another consumer.
+If a lease-capable custom gateway omits `message_visibility_timeout_seconds`,
+the queue cannot detect that lease semantics are in play and will treat the
+gateway as a non-lease gateway. In that misconfigured state, lease-token safety
+checks and heartbeat validation are bypassed.
+
+A custom `retry_strategy` MUST have a total retry budget no longer than
+`max(message_visibility_timeout_seconds, 300)` seconds. That value is the TTL
+of the built-in gateway's ambiguous-success cache: if a retry arrives after the
+cache has expired, the gateway re-runs the Lua script and — because the message
+was already acked on the first attempt — sees `LREM=0` and returns `False`. This
+surfaces as a misleading "cleanup was a no-op" warning from `process_message`;
+no data is lost or double-processed, but a `max_completed_length` /
+`max_failed_length` bound may be skipped on that call. The default
+`tenacity.stop_after_delay(120)` is safely within the 300 s floor.
 
 When using a custom gateway with dead-letter queue support, configure `max_delivery_count`
 and `dead_letter_queue` directly on the gateway — do **not** pass `max_delivery_count` to
@@ -261,9 +290,10 @@ await client.aclose()
 
 - **No metrics or observability hooks.** The library logs warnings (stale leases, heartbeat failures, transient errors) via Python's `logging` module but does not expose callbacks, event hooks, or metric counters. To monitor queue health, inspect the underlying Redis keys directly or parse log output.
 - **Timed waits use polling claim loops.** To make claims recoverable after ambiguous connection drops, `wait_for_message_and_move()` uses idempotent Lua claim polling instead of raw blocking list-move commands. This adds a small polling cadence during timed waits.
-- **Low-level gateway boolean returns can be conservative after retries.** If a connection drops after Redis already applied an idempotent operation, direct gateway calls such as `publish_message()`, plus non-lease `move_message()` / `remove_message()`, may return `False` on retry even though Redis state is already correct.
+- **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
 - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
 - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
+- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk.
 
 For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
 

{redis_message_queue-2.0.0 → redis_message_queue-2.1.0}/README.md

@@ -1,6 +1,6 @@
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v2.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v2.1.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -62,7 +62,7 @@ while True:
 | **Dead-letter queue** | Poison messages that exceed a configurable delivery count are automatically routed to a dead-letter queue instead of being redelivered indefinitely |
 | **Graceful shutdown** | Built-in interrupt handler lets consumers finish current work before stopping |
 | **Lease heartbeats** | Optional background lease renewal keeps long-running handlers from being redelivered prematurely |
-| **Connection retries** | Exponential backoff with jitter for Redis operations (deduplicated publish, ack, lease renewal). Message-claim paths use idempotent Lua claim IDs so retryable errors can recover the original claim safely, including on the next call from the same gateway instance if the original wait call had to give up before Redis became reachable again. Non-deduplicated publish is not retried — the exception propagates so the caller can decide whether to retry (accepting potential duplicates) |
+| **Connection retries** | Exponential backoff with jitter for Redis operations (deduplicated publish, ack, lease renewal). Publish and cleanup paths use replay markers so retryable connection drops preserve the original result within the same call. Message-claim paths use idempotent Lua claim IDs plus persisted claim metadata so retryable errors can recover the original claim safely, either in the same wait call or on the next call from the same gateway instance if the original wait had to give up before Redis became reachable again. Active waits keep their in-flight claim IDs private until they exit, so a concurrent caller on the same gateway instance cannot recover the same claim twice. Timed waits also stay bounded: once the configured wait window expires, the queue only replays persisted state for that same claim attempt and will not claim fresh work after the deadline. If a graceful interrupt arrives during claim recovery, the wait call stops instead of taking fresh work. Non-deduplicated publish is not retried — the exception propagates so the caller can decide whether to retry (accepting potential duplicates) |
 | **Async support** | Drop-in async variant with identical API |
 
 All features are optional and can be enabled or disabled as needed.
@@ -139,6 +139,19 @@ Tradeoffs:
 - heartbeats add background renewal work for active messages
 - if a heartbeat fails (network error or stale lease), the heartbeat stops silently; the consumer continues processing but may find at ack time that the message was reclaimed by another consumer
 
+Pass `on_heartbeat_failure` to receive a best-effort callback when the heartbeat stops because renewal failed:
+
+```python
+queue = RedisMessageQueue(
+    "q", client=client,
+    visibility_timeout_seconds=300,
+    heartbeat_interval_seconds=60,
+    on_heartbeat_failure=lambda: log.warning("heartbeat failed; lease may be stale"),
+)
+```
+
+The callback is **advisory** — it may fire briefly after a successful `process_message` exit when a final renewal coincided with the success path. Use it for metrics or alerting, not as a correctness signal. For the async queue (`redis_message_queue.asyncio`), the callback may also be `async def`.
+
 Without a visibility timeout, messages already moved to `processing` remain there indefinitely after a consumer crash and are not redelivered, even if the crash happened before your handler started running.
 
 ### Dead-letter queue
@@ -160,6 +173,7 @@ Notes:
 - the delivery count increments when Redis grants the claim/lease, not when your handler begins running. If a process exits after Redis claims a message, that claim still counts toward `max_delivery_count`
 - `max_delivery_count=1` means the message is delivered once; any reclaim routes it to the dead-letter queue
 - without `max_delivery_count`, messages are redelivered indefinitely (existing behavior)
+- dead-lettered messages contain the **raw payload** only — the internal envelope (which carries a per-delivery UUID) is stripped before pushing to the DLQ, consistent with how completed/failed queues store messages. Two identical payloads dead-lettered separately are indistinguishable in the DLQ
 
 ### Graceful shutdown
 
@@ -180,8 +194,9 @@ while not interrupt.is_interrupted():
 > its signals (default: SIGINT, SIGTERM, SIGHUP), but only when those signals are
 > still using Python's default disposition. If another handler is already installed,
 > or if another `GracefulInterruptHandler` already owns the signal, construction raises
-> `ValueError`. If you need multiple shutdown hooks, use a single handler and fan out
-> in your own code.
+> `ValueError`. A repeated owned signal falls back to the default behavior
+> (for example, a second Ctrl+C raises `KeyboardInterrupt`). If you need multiple
+> shutdown hooks, use a single handler and fan out in your own code.
 
 ### Custom gateway
 
@@ -204,6 +219,20 @@ If your custom gateway uses visibility timeouts, it must expose a public
 `wait_for_message_and_move()`. The queue now fails closed if a lease-capable
 gateway returns plain `str`/`bytes`, because cleanup without a lease token can
 ack a message that has already been reclaimed by another consumer.
+If a lease-capable custom gateway omits `message_visibility_timeout_seconds`,
+the queue cannot detect that lease semantics are in play and will treat the
+gateway as a non-lease gateway. In that misconfigured state, lease-token safety
+checks and heartbeat validation are bypassed.
+
+A custom `retry_strategy` MUST have a total retry budget no longer than
+`max(message_visibility_timeout_seconds, 300)` seconds. That value is the TTL
+of the built-in gateway's ambiguous-success cache: if a retry arrives after the
+cache has expired, the gateway re-runs the Lua script and — because the message
+was already acked on the first attempt — sees `LREM=0` and returns `False`. This
+surfaces as a misleading "cleanup was a no-op" warning from `process_message`;
+no data is lost or double-processed, but a `max_completed_length` /
+`max_failed_length` bound may be skipped on that call. The default
+`tenacity.stop_after_delay(120)` is safely within the 300 s floor.
 
 When using a custom gateway with dead-letter queue support, configure `max_delivery_count`
 and `dead_letter_queue` directly on the gateway — do **not** pass `max_delivery_count` to
@@ -245,9 +274,10 @@ await client.aclose()
 
 - **No metrics or observability hooks.** The library logs warnings (stale leases, heartbeat failures, transient errors) via Python's `logging` module but does not expose callbacks, event hooks, or metric counters. To monitor queue health, inspect the underlying Redis keys directly or parse log output.
 - **Timed waits use polling claim loops.** To make claims recoverable after ambiguous connection drops, `wait_for_message_and_move()` uses idempotent Lua claim polling instead of raw blocking list-move commands. This adds a small polling cadence during timed waits.
-- **Low-level gateway boolean returns can be conservative after retries.** If a connection drops after Redis already applied an idempotent operation, direct gateway calls such as `publish_message()`, plus non-lease `move_message()` / `remove_message()`, may return `False` on retry even though Redis state is already correct.
+- **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
 - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
 - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
+- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk.
 
 For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
 

{redis_message_queue-2.0.0 → redis_message_queue-2.1.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "redis-message-queue"
-version = "2.0.0"
+version = "2.1.0"
 description = "Python message queuing with Redis and message deduplication"
 authors = ["Elijas <4084885+Elijas@users.noreply.github.com>"]
 readme = "README.md"

{redis_message_queue-2.0.0 → redis_message_queue-2.1.0}/redis_message_queue/_abstract_redis_gateway.py

@@ -15,6 +15,9 @@ class AbstractRedisGateway(ABC):
     a ``message_visibility_timeout_seconds`` property (int or None). This is not
     abstract because it is configuration rather than protocol, but it is required
     when the queue is configured with ``heartbeat_interval_seconds``.
+    Lease-capable custom gateways should always expose this property; otherwise
+    the queue cannot enforce lease-specific fail-closed checks and will treat the
+    gateway as a non-lease implementation.
 
     Concurrency
     -----------
@@ -57,9 +60,13 @@ class AbstractRedisGateway(ABC):
 
         When ``lease_token`` is provided, the implementation MUST validate that
         the token matches the current lease holder before moving. If the token
-        is stale or the lease has expired, the method MUST return False and
-        leave the message in ``from_queue``. Ignoring ``lease_token`` silently
-        breaks mutual exclusion.
+        no longer matches the current lease holder (i.e. another consumer has
+        reclaimed the message), the method MUST return False and leave the
+        message in ``from_queue``. Note: the built-in gateway intentionally
+        does NOT reject completions whose wall-clock deadline has passed but
+        where no other consumer has reclaimed the message — that path keeps
+        at-least-once semantics from producing spurious double-processing.
+        Ignoring ``lease_token`` entirely silently breaks mutual exclusion.
 
         Returns True if the message was moved, False otherwise.
         """
@@ -75,8 +82,13 @@ class AbstractRedisGateway(ABC):
 
         When ``lease_token`` is provided, the implementation MUST validate that
         the token matches the current lease holder before removing. If the token
-        is stale or the lease has expired, the method MUST return False.
-        Ignoring ``lease_token`` silently breaks mutual exclusion.
+        no longer matches the current lease holder (i.e. another consumer has
+        reclaimed the message), the method MUST return False. Note: the
+        built-in gateway intentionally does NOT reject completions whose
+        wall-clock deadline has passed but where no other consumer has
+        reclaimed the message — that path keeps at-least-once semantics from
+        producing spurious double-processing. Ignoring ``lease_token``
+        entirely silently breaks mutual exclusion.
 
         Returns True if the message was removed, False otherwise.
         """