redis-message-queue 3.1.2__tar.gz → 5.0.0__tar.gz

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 3.1.2
+Version: 5.0.0
 Summary: Python message queuing with Redis and message deduplication
 License: MIT
 License-File: LICENSE
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v3.1.2-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v5.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -50,7 +50,7 @@ Requires Redis server >= 6.2.
 from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
-client = Redis.from_url("redis://localhost:6379/0")
+client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
 queue = RedisMessageQueue("my_queue", client=client, deduplication=True)
 
 queue.publish("order:1234")           # returns True
@@ -74,6 +74,9 @@ while True:
             # Auto-acknowledged on success; cleaned up on exception
 ```
 
+`RedisMessageQueue` itself is not a context manager. Use
+`with queue.process_message() as message:` for each message.
+
 ## Why redis-message-queue
 
 **The problem:** You're sending messages between services or workers and need guarantees. Simple Redis LPUSH/BRPOP loses messages on crashes, doesn't deduplicate, and gives you no visibility into what succeeded or failed.
@@ -97,8 +100,8 @@ All features are optional and can be enabled or disabled as needed.
 
 | Configuration | Delivery guarantee |
 |---|---|
-| Default (no visibility timeout) | **At-most-once** — a consumer crash loses the in-flight message |
-| With `visibility_timeout_seconds` | **At-least-once** — expired messages are reclaimed and redelivered |
+| Default (`visibility_timeout_seconds=300`) | **At-least-once** — expired messages are reclaimed and redelivered |
+| With `visibility_timeout_seconds=None` | **At-most-once** — a consumer crash loses the in-flight message |
 
 See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-timeout) for details and tradeoffs.
 
@@ -107,7 +110,7 @@ See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-tim
 ### Deduplication
 
 ```python
-# Default: deduplicate by full message content (1-hour TTL)
+# Default: deduplicate by SHA-256 hash of canonical message content (1-hour TTL)
 queue = RedisMessageQueue("q", client=client, deduplication=True)
 
 # Custom dedup key (e.g., deduplicate by order ID only)
@@ -131,7 +134,8 @@ queue = RedisMessageQueue(
 )
 ```
 
-To prevent unbounded growth, cap the queue lengths:
+Completed and failed tracking queues are capped at 1,000 entries by default
+when enabled. Override the caps when you need a different retention window:
 
 ```python
 queue = RedisMessageQueue(
@@ -144,6 +148,8 @@ queue = RedisMessageQueue(
 ```
 
 When set, `LTRIM` is called after each message is moved to the completed/failed queue. This is best-effort cleanup — if the trim fails, the queue is slightly longer until the next successful trim.
+Pass `max_completed_length=None` or `max_failed_length=None` explicitly if you
+want unbounded tracking queues.
 
 ### Crash recovery with visibility timeout
 
@@ -191,14 +197,14 @@ queue = RedisMessageQueue(
 )
 ```
 
-When a message has been delivered more than `max_delivery_count` times (due to consumer crashes causing visibility-timeout reclaim), it is automatically routed to a dead-letter queue (`{name}::dead_letter`) instead of being redelivered. This prevents poison messages from cycling indefinitely.
+When a message has been delivered more than `max_delivery_count` times (due to consumer crashes causing visibility-timeout reclaim), it is automatically routed to a dead-letter queue (`{name}::dlq`) instead of being redelivered. `max_delivery_count` defaults to `10` on the built-in `client=` path, with the DLQ name auto-derived from the queue name. This prevents poison messages from cycling indefinitely.
 
 Notes:
 - requires `visibility_timeout_seconds` to be set (poison messages are only a concern with VT reclaim)
 - the delivery count is tracked per-message in a Redis HASH and cleaned up on successful ack or move to completed/failed
 - the delivery count increments when Redis grants the claim/lease, not when your handler begins running. If a process exits after Redis claims a message, that claim still counts toward `max_delivery_count`
 - `max_delivery_count=1` means the message is delivered once; any reclaim routes it to the dead-letter queue
-- without `max_delivery_count`, messages are redelivered indefinitely (existing behavior)
+- set `max_delivery_count=None` explicitly for unlimited redelivery
 - dead-lettered messages contain the **raw payload** only — the internal envelope (which carries a per-delivery UUID) is stripped before pushing to the DLQ, consistent with how completed/failed queues store messages. Two identical payloads dead-lettered separately are indistinguishable in the DLQ
 
 ### Graceful shutdown
@@ -227,7 +233,7 @@ while not interrupt.is_interrupted():
 ### Custom gateway
 
 ```python
-from redis_message_queue._redis_gateway import RedisGateway
+from redis_message_queue import RedisGateway
 
 # Tune retry budget, dedup TTL, or wait interval
 gateway = RedisGateway(
@@ -253,8 +259,8 @@ impossible. Note: tenacity may allow one additional attempt beyond the budget if
 
 To plug in a different retry library (`backoff`, `asyncstdlib.retry`, or your
 own logic) or fundamentally different semantics, subclass
-`AbstractRedisGateway` from `redis_message_queue._abstract_redis_gateway`
-(or `redis_message_queue.asyncio._abstract_redis_gateway`) and override the
+`AbstractRedisGateway` from `redis_message_queue` (or
+`redis_message_queue.asyncio` for the async sibling) and override the
 operation methods directly.
 
 If your custom gateway uses visibility timeouts, it must expose a public
@@ -285,6 +291,16 @@ Use a separate gateway instance per queue when `max_delivery_count` is enabled.
 Dead-letter routing is gateway-scoped, so reusing the same gateway across different
 queues is rejected.
 
+If you use Redis Sentinel, pass the Redis client returned by
+`sentinel.master_for(name)` to `client=` or `RedisGateway(redis_client=...)`, not
+the `sentinel` object itself.
+
+### Connection pool sizing
+
+Each queue with `heartbeat_interval_seconds` set uses up to 2 simultaneous
+connections: one for the main operation and one for heartbeat renewal. Size Redis
+client pools with `max_connections >= 2 * number_of_queues + headroom`.
+
 ## Async API
 
 Replace the import to use the async variant — the API is identical:
@@ -293,6 +309,11 @@ Replace the import to use the async variant — the API is identical:
 from redis_message_queue.asyncio import RedisMessageQueue
 ```
 
+The sync and async classes intentionally share names. In modules that use both,
+alias the imports explicitly, for example
+`from redis_message_queue import RedisMessageQueue as SyncRedisMessageQueue` and
+`from redis_message_queue.asyncio import RedisMessageQueue as AsyncRedisMessageQueue`.
+
 All examples work the same way. Remember to close the connection when done:
 
 ```python
@@ -303,6 +324,9 @@ client = redis.Redis()
 await client.aclose()
 ```
 
+For the sync Redis client, call `client.close()` during application shutdown when
+you own the client lifecycle.
+
 ## Known limitations
 
 - **No metrics or observability hooks.** The library logs warnings (stale leases, heartbeat failures, transient errors) via Python's `logging` module but does not expose callbacks, event hooks, or metric counters. To monitor queue health, inspect the underlying Redis keys directly or parse log output.
@@ -310,11 +334,11 @@ await client.aclose()
 - **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
 - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
 - **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
-- **Default dedup key is the full message.** Without a custom `get_deduplication_key`, the entire serialized message becomes a Redis key name for dedup tracking. For large messages (>1KB), provide a custom key function to avoid excessive Redis memory usage.
 - **Cluster detection uses `isinstance(client, RedisCluster)`.** Wrapped or instrumented cluster clients that delegate without inheriting will bypass hash-tag validation. Custom gateways should set `is_redis_cluster = True` explicitly.
 - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
 - **Non-ASCII payloads use ~2x storage.** The default `ensure_ascii=True` in JSON serialization encodes non-ASCII characters as `\uXXXX` escape sequences. This is a deliberate compatibility choice.
 - **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
+- **Redis Cluster default retry can stack with this library's retry budget.** In redis-py 6.0+, `RedisCluster()` constructs a default `ExponentialWithJitterBackoff` retry below this library's `retry_budget_seconds`. If you need a single retry surface, pass `retry=Retry(NoBackoff(), 0)` to the cluster client or reduce `retry_budget_seconds` to account for the lower-level retry window.
 
 For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
 
@@ -324,9 +348,17 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 
 > **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
 
-- **Do not change `key_separator` on a live queue.** All existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing separators.
-- **Do not switch from no-VT to VT with messages in processing.** Messages claimed by non-VT consumers have no lease deadline entries. VT-enabled consumers cannot reclaim them. Drain the processing queue first.
+- **Do not change `name` or `key_separator` on a live queue.** Both settings define the Redis key namespace. Existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing either value.
+- **Do not rename `dead_letter_queue` on a live queue.** Existing DLQ records stay in the old list, while new failures route to the new list. Inspect or drain the old DLQ manually before switching names.
+- **Do not toggle visibility timeout in either direction with messages in processing.** Messages claimed by non-VT consumers have no lease metadata, so VT-enabled consumers cannot reclaim them. Disabling VT later orphans existing lease deadline, lease token, and delivery count metadata and removes crash recovery for those in-flight messages. Drain the processing queue first.
 - **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
+- **Changing `max_delivery_count` from a number to `None` leaves delivery metadata behind.** The delivery count hash continues to exist but is no longer consulted. Use this only after draining or after planning manual cleanup of the delivery-count hash.
+- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching between the default hash, explicit `None`, or a custom key function.
+- **Disabling `deduplication` has a retention-window overlap.** Existing dedup records remain in Redis until their TTL expires, but new publishes bypass them. Republishes that would have been suppressed under the old setting can enqueue during that window.
+- **Disabling `enable_failed_queue` stops recording handler failures.** Existing failed entries remain in Redis, but new failures are removed from `processing` without being appended to the failed queue. If `max_delivery_count=None` is also set, repeated handler failures can be dropped with no DLQ or failed-queue record; see [Dead-letter queue](#dead-letter-queue).
+- **Lowering `max_completed_length` or `max_failed_length` trims existing history.** The next completed or failed move calls `LTRIM`, so changing `None` to `N` or lowering `N` can immediately reduce historical entries to the new cap.
+- **Do not switch sync and async gateway instances mid-process while claims are active.** Redis state is compatible across deploys, but each gateway instance keeps its own pending claim-recovery IDs. In-flight claim recovery state does not transfer between instances.
+- **Switching between `gateway=` and `client=` can retarget the DLQ.** The built-in `client=` path derives the DLQ from the queue name. If a custom gateway used a different `dead_letter_queue`, switching paths has the same orphaning impact as renaming the DLQ.
 
 ### v2 to v3 migration
 

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/README.md

@@ -1,6 +1,6 @@
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v3.1.2-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v5.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -24,7 +24,7 @@ Requires Redis server >= 6.2.
 from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
-client = Redis.from_url("redis://localhost:6379/0")
+client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
 queue = RedisMessageQueue("my_queue", client=client, deduplication=True)
 
 queue.publish("order:1234")           # returns True
@@ -48,6 +48,9 @@ while True:
             # Auto-acknowledged on success; cleaned up on exception
 ```
 
+`RedisMessageQueue` itself is not a context manager. Use
+`with queue.process_message() as message:` for each message.
+
 ## Why redis-message-queue
 
 **The problem:** You're sending messages between services or workers and need guarantees. Simple Redis LPUSH/BRPOP loses messages on crashes, doesn't deduplicate, and gives you no visibility into what succeeded or failed.
@@ -71,8 +74,8 @@ All features are optional and can be enabled or disabled as needed.
 
 | Configuration | Delivery guarantee |
 |---|---|
-| Default (no visibility timeout) | **At-most-once** — a consumer crash loses the in-flight message |
-| With `visibility_timeout_seconds` | **At-least-once** — expired messages are reclaimed and redelivered |
+| Default (`visibility_timeout_seconds=300`) | **At-least-once** — expired messages are reclaimed and redelivered |
+| With `visibility_timeout_seconds=None` | **At-most-once** — a consumer crash loses the in-flight message |
 
 See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-timeout) for details and tradeoffs.
 
@@ -81,7 +84,7 @@ See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-tim
 ### Deduplication
 
 ```python
-# Default: deduplicate by full message content (1-hour TTL)
+# Default: deduplicate by SHA-256 hash of canonical message content (1-hour TTL)
 queue = RedisMessageQueue("q", client=client, deduplication=True)
 
 # Custom dedup key (e.g., deduplicate by order ID only)
@@ -105,7 +108,8 @@ queue = RedisMessageQueue(
 )
 ```
 
-To prevent unbounded growth, cap the queue lengths:
+Completed and failed tracking queues are capped at 1,000 entries by default
+when enabled. Override the caps when you need a different retention window:
 
 ```python
 queue = RedisMessageQueue(
@@ -118,6 +122,8 @@ queue = RedisMessageQueue(
 ```
 
 When set, `LTRIM` is called after each message is moved to the completed/failed queue. This is best-effort cleanup — if the trim fails, the queue is slightly longer until the next successful trim.
+Pass `max_completed_length=None` or `max_failed_length=None` explicitly if you
+want unbounded tracking queues.
 
 ### Crash recovery with visibility timeout
 
@@ -165,14 +171,14 @@ queue = RedisMessageQueue(
 )
 ```
 
-When a message has been delivered more than `max_delivery_count` times (due to consumer crashes causing visibility-timeout reclaim), it is automatically routed to a dead-letter queue (`{name}::dead_letter`) instead of being redelivered. This prevents poison messages from cycling indefinitely.
+When a message has been delivered more than `max_delivery_count` times (due to consumer crashes causing visibility-timeout reclaim), it is automatically routed to a dead-letter queue (`{name}::dlq`) instead of being redelivered. `max_delivery_count` defaults to `10` on the built-in `client=` path, with the DLQ name auto-derived from the queue name. This prevents poison messages from cycling indefinitely.
 
 Notes:
 - requires `visibility_timeout_seconds` to be set (poison messages are only a concern with VT reclaim)
 - the delivery count is tracked per-message in a Redis HASH and cleaned up on successful ack or move to completed/failed
 - the delivery count increments when Redis grants the claim/lease, not when your handler begins running. If a process exits after Redis claims a message, that claim still counts toward `max_delivery_count`
 - `max_delivery_count=1` means the message is delivered once; any reclaim routes it to the dead-letter queue
-- without `max_delivery_count`, messages are redelivered indefinitely (existing behavior)
+- set `max_delivery_count=None` explicitly for unlimited redelivery
 - dead-lettered messages contain the **raw payload** only — the internal envelope (which carries a per-delivery UUID) is stripped before pushing to the DLQ, consistent with how completed/failed queues store messages. Two identical payloads dead-lettered separately are indistinguishable in the DLQ
 
 ### Graceful shutdown
@@ -201,7 +207,7 @@ while not interrupt.is_interrupted():
 ### Custom gateway
 
 ```python
-from redis_message_queue._redis_gateway import RedisGateway
+from redis_message_queue import RedisGateway
 
 # Tune retry budget, dedup TTL, or wait interval
 gateway = RedisGateway(
@@ -227,8 +233,8 @@ impossible. Note: tenacity may allow one additional attempt beyond the budget if
 
 To plug in a different retry library (`backoff`, `asyncstdlib.retry`, or your
 own logic) or fundamentally different semantics, subclass
-`AbstractRedisGateway` from `redis_message_queue._abstract_redis_gateway`
-(or `redis_message_queue.asyncio._abstract_redis_gateway`) and override the
+`AbstractRedisGateway` from `redis_message_queue` (or
+`redis_message_queue.asyncio` for the async sibling) and override the
 operation methods directly.
 
 If your custom gateway uses visibility timeouts, it must expose a public
@@ -259,6 +265,16 @@ Use a separate gateway instance per queue when `max_delivery_count` is enabled.
 Dead-letter routing is gateway-scoped, so reusing the same gateway across different
 queues is rejected.
 
+If you use Redis Sentinel, pass the Redis client returned by
+`sentinel.master_for(name)` to `client=` or `RedisGateway(redis_client=...)`, not
+the `sentinel` object itself.
+
+### Connection pool sizing
+
+Each queue with `heartbeat_interval_seconds` set uses up to 2 simultaneous
+connections: one for the main operation and one for heartbeat renewal. Size Redis
+client pools with `max_connections >= 2 * number_of_queues + headroom`.
+
 ## Async API
 
 Replace the import to use the async variant — the API is identical:
@@ -267,6 +283,11 @@ Replace the import to use the async variant — the API is identical:
 from redis_message_queue.asyncio import RedisMessageQueue
 ```
 
+The sync and async classes intentionally share names. In modules that use both,
+alias the imports explicitly, for example
+`from redis_message_queue import RedisMessageQueue as SyncRedisMessageQueue` and
+`from redis_message_queue.asyncio import RedisMessageQueue as AsyncRedisMessageQueue`.
+
 All examples work the same way. Remember to close the connection when done:
 
 ```python
@@ -277,6 +298,9 @@ client = redis.Redis()
 await client.aclose()
 ```
 
+For the sync Redis client, call `client.close()` during application shutdown when
+you own the client lifecycle.
+
 ## Known limitations
 
 - **No metrics or observability hooks.** The library logs warnings (stale leases, heartbeat failures, transient errors) via Python's `logging` module but does not expose callbacks, event hooks, or metric counters. To monitor queue health, inspect the underlying Redis keys directly or parse log output.
@@ -284,11 +308,11 @@ await client.aclose()
 - **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
 - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
 - **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
-- **Default dedup key is the full message.** Without a custom `get_deduplication_key`, the entire serialized message becomes a Redis key name for dedup tracking. For large messages (>1KB), provide a custom key function to avoid excessive Redis memory usage.
 - **Cluster detection uses `isinstance(client, RedisCluster)`.** Wrapped or instrumented cluster clients that delegate without inheriting will bypass hash-tag validation. Custom gateways should set `is_redis_cluster = True` explicitly.
 - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
 - **Non-ASCII payloads use ~2x storage.** The default `ensure_ascii=True` in JSON serialization encodes non-ASCII characters as `\uXXXX` escape sequences. This is a deliberate compatibility choice.
 - **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
+- **Redis Cluster default retry can stack with this library's retry budget.** In redis-py 6.0+, `RedisCluster()` constructs a default `ExponentialWithJitterBackoff` retry below this library's `retry_budget_seconds`. If you need a single retry surface, pass `retry=Retry(NoBackoff(), 0)` to the cluster client or reduce `retry_budget_seconds` to account for the lower-level retry window.
 
 For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
 
@@ -298,9 +322,17 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 
 > **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
 
-- **Do not change `key_separator` on a live queue.** All existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing separators.
-- **Do not switch from no-VT to VT with messages in processing.** Messages claimed by non-VT consumers have no lease deadline entries. VT-enabled consumers cannot reclaim them. Drain the processing queue first.
+- **Do not change `name` or `key_separator` on a live queue.** Both settings define the Redis key namespace. Existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing either value.
+- **Do not rename `dead_letter_queue` on a live queue.** Existing DLQ records stay in the old list, while new failures route to the new list. Inspect or drain the old DLQ manually before switching names.
+- **Do not toggle visibility timeout in either direction with messages in processing.** Messages claimed by non-VT consumers have no lease metadata, so VT-enabled consumers cannot reclaim them. Disabling VT later orphans existing lease deadline, lease token, and delivery count metadata and removes crash recovery for those in-flight messages. Drain the processing queue first.
 - **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
+- **Changing `max_delivery_count` from a number to `None` leaves delivery metadata behind.** The delivery count hash continues to exist but is no longer consulted. Use this only after draining or after planning manual cleanup of the delivery-count hash.
+- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching between the default hash, explicit `None`, or a custom key function.
+- **Disabling `deduplication` has a retention-window overlap.** Existing dedup records remain in Redis until their TTL expires, but new publishes bypass them. Republishes that would have been suppressed under the old setting can enqueue during that window.
+- **Disabling `enable_failed_queue` stops recording handler failures.** Existing failed entries remain in Redis, but new failures are removed from `processing` without being appended to the failed queue. If `max_delivery_count=None` is also set, repeated handler failures can be dropped with no DLQ or failed-queue record; see [Dead-letter queue](#dead-letter-queue).
+- **Lowering `max_completed_length` or `max_failed_length` trims existing history.** The next completed or failed move calls `LTRIM`, so changing `None` to `N` or lowering `N` can immediately reduce historical entries to the new cap.
+- **Do not switch sync and async gateway instances mid-process while claims are active.** Redis state is compatible across deploys, but each gateway instance keeps its own pending claim-recovery IDs. In-flight claim recovery state does not transfer between instances.
+- **Switching between `gateway=` and `client=` can retarget the DLQ.** The built-in `client=` path derives the DLQ from the queue name. If a custom gateway used a different `dead_letter_queue`, switching paths has the same orphaning impact as renaming the DLQ.
 
 ### v2 to v3 migration
 

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/pyproject.toml

@@ -1,10 +1,11 @@
 [tool.poetry]
 name = "redis-message-queue"
-version = "3.1.2"
+version = "5.0.0"
 description = "Python message queuing with Redis and message deduplication"
 authors = ["Elijas <4084885+Elijas@users.noreply.github.com>"]
 readme = "README.md"
 license = "MIT"
+include = ["redis_message_queue/py.typed"]
 keywords = ["redis", "message-queue", "deduplication", "task-queue"]
 classifiers = [
     "Development Status :: 5 - Production/Stable",
@@ -46,6 +47,16 @@ ignore = ["E731"]
 
 [tool.pytest.ini_options]
 asyncio_default_fixture_loop_scope = "function"
+filterwarnings = [
+    # Channel-rule warnings are emitted at user-actionable failure sites
+    # (see _STALE_LEASE_*_WARNING, heartbeat-failure, etc.). Dedicated B8
+    # tests assert emission via pytest.warns(); other tests trigger these
+    # warnings incidentally as a side effect of exercising lifecycle paths
+    # and would fail under "error::" promotion. Ignore at the default
+    # filter so emission is silent in incidental tests but pytest.warns
+    # assertions still verify emission where intended.
+    "ignore::RuntimeWarning:redis_message_queue",
+]
 markers = [
     "integration: tests that require a real Redis server",
 ]

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/redis_message_queue/_abstract_redis_gateway.py

@@ -20,12 +20,18 @@ class AbstractRedisGateway(ABC):
     causing the queue to treat the gateway as a non-lease implementation.
 
     The queue also reads ``max_delivery_count`` and ``dead_letter_queue``
-    from the gateway via ``getattr``. Avoid using these attribute names for
-    unrelated purposes on custom gateway implementations.
-
-    Gateways that wrap a Redis Cluster client should expose an
-    ``is_redis_cluster`` property returning ``True`` so the queue can apply
-    hash-tag validation at construction time.
+    from the gateway. The abstract base provides ``None`` defaults via
+    ``@property``; lease + DLQ-enabled custom gateways MUST override both
+    to enable poison-message routing. Avoid using these attribute names
+    for unrelated purposes on custom gateway implementations.
+    When DLQ routing is enabled, the queue also attaches an internal
+    ``_rmq_bound_pending_queue`` attribute to gateway instances to reject
+    reusing one DLQ-bound gateway across multiple pending queues.
+
+    Gateways that wrap a Redis Cluster client should override the
+    ``is_redis_cluster`` property to return ``True`` so the queue can
+    apply hash-tag validation at construction time. The abstract base
+    provides ``False`` as the default; non-cluster gateways inherit it.
 
     Concurrency
     -----------
@@ -35,8 +41,29 @@ class AbstractRedisGateway(ABC):
     ``remove_message``. Implementations must be safe for concurrent calls
     across these methods. The built-in gateway achieves this via atomic
     Lua scripts.
+
+    Server-side atomicity (Lua scripts, redis-py transactions, or a
+    single-command-per-mutation discipline) is the recommended pattern.
+    Python-level locks (``threading.Lock``, ``asyncio.Lock``) shared across
+    ``renew_message_lease`` and ``move_message`` / ``remove_message`` are an
+    anti-pattern: they serialize without giving Redis-server-side atomicity,
+    leave partial-failure orphans, and can deadlock against the heartbeat
+    lifecycle (the heartbeat is awaited from the same finally block that
+    issues the cleanup move/remove).
     """
 
+    @property
+    def is_redis_cluster(self) -> bool:
+        return False
+
+    @property
+    def max_delivery_count(self) -> int | None:
+        return None
+
+    @property
+    def dead_letter_queue(self) -> str | None:
+        return None
+
     @abstractmethod
     def publish_message(self, queue: str, message: str, dedup_key: str) -> bool:
         """Publish a message with deduplication.
@@ -48,7 +75,20 @@ class AbstractRedisGateway(ABC):
 
     @abstractmethod
     def add_message(self, queue: str, message: str) -> None:
-        """Unconditionally enqueue a message. No deduplication is performed."""
+        """Unconditionally enqueue a message. No deduplication is performed.
+
+        This library deliberately does not wrap the underlying enqueue in a
+        retry — retrying after the server may already have executed the
+        command can silently duplicate the message. The caller can still
+        retry (accepting duplicates).
+
+        Note: a client-level retry policy bypasses this guarantee. If the
+        underlying ``redis.Redis`` / ``redis.asyncio.Redis`` client was
+        constructed with ``retry=Retry(...)``, redis-py retries on
+        ``ConnectionError`` / ``TimeoutError`` below this call and may
+        duplicate. Pass ``retry=None`` (the default) when strict at-most-once
+        is required for non-deduplicated publishes.
+        """
 
     @abstractmethod
     def move_message(
@@ -131,7 +171,12 @@ class AbstractRedisGateway(ABC):
 
         Implementations MUST respect a reasonable timeout or return None
         periodically so the consumer can check for interrupts. Blocking
-        indefinitely without returning prevents graceful shutdown.
+        indefinitely without returning prevents graceful shutdown. As a
+        concrete reference, the built-in gateway uses a 5s outer wait
+        decomposed into 0.25s polling steps (see
+        ``_VISIBILITY_TIMEOUT_POLL_INTERVAL_SECONDS``); custom
+        implementations should keep the longest single block below ~5s so
+        an interrupt is observed within one polling step.
         """
 
     @abstractmethod

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/redis_message_queue/_config.py

@@ -112,13 +112,16 @@ def validate_gateway_parameters(
     if not isinstance(message_deduplication_log_ttl_seconds, int) or isinstance(
         message_deduplication_log_ttl_seconds, bool
     ):
+        bool_hint = " (use True or False, not 1/0)" if isinstance(message_deduplication_log_ttl_seconds, bool) else ""
         raise TypeError(
             f"'message_deduplication_log_ttl_seconds' must be an int, "
-            f"got {type(message_deduplication_log_ttl_seconds).__name__}"
+            f"got {type(message_deduplication_log_ttl_seconds).__name__}{bool_hint}"
         )
     if not isinstance(message_wait_interval_seconds, int) or isinstance(message_wait_interval_seconds, bool):
+        bool_hint = " (use True or False, not 1/0)" if isinstance(message_wait_interval_seconds, bool) else ""
         raise TypeError(
-            f"'message_wait_interval_seconds' must be an int, got {type(message_wait_interval_seconds).__name__}"
+            f"'message_wait_interval_seconds' must be an int, "
+            f"got {type(message_wait_interval_seconds).__name__}{bool_hint}"
         )
     if message_deduplication_log_ttl_seconds <= 0:
         raise ValueError(
@@ -130,9 +133,10 @@ def validate_gateway_parameters(
         if not isinstance(message_visibility_timeout_seconds, int) or isinstance(
             message_visibility_timeout_seconds, bool
         ):
+            bool_hint = " (use True or False, not 1/0)" if isinstance(message_visibility_timeout_seconds, bool) else ""
             raise TypeError(
                 "'message_visibility_timeout_seconds' must be an int or None, "
-                f"got {type(message_visibility_timeout_seconds).__name__}"
+                f"got {type(message_visibility_timeout_seconds).__name__}{bool_hint}"
             )
         if message_visibility_timeout_seconds <= 0:
             raise ValueError(
@@ -141,18 +145,24 @@ def validate_gateway_parameters(
             )
 
     if not isinstance(retry_budget_seconds, int) or isinstance(retry_budget_seconds, bool):
-        raise TypeError(f"'retry_budget_seconds' must be an int, got {type(retry_budget_seconds).__name__}")
+        bool_hint = " (use True or False, not 1/0)" if isinstance(retry_budget_seconds, bool) else ""
+        raise TypeError(f"'retry_budget_seconds' must be an int, got {type(retry_budget_seconds).__name__}{bool_hint}")
     if retry_budget_seconds < 0:
         raise ValueError(f"'retry_budget_seconds' must be non-negative, got {retry_budget_seconds}")
 
     if isinstance(retry_max_delay_seconds, bool) or not isinstance(retry_max_delay_seconds, (int, float)):
-        raise TypeError(f"'retry_max_delay_seconds' must be a number, got {type(retry_max_delay_seconds).__name__}")
+        bool_hint = " (use True or False, not 1/0)" if isinstance(retry_max_delay_seconds, bool) else ""
+        raise TypeError(
+            f"'retry_max_delay_seconds' must be a number, got {type(retry_max_delay_seconds).__name__}{bool_hint}"
+        )
     if not math.isfinite(retry_max_delay_seconds) or retry_max_delay_seconds <= 0:
         raise ValueError(f"'retry_max_delay_seconds' must be a finite positive number, got {retry_max_delay_seconds}")
 
     if isinstance(retry_initial_delay_seconds, bool) or not isinstance(retry_initial_delay_seconds, (int, float)):
+        bool_hint = " (use True or False, not 1/0)" if isinstance(retry_initial_delay_seconds, bool) else ""
         raise TypeError(
-            f"'retry_initial_delay_seconds' must be a number, got {type(retry_initial_delay_seconds).__name__}"
+            f"'retry_initial_delay_seconds' must be a number, "
+            f"got {type(retry_initial_delay_seconds).__name__}{bool_hint}"
         )
     if not math.isfinite(retry_initial_delay_seconds) or retry_initial_delay_seconds <= 0:
         raise ValueError(
@@ -172,15 +182,19 @@ def validate_dead_letter_parameters(
 ) -> None:
     if max_delivery_count is not None:
         if not isinstance(max_delivery_count, int) or isinstance(max_delivery_count, bool):
-            raise TypeError(f"'max_delivery_count' must be an int or None, got {type(max_delivery_count).__name__}")
+            bool_hint = " (use True or False, not 1/0)" if isinstance(max_delivery_count, bool) else ""
+            raise TypeError(
+                f"'max_delivery_count' must be an int or None, got {type(max_delivery_count).__name__}{bool_hint}"
+            )
         if max_delivery_count <= 0:
             raise ValueError(f"'max_delivery_count' must be positive, got {max_delivery_count}")
         if message_visibility_timeout_seconds is None:
             raise ValueError("'max_delivery_count' requires 'message_visibility_timeout_seconds' to be set.")
     if dead_letter_queue is not None and not isinstance(dead_letter_queue, str):
-        raise TypeError(f"'dead_letter_queue' must be a str or None, got {type(dead_letter_queue).__name__}")
+        bool_hint = " (use True or False, not 1/0)" if isinstance(dead_letter_queue, bool) else ""
+        raise TypeError(f"'dead_letter_queue' must be a str or None, got {type(dead_letter_queue).__name__}{bool_hint}")
     if isinstance(dead_letter_queue, str) and dead_letter_queue and not dead_letter_queue.strip():
-        raise ValueError("'dead_letter_queue' must be a non-empty string")
+        raise ValueError(f"'dead_letter_queue' must contain non-whitespace characters; got {dead_letter_queue!r}")
     if max_delivery_count is not None and not dead_letter_queue:
         raise ValueError("'dead_letter_queue' is required when 'max_delivery_count' is set.")
     if dead_letter_queue and max_delivery_count is None:
@@ -233,7 +247,17 @@ end
 local result = 0
 local was_set = redis.call('SET', KEYS[1], '', 'NX', 'EX', tonumber(ARGV[1]))
 if was_set then
-    redis.call('LPUSH', KEYS[2], ARGV[2])
+    -- pcall guards against LPUSH OOM after the dedup key was committed.
+    -- Without this, OOM would strand the publish: dedup says "already
+    -- published" on retry, but the message is in no queue. Compensate by
+    -- clearing the dedup key so the retry can re-attempt.
+    local ok = pcall(function()
+        redis.call('LPUSH', KEYS[2], ARGV[2])
+    end)
+    if not ok then
+        redis.pcall('DEL', KEYS[1])
+        return redis.error_reply('OOM during publish; dedup key cleared for retry')
+    end
     result = 1
 end
 

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/redis_message_queue/_queue_key_manager.py

@@ -22,13 +22,15 @@ class QueueKeyManager:
         if not isinstance(queue_name, str):
             raise TypeError(f"'name' must be a string, got {type(queue_name).__name__}")
         if not queue_name.strip():
-            raise ValueError("'name' must be a non-empty string")
+            raise ValueError(f"'name' must be a non-empty string with non-whitespace characters; got {queue_name!r}")
         if "\x00" in queue_name:
             raise ValueError("queue name must not contain null bytes")
         if not isinstance(key_separator, str):
             raise TypeError(f"'key_separator' must be a string, got {type(key_separator).__name__}")
         if not key_separator.strip():
-            raise ValueError("'key_separator' must be a non-empty string")
+            raise ValueError(
+                f"'key_separator' must be a non-empty string with non-whitespace characters; got {key_separator!r}"
+            )
         # Reject names containing the separator: ``QueueKeyManager('q').deduplication('pending')``
         # and ``QueueKeyManager('q::deduplication').pending`` would both map to
         # ``'q::deduplication::pending'`` — a string key colliding with a list key, producing

{redis_message_queue-3.1.2 → redis_message_queue-5.0.0}/redis_message_queue/_redis_cluster.py

@@ -1,6 +1,6 @@
 import re
 
-from redis.cluster import key_slot
+from redis.crc import key_slot
 
 from redis_message_queue._queue_key_manager import QueueKeyManager
 
@@ -42,4 +42,8 @@ def validate_queue_keys_for_redis_cluster(
         queue_slot = next(iter(slots))
         dead_letter_slot = _redis_cluster_key_slot(dead_letter_queue)
         if dead_letter_slot != queue_slot:
-            raise ValueError("'dead_letter_queue' must share the same Redis Cluster hash tag as the queue keys.")
+            raise ValueError(
+                "'dead_letter_queue' must share the same Redis Cluster hash tag as the queue keys."
+                " For example, both '{myqueue}::pending' and '{myqueue}::dlq' share the '{myqueue}' tag —"
+                " give your DLQ a name with the same braces."
+            )