redis-message-queue 3.1.1__tar.gz → 3.1.2__tar.gz

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 3.1.1
+Version: 3.1.2
 Summary: Python message queuing with Redis and message deduplication
 License: MIT
 License-File: LICENSE
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v3.1.1-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v3.1.2-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -310,12 +310,28 @@ await client.aclose()
 - **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
 - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
 - **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
+- **Default dedup key is the full message.** Without a custom `get_deduplication_key`, the entire serialized message becomes a Redis key name for dedup tracking. For large messages (>1KB), provide a custom key function to avoid excessive Redis memory usage.
 - **Cluster detection uses `isinstance(client, RedisCluster)`.** Wrapped or instrumented cluster clients that delegate without inheriting will bypass hash-tag validation. Custom gateways should set `is_redis_cluster = True` explicitly.
 - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
+- **Non-ASCII payloads use ~2x storage.** The default `ensure_ascii=True` in JSON serialization encodes non-ASCII characters as `\uXXXX` escape sequences. This is a deliberate compatibility choice.
 - **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
 
 For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
 
+## Upgrading
+
+### Configuration changes on live queues
+
+> **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
+
+- **Do not change `key_separator` on a live queue.** All existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing separators.
+- **Do not switch from no-VT to VT with messages in processing.** Messages claimed by non-VT consumers have no lease deadline entries. VT-enabled consumers cannot reclaim them. Drain the processing queue first.
+- **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
+
+### v2 to v3 migration
+
+v3.0.0 replaced the `retry_strategy: Callable` constructor parameter with `retry_budget_seconds`, `retry_max_delay_seconds`, and `retry_initial_delay_seconds`. Users with custom retry strategies should subclass `AbstractRedisGateway` instead (see [Custom gateway](#custom-gateway)).
+
 ## Running locally
 
 You'll need a Redis server:

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/README.md

@@ -1,6 +1,6 @@
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v3.1.1-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v3.1.2-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -284,12 +284,28 @@ await client.aclose()
 - **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
 - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
 - **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
+- **Default dedup key is the full message.** Without a custom `get_deduplication_key`, the entire serialized message becomes a Redis key name for dedup tracking. For large messages (>1KB), provide a custom key function to avoid excessive Redis memory usage.
 - **Cluster detection uses `isinstance(client, RedisCluster)`.** Wrapped or instrumented cluster clients that delegate without inheriting will bypass hash-tag validation. Custom gateways should set `is_redis_cluster = True` explicitly.
 - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
+- **Non-ASCII payloads use ~2x storage.** The default `ensure_ascii=True` in JSON serialization encodes non-ASCII characters as `\uXXXX` escape sequences. This is a deliberate compatibility choice.
 - **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
 
 For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
 
+## Upgrading
+
+### Configuration changes on live queues
+
+> **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
+
+- **Do not change `key_separator` on a live queue.** All existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing separators.
+- **Do not switch from no-VT to VT with messages in processing.** Messages claimed by non-VT consumers have no lease deadline entries. VT-enabled consumers cannot reclaim them. Drain the processing queue first.
+- **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
+
+### v2 to v3 migration
+
+v3.0.0 replaced the `retry_strategy: Callable` constructor parameter with `retry_budget_seconds`, `retry_max_delay_seconds`, and `retry_initial_delay_seconds`. Users with custom retry strategies should subclass `AbstractRedisGateway` instead (see [Custom gateway](#custom-gateway)).
+
 ## Running locally
 
 You'll need a Redis server:

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "redis-message-queue"
-version = "3.1.1"
+version = "3.1.2"
 description = "Python message queuing with Redis and message deduplication"
 authors = ["Elijas <4084885+Elijas@users.noreply.github.com>"]
 readme = "README.md"

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/__init__.py

@@ -1,4 +1,5 @@
 from redis_message_queue._abstract_redis_gateway import AbstractRedisGateway
+from redis_message_queue._redis_gateway import RedisGateway
 from redis_message_queue._stored_message import ClaimedMessage, MessageData
 from redis_message_queue.interrupt_handler import (
     BaseGracefulInterruptHandler,
@@ -8,6 +9,7 @@ from redis_message_queue.redis_message_queue import RedisMessageQueue
 
 __all__ = [
     "RedisMessageQueue",
+    "RedisGateway",
     "AbstractRedisGateway",
     "ClaimedMessage",
     "MessageData",

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/_abstract_redis_gateway.py

@@ -128,6 +128,10 @@ class AbstractRedisGateway(ABC):
         use leases.
 
         Return None if no message was available (e.g. timeout or interrupt).
+
+        Implementations MUST respect a reasonable timeout or return None
+        periodically so the consumer can check for interrupts. Blocking
+        indefinitely without returning prevents graceful shutdown.
         """
 
     @abstractmethod

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/_config.py

@@ -36,13 +36,13 @@ def is_redis_retryable_exception(exception):
             ),
         )
 
-    # 2. Explicit retryable exceptions
+    # 2. Explicit retryable exceptions (BusyLoadingError is a ConnectionError
+    #    subclass, so it is already handled by branch 1 above)
     return isinstance(
         exception,
         (
             # Network/availability issues
             redis.exceptions.TimeoutError,  # Socket or server-side timeout
-            redis.exceptions.BusyLoadingError,  # Server loading data
             # Cluster transient failures
             redis.exceptions.ClusterDownError,  # Covers ClusterDown + MasterDown
             redis.exceptions.TryAgainError,  # Cluster state requires retry
@@ -275,6 +275,7 @@ if cached_result then
     return tonumber(cached_result)
 end
 
+redis.call('LPUSH', KEYS[2], ARGV[2])
 local removed = redis.call('LREM', KEYS[1], 1, ARGV[1])
 if removed == 1 then
     local claim_id = redis.call('HGET', KEYS[4], ARGV[1])
@@ -282,7 +283,8 @@ if removed == 1 then
         redis.call('HDEL', KEYS[3], claim_id)
         redis.call('HDEL', KEYS[4], ARGV[1])
     end
-    redis.call('LPUSH', KEYS[2], ARGV[2])
+else
+    redis.call('LREM', KEYS[2], 1, ARGV[2])
 end
 
 redis.call('SET', KEYS[5], tostring(removed), 'PX', tonumber(ARGV[3]))
@@ -455,6 +457,9 @@ local function redis_message_queue_decode_claim(cached_claim)
     return nil
 end
 
+local time = redis.call('TIME')
+local now_ms = tonumber(time[1]) * 1000 + math.floor(tonumber(time[2]) / 1000)
+
 -- Cache replay paths below return the ORIGINAL claim (same lease_token) even if
 -- the lease deadline has passed in wall-clock time. Safe because ack is gated by
 -- the server-side HGET lease_tokens check in MOVE/REMOVE_WITH_LEASE_TOKEN: if
@@ -469,6 +474,7 @@ if cached_claim then
         redis.call('HSET', KEYS[10], ARGV[4], cached_claim)
         redis.call('HSET', KEYS[11], claim[2], ARGV[4])
         redis.call('HSET', KEYS[9], claim[2], KEYS[8])
+        redis.call('ZADD', KEYS[3], now_ms + tonumber(ARGV[1]), claim[1])
         return {claim[1], claim[2]}
     end
     redis.call('DEL', KEYS[8])
@@ -481,14 +487,12 @@ if cached_recovery then
         redis.call('SET', KEYS[8], cached_recovery, 'PX', tonumber(ARGV[3]))
         redis.call('HSET', KEYS[11], claim[2], ARGV[4])
         redis.call('HSET', KEYS[9], claim[2], KEYS[8])
+        redis.call('ZADD', KEYS[3], now_ms + tonumber(ARGV[1]), claim[1])
         return {claim[1], claim[2]}
     end
     redis.call('HDEL', KEYS[10], ARGV[4])
 end
 
-local time = redis.call('TIME')
-local now_ms = tonumber(time[1]) * 1000 + math.floor(tonumber(time[2]) / 1000)
-
 -- Cap at 100 to bound Lua execution time (Redis blocks during scripts).
 -- With a single consumer polling at default interval, 1000 expired leases drain in ~2.5s.
 local expired = redis.call('ZRANGEBYSCORE', KEYS[3], '-inf', now_ms, 'LIMIT', 0, 100)
@@ -518,15 +522,24 @@ if #to_requeue > 0 then
 end
 
 local function store_claim_and_return(stored)
-    local lease_token = tostring(redis.call('INCR', KEYS[5]))
-    local claim_payload = cjson.encode({stored, lease_token})
-    redis.call('ZADD', KEYS[3], now_ms + tonumber(ARGV[1]), stored)
-    redis.call('HSET', KEYS[4], stored, lease_token)
-    redis.call('SET', KEYS[8], claim_payload, 'PX', tonumber(ARGV[3]))
-    redis.call('HSET', KEYS[9], lease_token, KEYS[8])
-    redis.call('HSET', KEYS[10], ARGV[4], claim_payload)
-    redis.call('HSET', KEYS[11], lease_token, ARGV[4])
-    return {stored, lease_token}
+    -- pcall guards against OOM mid-write: compensate by returning message to pending
+    local ok, result = pcall(function()
+        local lease_token = tostring(redis.call('INCR', KEYS[5]))
+        local claim_payload = cjson.encode({stored, lease_token})
+        redis.call('ZADD', KEYS[3], now_ms + tonumber(ARGV[1]), stored)
+        redis.call('HSET', KEYS[4], stored, lease_token)
+        redis.call('SET', KEYS[8], claim_payload, 'PX', tonumber(ARGV[3]))
+        redis.call('HSET', KEYS[9], lease_token, KEYS[8])
+        redis.call('HSET', KEYS[10], ARGV[4], claim_payload)
+        redis.call('HSET', KEYS[11], lease_token, ARGV[4])
+        return {stored, lease_token}
+    end)
+    if not ok then
+        redis.call('LREM', KEYS[2], 1, stored)
+        redis.pcall('RPUSH', KEYS[1], stored)
+        return false
+    end
+    return result
 end
 
 local claim_attempts = 0
@@ -705,6 +718,7 @@ end
 
 -- See REMOVE_MESSAGE_WITH_LEASE_TOKEN_LUA_SCRIPT for the bounded-leak rationale
 -- on the removed == 0 branch (externally-removed message + valid lease token).
+redis.call('LPUSH', KEYS[2], ARGV[2])
 local removed = redis.call('LREM', KEYS[1], 1, ARGV[1])
 if removed == 1 then
     redis.call('ZREM', KEYS[3], ARGV[1])
@@ -720,8 +734,9 @@ if removed == 1 then
         redis.call('HDEL', KEYS[8], ARGV[3])
     end
     redis.call('HDEL', KEYS[5], ARGV[1])
-    redis.call('LPUSH', KEYS[2], ARGV[2])
     redis.call('SET', KEYS[9], '1', 'PX', tonumber(ARGV[4]))
+else
+    redis.call('LREM', KEYS[2], 1, ARGV[2])
 end
 
 return removed

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/_queue_key_manager.py

@@ -23,6 +23,8 @@ class QueueKeyManager:
             raise TypeError(f"'name' must be a string, got {type(queue_name).__name__}")
         if not queue_name.strip():
             raise ValueError("'name' must be a non-empty string")
+        if "\x00" in queue_name:
+            raise ValueError("queue name must not contain null bytes")
         if not isinstance(key_separator, str):
             raise TypeError(f"'key_separator' must be a string, got {type(key_separator).__name__}")
         if not key_separator.strip():

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/_redis_gateway.py

@@ -88,7 +88,7 @@ class RedisGateway(AbstractRedisGateway):
         if isinstance(redis_client, redis.asyncio.Redis):
             raise TypeError(
                 "'redis_client' is an async Redis client (redis.asyncio.Redis); "
-                "use the async RedisGateway from redis_message_queue.asyncio instead"
+                "use the async RedisMessageQueue from redis_message_queue.asyncio instead"
             )
         if isinstance(redis_client, (redis.client.Pipeline, redis.asyncio.client.Pipeline)):
             raise TypeError(
@@ -380,6 +380,7 @@ class RedisGateway(AbstractRedisGateway):
                     claimed_message = claim_message(from_queue, to_queue, claim_id)
                 except Exception as exc:
                     if not is_redis_retryable_exception(exc):
+                        pending_claim_id_to_share = claim_id
                         raise
                     claim_may_need_recovery = True
                     logger.warning(non_blocking_retry_log, type(exc).__name__)
@@ -414,8 +415,7 @@ class RedisGateway(AbstractRedisGateway):
                     claimed_message = claim_message(from_queue, to_queue, claim_id)
                 except Exception as exc:
                     if not is_redis_retryable_exception(exc):
-                        if claim_may_need_recovery:
-                            pending_claim_id_to_share = claim_id
+                        pending_claim_id_to_share = claim_id
                         raise
                     claim_may_need_recovery = True
                     logger.warning(polling_retry_log, type(exc).__name__)

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/asyncio/__init__.py

@@ -1,5 +1,6 @@
 from redis_message_queue._stored_message import ClaimedMessage, MessageData
 from redis_message_queue.asyncio._abstract_redis_gateway import AbstractRedisGateway
+from redis_message_queue.asyncio._redis_gateway import RedisGateway
 from redis_message_queue.asyncio.redis_message_queue import RedisMessageQueue
 
-__all__ = ["RedisMessageQueue", "AbstractRedisGateway", "ClaimedMessage", "MessageData"]
+__all__ = ["RedisMessageQueue", "RedisGateway", "AbstractRedisGateway", "ClaimedMessage", "MessageData"]

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/asyncio/_abstract_redis_gateway.py

@@ -128,6 +128,10 @@ class AbstractRedisGateway(ABC):
         use leases.
 
         Return None if no message was available (e.g. timeout or interrupt).
+
+        Implementations MUST respect a reasonable timeout or return None
+        periodically so the consumer can check for interrupts. Blocking
+        indefinitely without returning prevents graceful shutdown.
         """
 
     @abstractmethod

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/asyncio/_redis_gateway.py

@@ -88,7 +88,7 @@ class RedisGateway(AbstractRedisGateway):
         if isinstance(redis_client, redis.Redis) and not isinstance(redis_client, redis.asyncio.Redis):
             raise TypeError(
                 "'redis_client' is a sync Redis client (redis.Redis); "
-                "use the sync RedisGateway from redis_message_queue instead"
+                "use the sync RedisMessageQueue from redis_message_queue instead"
             )
         if isinstance(redis_client, (redis.client.Pipeline, redis.asyncio.client.Pipeline)):
             raise TypeError(
@@ -380,6 +380,7 @@ class RedisGateway(AbstractRedisGateway):
                     claimed_message = await claim_message(from_queue, to_queue, claim_id)
                 except Exception as exc:
                     if not is_redis_retryable_exception(exc):
+                        pending_claim_id_to_share = claim_id
                         raise
                     claim_may_need_recovery = True
                     logger.warning(non_blocking_retry_log, type(exc).__name__)
@@ -415,8 +416,7 @@ class RedisGateway(AbstractRedisGateway):
                     claimed_message = await claim_message(from_queue, to_queue, claim_id)
                 except Exception as exc:
                     if not is_redis_retryable_exception(exc):
-                        if claim_may_need_recovery:
-                            pending_claim_id_to_share = claim_id
+                        pending_claim_id_to_share = claim_id
                         raise
                     claim_may_need_recovery = True
                     logger.warning(polling_retry_log, type(exc).__name__)

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/asyncio/redis_message_queue.py

@@ -327,7 +327,7 @@ class RedisMessageQueue:
         max_failed_length: int | None = None,
         max_delivery_count: int | None = None,
         key_separator: str = "::",
-        get_deduplication_key: Optional[Callable] = None,
+        get_deduplication_key: Optional[Callable[[str | dict], str]] = None,
         interrupt: BaseGracefulInterruptHandler | None = None,
         on_heartbeat_failure: Callable[[], Awaitable[None] | None] | None = None,
     ):

{redis_message_queue-3.1.1 → redis_message_queue-3.1.2}/redis_message_queue/redis_message_queue.py

@@ -270,7 +270,7 @@ class RedisMessageQueue:
         max_failed_length: int | None = None,
         max_delivery_count: int | None = None,
         key_separator: str = "::",
-        get_deduplication_key: Optional[Callable] = None,
+        get_deduplication_key: Optional[Callable[[str | dict], str]] = None,
         interrupt: BaseGracefulInterruptHandler | None = None,
         on_heartbeat_failure: Callable[[], None] | None = None,
     ):
@@ -314,7 +314,7 @@ class RedisMessageQueue:
                 )
         if get_deduplication_key is not None and not callable(get_deduplication_key):
             raise TypeError(f"'get_deduplication_key' must be callable, got {type(get_deduplication_key).__name__}")
-        if get_deduplication_key is not None and inspect.iscoroutinefunction(get_deduplication_key):
+        if get_deduplication_key is not None and is_async_callable(get_deduplication_key):
             raise TypeError(
                 "'get_deduplication_key' is an async callable; "
                 "use the async RedisMessageQueue from redis_message_queue.asyncio instead"