redis-message-queue 6.0.0__tar.gz → 6.0.1__tar.gz

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 6.0.0
+Version: 6.0.1
 Summary: Python message queuing with Redis and message deduplication
 License: MIT
 License-File: LICENSE
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v6.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v6.0.1-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -587,6 +587,44 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 - **Do not switch sync and async gateway instances mid-process while claims are active.** Redis state is compatible across deploys, but each gateway instance keeps its own pending claim-recovery IDs. In-flight claim recovery state does not transfer between instances.
 - **Switching between `gateway=` and `client=` can retarget the DLQ.** The built-in `client=` path derives the DLQ from the queue name. If a custom gateway used a different `dead_letter_queue`, switching paths has the same orphaning impact as renaming the DLQ.
 
+### v5 to v6 migration
+
+v6.0.0 is a non-breaking-defaults release that adds new public APIs. v5 code continues to work; v6 adds opt-in features.
+
+**New APIs (opt in as needed):**
+
+- `max_pending_length=N` caps pending-list depth; with `pending_overload_policy="raise"` (default) producers see `QueueBackpressureError` when the cap is hit; `"block"` waits up to `pending_overload_block_timeout_seconds`; `"drop_oldest"` evicts silently, so use it only when data loss is acceptable.
+- `queue.drain(timeout=...)` (sync) and `await queue.aclose(timeout=...)` (async) are explicit graceful-shutdown hooks. They refuse new claims and recover pending claim IDs but do not cancel in-flight handlers; join or await your worker separately.
+- `on_event=callback` receives a `QueueEvent` dataclass for every publish/claim/ack/reclaim/dedup/cleanup lifecycle event. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
+
+**New constructor rejections:**
+
+- Passing a `redis.sentinel.Sentinel` manager object now raises at construction. Use `sentinel.master_for(name)` instead.
+- Passing `redis.Redis(single_connection_client=True)` to the sync built-in gateway now raises. Use a normal pooled `redis.Redis` client.
+
+**Custom gateway migration:**
+
+If you subclass `AbstractRedisGateway` and override `renew_message_lease`, add a keyword argument:
+
+```python
+def renew_message_lease(
+    self,
+    queue,
+    message,
+    lease_token,
+    *,
+    is_interrupted=None,  # NEW in v6: heartbeat passes a stop-signal observer
+) -> bool:
+    ...
+```
+
+Async gateways need the same signature on `async def`. Honor `is_interrupted.is_interrupted()` in your retry loops to stop renewing when the queue is shutting down.
+
+**Exception handling:**
+
+- Catch `QueueBackpressureError` around publish if you opt into backpressure.
+- A new base class `RedisMessageQueueError` lets you catch all library-owned exceptions in one place. Existing catches for `ValueError`, `TypeError`, `redis.RedisError`, etc. continue to work because the new subclasses preserve those bases.
+
 ### v2 to v3 migration
 
 v3.0.0 replaced the `retry_strategy: Callable` constructor parameter with `retry_budget_seconds`, `retry_max_delay_seconds`, and `retry_initial_delay_seconds`. Users with custom retry strategies should subclass `AbstractRedisGateway` instead (see [Custom gateway](#custom-gateway)).

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/README.md

@@ -1,6 +1,6 @@
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v6.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v6.0.1-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -561,6 +561,44 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 - **Do not switch sync and async gateway instances mid-process while claims are active.** Redis state is compatible across deploys, but each gateway instance keeps its own pending claim-recovery IDs. In-flight claim recovery state does not transfer between instances.
 - **Switching between `gateway=` and `client=` can retarget the DLQ.** The built-in `client=` path derives the DLQ from the queue name. If a custom gateway used a different `dead_letter_queue`, switching paths has the same orphaning impact as renaming the DLQ.
 
+### v5 to v6 migration
+
+v6.0.0 is a non-breaking-defaults release that adds new public APIs. v5 code continues to work; v6 adds opt-in features.
+
+**New APIs (opt in as needed):**
+
+- `max_pending_length=N` caps pending-list depth; with `pending_overload_policy="raise"` (default) producers see `QueueBackpressureError` when the cap is hit; `"block"` waits up to `pending_overload_block_timeout_seconds`; `"drop_oldest"` evicts silently, so use it only when data loss is acceptable.
+- `queue.drain(timeout=...)` (sync) and `await queue.aclose(timeout=...)` (async) are explicit graceful-shutdown hooks. They refuse new claims and recover pending claim IDs but do not cancel in-flight handlers; join or await your worker separately.
+- `on_event=callback` receives a `QueueEvent` dataclass for every publish/claim/ack/reclaim/dedup/cleanup lifecycle event. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
+
+**New constructor rejections:**
+
+- Passing a `redis.sentinel.Sentinel` manager object now raises at construction. Use `sentinel.master_for(name)` instead.
+- Passing `redis.Redis(single_connection_client=True)` to the sync built-in gateway now raises. Use a normal pooled `redis.Redis` client.
+
+**Custom gateway migration:**
+
+If you subclass `AbstractRedisGateway` and override `renew_message_lease`, add a keyword argument:
+
+```python
+def renew_message_lease(
+    self,
+    queue,
+    message,
+    lease_token,
+    *,
+    is_interrupted=None,  # NEW in v6: heartbeat passes a stop-signal observer
+) -> bool:
+    ...
+```
+
+Async gateways need the same signature on `async def`. Honor `is_interrupted.is_interrupted()` in your retry loops to stop renewing when the queue is shutting down.
+
+**Exception handling:**
+
+- Catch `QueueBackpressureError` around publish if you opt into backpressure.
+- A new base class `RedisMessageQueueError` lets you catch all library-owned exceptions in one place. Existing catches for `ValueError`, `TypeError`, `redis.RedisError`, etc. continue to work because the new subclasses preserve those bases.
+
 ### v2 to v3 migration
 
 v3.0.0 replaced the `retry_strategy: Callable` constructor parameter with `retry_budget_seconds`, `retry_max_delay_seconds`, and `retry_initial_delay_seconds`. Users with custom retry strategies should subclass `AbstractRedisGateway` instead (see [Custom gateway](#custom-gateway)).

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "redis-message-queue"
-version = "6.0.0"
+version = "6.0.1"
 description = "Python message queuing with Redis and message deduplication"
 authors = ["Elijas <4084885+Elijas@users.noreply.github.com>"]
 readme = "README.md"

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/redis_message_queue/_config.py

@@ -14,7 +14,7 @@ from tenacity import (
     wait_exponential_jitter,
 )
 
-from redis_message_queue._exceptions import ConfigurationError
+from redis_message_queue._exceptions import ConfigurationError, RetryBudgetExhaustedError
 from redis_message_queue.interrupt_handler._interface import (
     BaseGracefulInterruptHandler,
 )
@@ -91,6 +91,15 @@ def _noop_retry(func):
     return func
 
 
+def _raise_retry_budget_exhausted(retry_state: RetryCallState) -> typing.NoReturn:
+    exc = retry_state.outcome.exception() if retry_state.outcome is not None else None
+    if exc is None:
+        raise RetryBudgetExhaustedError("Redis retry budget exhausted")
+    raise RetryBudgetExhaustedError(
+        f"Redis retry budget exhausted after {retry_state.attempt_number} attempts"
+    ) from exc
+
+
 def build_retry_strategy(
     *,
     retry_budget_seconds: int,
@@ -113,6 +122,7 @@ def build_retry_strategy(
             get_parent_retry=lambda: retry_if_exception(is_redis_retryable_exception),
         ),
         after=after_log(logger, logging.WARNING),
+        retry_error_callback=_raise_retry_budget_exhausted,
         reraise=True,
     )
 

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/redis_message_queue/_redis_gateway.py

@@ -35,7 +35,11 @@ from redis_message_queue._config import (
     validate_pending_backpressure_parameters,
 )
 from redis_message_queue._event import EventOperation, EventOutcome
-from redis_message_queue._exceptions import QueueBackpressureError, wrap_lua_response_error
+from redis_message_queue._exceptions import (
+    QueueBackpressureError,
+    RetryBudgetExhaustedError,
+    wrap_lua_response_error,
+)
 from redis_message_queue._stored_message import (
     ClaimedMessage,
     MessageData,
@@ -555,7 +559,9 @@ class RedisGateway(AbstractRedisGateway):
                             claim_id=claim_id,
                             exception_type=type(retry_exc).__name__,
                         )
-                        raise
+                        raise RetryBudgetExhaustedError(
+                            "Redis retry budget exhausted during message claim"
+                        ) from retry_exc
                     except BaseException:
                         pending_claim_id_to_share = claim_id
                         raise
@@ -624,7 +630,9 @@ class RedisGateway(AbstractRedisGateway):
                             claim_id=claim_id,
                             exception_type=type(last_retryable_exception).__name__,
                         )
-                        raise last_retryable_exception
+                        raise RetryBudgetExhaustedError(
+                            "Redis retry budget exhausted during message claim"
+                        ) from last_retryable_exception
                     return None
                 time.sleep(min(_VISIBILITY_TIMEOUT_POLL_INTERVAL_SECONDS, remaining))
         finally:
@@ -796,13 +804,15 @@ class RedisGateway(AbstractRedisGateway):
         try:
             self._redis_client.delete(claim_result_key)
         except Exception:
-            logger.debug("Failed to delete claim result key %s", claim_result_key, exc_info=True)
+            # Claim-result keys have bounded TTLs; this cleanup is intentionally best-effort.
+            logger.warning("Failed to delete claim result key %s", claim_result_key, exc_info=True)
 
     def _delete_claim_result_ref(self, claim_result_refs_key: str, lease_token: str) -> None:
         try:
             self._redis_client.hdel(claim_result_refs_key, lease_token)
         except Exception:
-            logger.debug(
+            # Claim-result refs have bounded TTLs; this cleanup is intentionally best-effort.
+            logger.warning(
                 "Failed to delete claim result reference %s[%s]",
                 claim_result_refs_key,
                 lease_token,

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/redis_message_queue/asyncio/_redis_gateway.py

@@ -34,7 +34,11 @@ from redis_message_queue._config import (
     validate_pending_backpressure_parameters,
 )
 from redis_message_queue._event import EventOperation, EventOutcome
-from redis_message_queue._exceptions import QueueBackpressureError, wrap_lua_response_error
+from redis_message_queue._exceptions import (
+    QueueBackpressureError,
+    RetryBudgetExhaustedError,
+    wrap_lua_response_error,
+)
 from redis_message_queue._stored_message import (
     ClaimedMessage,
     MessageData,
@@ -551,7 +555,9 @@ class RedisGateway(AbstractRedisGateway):
                             claim_id=claim_id,
                             exception_type=type(retry_exc).__name__,
                         )
-                        raise
+                        raise RetryBudgetExhaustedError(
+                            "Redis retry budget exhausted during message claim"
+                        ) from retry_exc
                     except BaseException:
                         pending_claim_id_to_share = claim_id
                         raise
@@ -621,7 +627,9 @@ class RedisGateway(AbstractRedisGateway):
                             claim_id=claim_id,
                             exception_type=type(last_retryable_exception).__name__,
                         )
-                        raise last_retryable_exception
+                        raise RetryBudgetExhaustedError(
+                            "Redis retry budget exhausted during message claim"
+                        ) from last_retryable_exception
                     return None
                 await asyncio.sleep(min(_VISIBILITY_TIMEOUT_POLL_INTERVAL_SECONDS, remaining))
         finally:
@@ -793,13 +801,15 @@ class RedisGateway(AbstractRedisGateway):
         try:
             await self._redis_client.delete(claim_result_key)
         except Exception:
-            logger.debug("Failed to delete claim result key %s", claim_result_key, exc_info=True)
+            # Claim-result keys have bounded TTLs; this cleanup is intentionally best-effort.
+            logger.warning("Failed to delete claim result key %s", claim_result_key, exc_info=True)
 
     async def _delete_claim_result_ref(self, claim_result_refs_key: str, lease_token: str) -> None:
         try:
             await self._redis_client.hdel(claim_result_refs_key, lease_token)
         except Exception:
-            logger.debug(
+            # Claim-result refs have bounded TTLs; this cleanup is intentionally best-effort.
+            logger.warning(
                 "Failed to delete claim result reference %s[%s]",
                 claim_result_refs_key,
                 lease_token,

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/redis_message_queue/asyncio/redis_message_queue.py

@@ -18,7 +18,7 @@ from redis_message_queue._config import (
     validate_pending_backpressure_parameters,
 )
 from redis_message_queue._event import EventOperation, EventOutcome, QueueEvent
-from redis_message_queue._exceptions import ConfigurationError, GatewayContractError
+from redis_message_queue._exceptions import CleanupFailedError, ConfigurationError, GatewayContractError
 from redis_message_queue._queue_key_manager import QueueKeyManager
 from redis_message_queue._redis_cluster import validate_queue_keys_for_redis_cluster
 from redis_message_queue._stored_message import (
@@ -573,6 +573,17 @@ class RedisMessageQueue:
                 " Expected a function that takes the message (str | dict) and returns a str (or an awaitable thereof)."
                 " Example: get_deduplication_key=lambda msg: msg['user_id']"
             )
+        if (
+            max_pending_length is not None
+            and pending_overload_policy == "drop_oldest"
+            and (deduplication or get_deduplication_key_was_configured)
+        ):
+            raise ConfigurationError(
+                "'pending_overload_policy=drop_oldest' cannot be used with deduplication because dropped messages "
+                "leave their deduplication keys in Redis, causing future publishes of the same payload to be "
+                "silently suppressed. Use 'raise' or 'block' for deduplicated queues, or disable deduplication if "
+                "'drop_oldest' is required."
+            )
         if not deduplication and get_deduplication_key_was_configured:
             raise ConfigurationError("'get_deduplication_key' cannot be provided when 'deduplication' is disabled.")
         if on_heartbeat_failure is not None and not callable(on_heartbeat_failure):
@@ -725,11 +736,13 @@ class RedisMessageQueue:
             await result
         except Exception as exc:
             logger.exception("on_event callback raised an exception")
-            warnings.warn(
-                f"on_event callback raised {type(exc).__name__}",
-                RuntimeWarning,
-                stacklevel=2,
-            )
+            with warnings.catch_warnings():
+                warnings.simplefilter("always", RuntimeWarning)
+                warnings.warn(
+                    f"on_event callback raised {type(exc).__name__}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
 
     async def publish(self, message: str | dict) -> bool:
         """Publish a message.
@@ -926,6 +939,7 @@ class RedisMessageQueue:
                     )
                     warnings.warn(_STALE_LEASE_NACK_WARNING, RuntimeWarning, stacklevel=2)
             except BaseException as cleanup_exc:
+                # The handler exception is the user-visible failure; cleanup failure is secondary.
                 logger.exception("Failed to clean up message from processing queue")
                 await self._emit_event(
                     "cleanup_failed",
@@ -952,7 +966,7 @@ class RedisMessageQueue:
                 cleanup_operation = self._remove_processed_message(stored_message, lease_token)
             try:
                 applied = await _await_preserving_cancellation(cleanup_operation)
-            except BaseException as cleanup_exc:
+            except Exception as cleanup_exc:
                 await self._emit_event(
                     "cleanup_failed",
                     "failure",
@@ -961,7 +975,7 @@ class RedisMessageQueue:
                     exception_type=type(cleanup_exc).__name__,
                     duration_ms=_duration_ms(cleanup_started_at),
                 )
-                raise
+                raise CleanupFailedError("Cleanup after successful processing failed") from cleanup_exc
             if self._enable_completed_queue:
                 await self._emit_event(
                     "completed",
@@ -1083,7 +1097,7 @@ class RedisMessageQueue:
             return True
         loop = asyncio.get_running_loop()
         deadline_monotonic = None if timeout is None else (loop.time() + float(timeout))
-        return await drainer(self.key.processing, deadline_monotonic=deadline_monotonic)
+        return await _await_preserving_cancellation(drainer(self.key.processing, deadline_monotonic=deadline_monotonic))
 
     def _build_lease_heartbeat(
         self,

{redis_message_queue-6.0.0 → redis_message_queue-6.0.1}/redis_message_queue/redis_message_queue.py

@@ -19,7 +19,7 @@ from redis_message_queue._config import (
     validate_pending_backpressure_parameters,
 )
 from redis_message_queue._event import EventOperation, EventOutcome, QueueEvent
-from redis_message_queue._exceptions import ConfigurationError, GatewayContractError
+from redis_message_queue._exceptions import CleanupFailedError, ConfigurationError, GatewayContractError
 from redis_message_queue._queue_key_manager import QueueKeyManager
 from redis_message_queue._redis_cluster import validate_queue_keys_for_redis_cluster
 from redis_message_queue._redis_gateway import RedisGateway
@@ -518,6 +518,17 @@ class RedisMessageQueue:
                 "'get_deduplication_key' is an async callable; "
                 "use the async RedisMessageQueue from redis_message_queue.asyncio instead"
             )
+        if (
+            max_pending_length is not None
+            and pending_overload_policy == "drop_oldest"
+            and (deduplication or get_deduplication_key_was_configured)
+        ):
+            raise ConfigurationError(
+                "'pending_overload_policy=drop_oldest' cannot be used with deduplication because dropped messages "
+                "leave their deduplication keys in Redis, causing future publishes of the same payload to be "
+                "silently suppressed. Use 'raise' or 'block' for deduplicated queues, or disable deduplication if "
+                "'drop_oldest' is required."
+            )
         if not deduplication and get_deduplication_key_was_configured:
             raise ConfigurationError("'get_deduplication_key' cannot be provided when 'deduplication' is disabled.")
         if on_heartbeat_failure is not None and not callable(on_heartbeat_failure):
@@ -682,11 +693,13 @@ class RedisMessageQueue:
                 )
         except Exception as exc:
             logger.exception("on_event callback raised an exception")
-            warnings.warn(
-                f"on_event callback raised {type(exc).__name__}",
-                RuntimeWarning,
-                stacklevel=2,
-            )
+            with warnings.catch_warnings():
+                warnings.simplefilter("always", RuntimeWarning)
+                warnings.warn(
+                    f"on_event callback raised {type(exc).__name__}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
 
     def publish(self, message: str | dict) -> bool:
         """Publish a message.
@@ -888,6 +901,7 @@ class RedisMessageQueue:
                     )
                     warnings.warn(_STALE_LEASE_NACK_WARNING, RuntimeWarning, stacklevel=2)
             except BaseException as cleanup_exc:
+                # The handler exception is the user-visible failure; cleanup failure is secondary.
                 logger.exception("Failed to clean up message from processing queue")
                 self._emit_event(
                     "cleanup_failed",
@@ -913,7 +927,7 @@ class RedisMessageQueue:
                     applied = self._move_processed_message(self.key.completed, stored_message, lease_token)
                 else:
                     applied = self._remove_processed_message(stored_message, lease_token)
-            except BaseException as cleanup_exc:
+            except Exception as cleanup_exc:
                 self._emit_event(
                     "cleanup_failed",
                     "failure",
@@ -922,7 +936,7 @@ class RedisMessageQueue:
                     exception_type=type(cleanup_exc).__name__,
                     duration_ms=_duration_ms(cleanup_started_at),
                 )
-                raise
+                raise CleanupFailedError("Cleanup after successful processing failed") from cleanup_exc
             if self._enable_completed_queue:
                 self._emit_event(
                     "completed",
@@ -1047,6 +1061,13 @@ class RedisMessageQueue:
         deadline_monotonic = None if timeout is None else (time.monotonic() + float(timeout))
         return drainer(self.key.processing, deadline_monotonic=deadline_monotonic)
 
+    def close(self, timeout: float | None = None) -> bool:
+        """Alias of :meth:`drain` for consistency with redis-py naming.
+
+        See :meth:`drain` for full semantics.
+        """
+        return self.drain(timeout)
+
     def _build_lease_heartbeat(
         self,
         stored_message: MessageData,