redis-message-queue 8.2.1__tar.gz → 8.2.3__tar.gz

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/.gitignore

@@ -90,7 +90,7 @@ ipython_config.py
 # pyenv
 #   For a library or package, you might want to ignore these files since the code is
 #   intended to run in multiple environments; otherwise, check them in:
-# .python-version
+.python-version
 
 # pipenv
 #   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
@@ -141,6 +141,15 @@ venv.bak/
 .dmypy.json
 dmypy.json
 
+# Ruff
+.ruff_cache/
+
+# Coverage tooling output
+lcov.info
+
+# PyPI upload config — credentials, never commit
+.pypirc
+
 # Pyre type checker
 .pyre/
 

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 8.2.1
+Version: 8.2.3
 Summary: Python message queuing with Redis and message deduplication
 Project-URL: Homepage, https://github.com/Elijas/redis-message-queue
 Project-URL: Repository, https://github.com/Elijas/redis-message-queue
@@ -34,7 +34,7 @@ Description-Content-Type: text/markdown
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=8.2.1,<9.0.0"
+pip install "redis-message-queue>=8.2.3,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
@@ -45,6 +45,7 @@ Redis must be running locally first: use `redis-server` or
 `docker run -p 6379:6379 redis:7`.
 
 ```python
+import json
 from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
@@ -58,7 +59,8 @@ queue = RedisMessageQueue(
 queue.publish({"id": "msg-1", "text": "hello"})
 with queue.process_message() as message:
     if message is not None:
-        print(f"got {message['text']}")
+        payload = json.loads(message)
+        print(f"got {payload['text']}")
 # Expected output: got hello
 ```
 
@@ -69,13 +71,15 @@ with queue.process_message() as message:
 > synchronous. If your handler is `async def`, returns a coroutine, or returns
 > any other awaitable, use `redis_message_queue.asyncio.RedisMessageQueue`.
 > The sync context manager does not inspect the handler's return value; an
-> unawaited coroutine can be dropped while the message is acked. An ergonomic
-> callback API that detects this is planned for v8.1.
+> unawaited coroutine can be dropped while the message is acked. For sync
+> callback-style handlers, use `process_message_callback(handler)`: it checks
+> for awaitable returns before acking and raises `TypeError` if one is returned.
 
 ### Async quickstart
 
 ```python
 import asyncio
+import json
 from redis.asyncio import Redis
 from redis_message_queue.asyncio import RedisMessageQueue
 
@@ -89,7 +93,8 @@ async def main():
     )
     await queue.publish({"id": "msg-1", "text": "hello"})
     async with queue.process_message() as message:
-        print(f"got {message['text']}")
+        payload = json.loads(message)
+        print(f"got {payload['text']}")
     await client.aclose()
 
 asyncio.run(main())  # Expected output: got hello
@@ -225,7 +230,9 @@ so concurrent publishers cannot race above the configured cap. Overload policies
 - `drop_oldest` removes the oldest pending message (`RPOP`) before enqueueing the
   new message. This is silent data loss by design; deduplication markers for
   dropped messages are not removed, so a dropped duplicate may still be
-  suppressed until its dedup TTL expires.
+  suppressed until its dedup TTL expires. The current event contract emits
+  `publish/success` for the new message, but no separate `on_event` signal for
+  the dropped message.
 - `block` retries the atomic check until space opens or
   `pending_overload_block_timeout_seconds` elapses (default: 1.0), then raises
   `QueueBackpressureError`.
@@ -497,6 +504,14 @@ gateway = RedisGateway(
 queue = RedisMessageQueue("q", gateway=gateway)
 ```
 
+When `gateway=` is supplied, queue-level constructor defaults are not copied
+into the gateway. For example, `RedisMessageQueue(..., gateway=gateway)`
+leaves visibility timeout and dead-letter routing disabled unless
+`message_visibility_timeout_seconds` and `max_delivery_count` are configured on
+the gateway itself. Passing the queue-level default values
+`visibility_timeout_seconds=300` or `max_delivery_count=10` with `gateway=`
+does not transfer those settings to the gateway.
+
 The retry knobs configure an internal `tenacity` strategy: exponential
 backoff with jitter, retry on transient Redis errors only, capped at
 `retry_budget_seconds`. The budget is monotonic elapsed time from the first attempt (including attempt duration), not inter-attempt delay; it is unaffected by Python-host NTP jumps. A single attempt that takes longer than the budget results in zero retries. Setting `retry_budget_seconds=0` disables retry
@@ -729,8 +744,8 @@ queue = RedisMessageQueue("jobs", client=client, on_event=on_event)
 ```
 
 Events cover publish, dedup hits, claim/empty polls, reclaim, ack/nack,
-completed/failed cleanup, DLQ moves, heartbeat renewal, stale leases, cleanup
-and trim failures, and retry attempts. Callback exceptions are logged and
+completed/failed cleanup, DLQ moves, heartbeat renewal, stale leases, drain,
+cleanup and trim failures, and retry attempts. Callback exceptions are logged and
 reported with `RuntimeWarning`, but never propagate into queue operations.
 `on_event` is telemetry only: use it for metrics, tracing, and logging, not for
 sagas, follow-up writes, billing callbacks, or other correctness-critical
@@ -823,6 +838,23 @@ Pre-commit and mid-flight exceptions:
   despite the exception. Treat them as "operation did not succeed from the
   caller's perspective", not "Redis did not commit".
 
+#### Drain events
+
+`drain()` and `close()` on the sync queue, and `drain()` and `aclose()` on the
+async queue, emit `drain` events:
+
+- `drain/start` when the queue-local drain flag is set.
+- `drain/success` when pending claim IDs were recovered or no gateway drain
+  hook is present.
+- `drain/skipped` when the queue was already drained and the cached successful
+  result is returned.
+- `drain/failure` when pending claim recovery times out or otherwise leaves
+  unresolved claim IDs.
+
+Drain events use `timeout_seconds` for the caller-supplied timeout,
+`pending_claim_ids` for the number of unresolved local claim IDs when known,
+and `exception_type` / `error` on failure.
+
 #### Intentionally silent paths
 
 The following operations have no `on_event` surface by design:
@@ -837,9 +869,11 @@ The following operations have no `on_event` surface by design:
   it back to pending, and returns `false`. Python translates that into
   `claim_empty/skipped`, the same shape as an empty poll. This is intentional
   fail-safe behavior; the message is not lost.
-- **`drain()` / `close()` / `aclose()` lifecycle:** explicit shutdown
-  operations do not emit lifecycle events. Pending-claim-drain recovery work
-  counts as `claim_reclaim` events when reached.
+- **`drop_oldest` evictions:** when publish backpressure uses
+  `pending_overload_policy="drop_oldest"`, the oldest pending message is
+  discarded before the new message is enqueued. The successful enqueue emits
+  `publish/success`, but there is no separate drop event for the discarded
+  message in the current feature set.
 - **Non-claim-loop retry attempts:** tenacity retries in deduplicated publish,
   ack/remove, move-to-completed/failed, and lease renewal collapse into the
   terminal operation's failure event. There is no per-attempt event for those
@@ -1038,7 +1072,7 @@ v6.0.0 is a non-breaking-defaults release that adds new public APIs. v5 code con
 
 - `max_pending_length=N` caps pending-list depth; with `pending_overload_policy="raise"` (default) producers see `QueueBackpressureError` when the cap is hit; `"block"` waits up to `pending_overload_block_timeout_seconds`; `"drop_oldest"` evicts silently, so use it only when data loss is acceptable.
 - `queue.drain(timeout=...)` (sync) and `await queue.aclose(timeout=...)` (async) are explicit graceful-shutdown hooks. They refuse new claims and recover pending claim IDs but do not cancel in-flight handlers; join or await your worker separately.
-- `on_event=callback` receives a `QueueEvent` dataclass for every publish/claim/ack/reclaim/dedup/cleanup lifecycle event. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
+- `on_event=callback` receives a `QueueEvent` dataclass for publish/claim/ack/reclaim/dedup/cleanup/drain lifecycle events. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
 - See [`examples/production/backpressure.py`](examples/production/backpressure.py) and [`examples/production/graceful_shutdown.py`](examples/production/graceful_shutdown.py) for sync production patterns, with async siblings under [`examples/production/asyncio/`](examples/production/asyncio/).
 
 > When using a pre-fork app server (gunicorn `--preload`, uvicorn workers that import the app at master startup), call `make_queue()` from your worker startup hook - NOT at module import. See [Fork safety](#fork-safety-and-pre-fork-servers) for why.

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/README.md

@@ -11,7 +11,7 @@
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=8.2.1,<9.0.0"
+pip install "redis-message-queue>=8.2.3,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
@@ -22,6 +22,7 @@ Redis must be running locally first: use `redis-server` or
 `docker run -p 6379:6379 redis:7`.
 
 ```python
+import json
 from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
@@ -35,7 +36,8 @@ queue = RedisMessageQueue(
 queue.publish({"id": "msg-1", "text": "hello"})
 with queue.process_message() as message:
     if message is not None:
-        print(f"got {message['text']}")
+        payload = json.loads(message)
+        print(f"got {payload['text']}")
 # Expected output: got hello
 ```
 
@@ -46,13 +48,15 @@ with queue.process_message() as message:
 > synchronous. If your handler is `async def`, returns a coroutine, or returns
 > any other awaitable, use `redis_message_queue.asyncio.RedisMessageQueue`.
 > The sync context manager does not inspect the handler's return value; an
-> unawaited coroutine can be dropped while the message is acked. An ergonomic
-> callback API that detects this is planned for v8.1.
+> unawaited coroutine can be dropped while the message is acked. For sync
+> callback-style handlers, use `process_message_callback(handler)`: it checks
+> for awaitable returns before acking and raises `TypeError` if one is returned.
 
 ### Async quickstart
 
 ```python
 import asyncio
+import json
 from redis.asyncio import Redis
 from redis_message_queue.asyncio import RedisMessageQueue
 
@@ -66,7 +70,8 @@ async def main():
     )
     await queue.publish({"id": "msg-1", "text": "hello"})
     async with queue.process_message() as message:
-        print(f"got {message['text']}")
+        payload = json.loads(message)
+        print(f"got {payload['text']}")
     await client.aclose()
 
 asyncio.run(main())  # Expected output: got hello
@@ -202,7 +207,9 @@ so concurrent publishers cannot race above the configured cap. Overload policies
 - `drop_oldest` removes the oldest pending message (`RPOP`) before enqueueing the
   new message. This is silent data loss by design; deduplication markers for
   dropped messages are not removed, so a dropped duplicate may still be
-  suppressed until its dedup TTL expires.
+  suppressed until its dedup TTL expires. The current event contract emits
+  `publish/success` for the new message, but no separate `on_event` signal for
+  the dropped message.
 - `block` retries the atomic check until space opens or
   `pending_overload_block_timeout_seconds` elapses (default: 1.0), then raises
   `QueueBackpressureError`.
@@ -474,6 +481,14 @@ gateway = RedisGateway(
 queue = RedisMessageQueue("q", gateway=gateway)
 ```
 
+When `gateway=` is supplied, queue-level constructor defaults are not copied
+into the gateway. For example, `RedisMessageQueue(..., gateway=gateway)`
+leaves visibility timeout and dead-letter routing disabled unless
+`message_visibility_timeout_seconds` and `max_delivery_count` are configured on
+the gateway itself. Passing the queue-level default values
+`visibility_timeout_seconds=300` or `max_delivery_count=10` with `gateway=`
+does not transfer those settings to the gateway.
+
 The retry knobs configure an internal `tenacity` strategy: exponential
 backoff with jitter, retry on transient Redis errors only, capped at
 `retry_budget_seconds`. The budget is monotonic elapsed time from the first attempt (including attempt duration), not inter-attempt delay; it is unaffected by Python-host NTP jumps. A single attempt that takes longer than the budget results in zero retries. Setting `retry_budget_seconds=0` disables retry
@@ -706,8 +721,8 @@ queue = RedisMessageQueue("jobs", client=client, on_event=on_event)
 ```
 
 Events cover publish, dedup hits, claim/empty polls, reclaim, ack/nack,
-completed/failed cleanup, DLQ moves, heartbeat renewal, stale leases, cleanup
-and trim failures, and retry attempts. Callback exceptions are logged and
+completed/failed cleanup, DLQ moves, heartbeat renewal, stale leases, drain,
+cleanup and trim failures, and retry attempts. Callback exceptions are logged and
 reported with `RuntimeWarning`, but never propagate into queue operations.
 `on_event` is telemetry only: use it for metrics, tracing, and logging, not for
 sagas, follow-up writes, billing callbacks, or other correctness-critical
@@ -800,6 +815,23 @@ Pre-commit and mid-flight exceptions:
   despite the exception. Treat them as "operation did not succeed from the
   caller's perspective", not "Redis did not commit".
 
+#### Drain events
+
+`drain()` and `close()` on the sync queue, and `drain()` and `aclose()` on the
+async queue, emit `drain` events:
+
+- `drain/start` when the queue-local drain flag is set.
+- `drain/success` when pending claim IDs were recovered or no gateway drain
+  hook is present.
+- `drain/skipped` when the queue was already drained and the cached successful
+  result is returned.
+- `drain/failure` when pending claim recovery times out or otherwise leaves
+  unresolved claim IDs.
+
+Drain events use `timeout_seconds` for the caller-supplied timeout,
+`pending_claim_ids` for the number of unresolved local claim IDs when known,
+and `exception_type` / `error` on failure.
+
 #### Intentionally silent paths
 
 The following operations have no `on_event` surface by design:
@@ -814,9 +846,11 @@ The following operations have no `on_event` surface by design:
   it back to pending, and returns `false`. Python translates that into
   `claim_empty/skipped`, the same shape as an empty poll. This is intentional
   fail-safe behavior; the message is not lost.
-- **`drain()` / `close()` / `aclose()` lifecycle:** explicit shutdown
-  operations do not emit lifecycle events. Pending-claim-drain recovery work
-  counts as `claim_reclaim` events when reached.
+- **`drop_oldest` evictions:** when publish backpressure uses
+  `pending_overload_policy="drop_oldest"`, the oldest pending message is
+  discarded before the new message is enqueued. The successful enqueue emits
+  `publish/success`, but there is no separate drop event for the discarded
+  message in the current feature set.
 - **Non-claim-loop retry attempts:** tenacity retries in deduplicated publish,
   ack/remove, move-to-completed/failed, and lease renewal collapse into the
   terminal operation's failure event. There is no per-attempt event for those
@@ -1015,7 +1049,7 @@ v6.0.0 is a non-breaking-defaults release that adds new public APIs. v5 code con
 
 - `max_pending_length=N` caps pending-list depth; with `pending_overload_policy="raise"` (default) producers see `QueueBackpressureError` when the cap is hit; `"block"` waits up to `pending_overload_block_timeout_seconds`; `"drop_oldest"` evicts silently, so use it only when data loss is acceptable.
 - `queue.drain(timeout=...)` (sync) and `await queue.aclose(timeout=...)` (async) are explicit graceful-shutdown hooks. They refuse new claims and recover pending claim IDs but do not cancel in-flight handlers; join or await your worker separately.
-- `on_event=callback` receives a `QueueEvent` dataclass for every publish/claim/ack/reclaim/dedup/cleanup lifecycle event. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
+- `on_event=callback` receives a `QueueEvent` dataclass for publish/claim/ack/reclaim/dedup/cleanup/drain lifecycle events. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
 - See [`examples/production/backpressure.py`](examples/production/backpressure.py) and [`examples/production/graceful_shutdown.py`](examples/production/graceful_shutdown.py) for sync production patterns, with async siblings under [`examples/production/asyncio/`](examples/production/asyncio/).
 
 > When using a pre-fork app server (gunicorn `--preload`, uvicorn workers that import the app at master startup), call `make_queue()` from your worker startup hook - NOT at module import. See [Fork safety](#fork-safety-and-pre-fork-servers) for why.

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "redis-message-queue"
-version = "8.2.1"
+version = "8.2.3"
 description = "Python message queuing with Redis and message deduplication"
 authors = [{ name = "Elijas", email = "4084885+Elijas@users.noreply.github.com" }]
 readme = "README.md"
@@ -48,7 +48,7 @@ default-groups = ["dev", "test"]
 ##############################
 
 [tool.bumpversion]
-current_version = "8.2.1"
+current_version = "8.2.3"
 parse = "(?P<major>\\d+)\\.(?P<minor>\\d+)\\.(?P<patch>\\d+)"
 serialize = ["{major}.{minor}.{patch}"]
 search = "{current_version}"
@@ -84,6 +84,12 @@ line-length = 120
 select = ["E", "F", "I", "W"]
 ignore = ["E731"]
 
+[tool.mypy]
+python_version = "3.12"
+files = ["redis_message_queue/"]
+explicit_package_bases = true
+show_error_codes = true
+
 [tool.pytest.ini_options]
 asyncio_default_fixture_loop_scope = "function"
 filterwarnings = [

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/redis_message_queue/_config.py

@@ -585,6 +585,53 @@ return removed
 """
 )
 
+RETURN_MESSAGE_TO_PENDING_LUA_SCRIPT = (
+    _LUA_KEY_TYPE_GUARD
+    + """
+local err = redis_message_queue_require_type(KEYS[1], 'list')
+if err then
+    return err
+end
+
+local err = redis_message_queue_require_type(KEYS[2], 'list')
+if err then
+    return err
+end
+
+local err = redis_message_queue_require_type(KEYS[3], 'hash')
+if err then
+    return err
+end
+
+local err = redis_message_queue_require_type(KEYS[4], 'hash')
+if err then
+    return err
+end
+
+local err = redis_message_queue_require_type(KEYS[5], 'string')
+if err then
+    return err
+end
+
+local cached_result = redis.call('GET', KEYS[5])
+if cached_result then
+    return tonumber(cached_result)
+end
+
+redis.call('RPUSH', KEYS[2], ARGV[1])
+local removed = redis.call('LREM', KEYS[1], 1, ARGV[1])
+if removed == 1 then
+    redis.call('HDEL', KEYS[3], ARGV[2])
+    redis.call('HDEL', KEYS[4], ARGV[1])
+else
+    redis.call('LREM', KEYS[2], 1, ARGV[1])
+end
+
+redis.call('SET', KEYS[5], tostring(removed), 'PX', tonumber(ARGV[3]))
+return removed
+"""
+)
+
 REMOVE_MESSAGE_LUA_SCRIPT = (
     _LUA_KEY_TYPE_GUARD
     + """
@@ -876,8 +923,6 @@ while claim_attempts < 100 do
 
     local count = redis.call('HINCRBY', KEYS[6], stored, 1)
     if max_delivery_count > 0 and count > max_delivery_count then
-        redis.call('LREM', KEYS[2], 1, stored)
-        redis.call('HDEL', KEYS[6], stored)
         -- Strip envelope to store raw payload in DLQ, consistent with completed/failed queues.
         -- The per-delivery UUID in the envelope is lost; see README dead-letter notes.
         local dead_letter_value = stored
@@ -886,6 +931,8 @@ while claim_attempts < 100 do
             dead_letter_value = envelope['payload']
         end
         redis.call('LPUSH', KEYS[7], dead_letter_value)
+        redis.call('LREM', KEYS[2], 1, stored)
+        redis.call('HDEL', KEYS[6], stored)
         table.insert(dead_lettered_events, {redis_message_queue_message_id(stored), tostring(count)})
     else
         return store_claim_and_return(stored)

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/redis_message_queue/_redis_gateway.py

@@ -31,6 +31,7 @@ from redis_message_queue._config import (
     REMOVE_MESSAGE_LUA_SCRIPT,
     REMOVE_MESSAGE_WITH_LEASE_TOKEN_LUA_SCRIPT,
     RENEW_MESSAGE_LEASE_LUA_SCRIPT,
+    RETURN_MESSAGE_TO_PENDING_LUA_SCRIPT,
     _ChainedInterrupt,
     build_retry_strategy,
     is_redis_retryable_exception,
@@ -79,6 +80,8 @@ _VISIBILITY_TIMEOUT_POLL_INTERVAL_SECONDS = 0.25
 _PENDING_OVERLOAD_INITIAL_BACKOFF_SECONDS = 0.010
 _PENDING_OVERLOAD_MAX_BACKOFF_SECONDS = 0.500
 _CLAIM_STORE_FAILED_LUA_SENTINEL_BYTES = CLAIM_STORE_FAILED_LUA_SENTINEL.encode("utf-8")
+_PENDING_QUEUE_SUFFIX = "pending"
+_PROCESSING_QUEUE_SUFFIX = "processing"
 
 
 class _DrainDeadlineExceeded(Exception):
@@ -301,8 +304,8 @@ class RedisGateway(AbstractRedisGateway):
 
     def _emit_event(
         self,
-        operation: EventOperation,
-        outcome: EventOutcome,
+        operation: EventOperation | str,
+        outcome: EventOutcome | str,
         *,
         message_id: str | None = None,
         claim_id: str | None = None,
@@ -328,7 +331,7 @@ class RedisGateway(AbstractRedisGateway):
 
     def _emit_repeated_event(
         self,
-        operation: EventOperation,
+        operation: EventOperation | str,
         attempts: list[_MessageAttemptEvent],
         *,
         destination_queue: str | None = None,
@@ -1225,6 +1228,53 @@ class RedisGateway(AbstractRedisGateway):
         _raise_if_drain_deadline_expired(deadline_monotonic)
         return ClaimedMessage(stored_message=stored_message, lease_token=lease_token)
 
+    def _pending_queue_from_processing_queue(self, processing_queue: str) -> str:
+        if not processing_queue.endswith(_PROCESSING_QUEUE_SUFFIX):
+            raise RuntimeError(f"cannot derive pending queue key from processing queue {processing_queue!r}")
+        return f"{processing_queue.removesuffix(_PROCESSING_QUEUE_SUFFIX)}{_PENDING_QUEUE_SUFFIX}"
+
+    def _return_recovered_non_visibility_timeout_claim_to_pending(
+        self,
+        processing_queue: str,
+        stored_message: MessageData,
+        claim_id: str,
+        *,
+        deadline_monotonic: float | None,
+    ) -> bool:
+        pending_queue = self._pending_queue_from_processing_queue(processing_queue)
+        operation_id = uuid.uuid4().hex
+        operation_result_key = self._operation_result_key(processing_queue, operation_id)
+
+        try:
+            _raise_if_drain_deadline_expired(deadline_monotonic)
+            result = _call_with_drain_deadline(
+                lambda: self._eval(
+                    RETURN_MESSAGE_TO_PENDING_LUA_SCRIPT,
+                    5,
+                    processing_queue,
+                    pending_queue,
+                    self._claim_result_ids_key(processing_queue),
+                    self._claim_result_backrefs_key(processing_queue),
+                    operation_result_key,
+                    stored_message,
+                    claim_id,
+                    self._operation_result_ttl_ms(),
+                ),
+                deadline_monotonic=deadline_monotonic,
+            )
+            _raise_if_drain_deadline_expired(deadline_monotonic)
+            return bool(_coerce_lua_count(result))
+        except RedisMessageQueueError as exc:
+            _set_exception_context(
+                exc,
+                queue=processing_queue,
+                message_id=extract_stored_message_id(stored_message),
+                operation="drain",
+            )
+            raise
+        finally:
+            self._delete_operation_result_key(operation_result_key)
+
     def _is_interrupted(self, is_interrupted: BaseGracefulInterruptHandler | None = None) -> bool:
         return (self._interrupt is not None and self._interrupt.is_interrupted()) or (
             is_interrupted is not None and is_interrupted.is_interrupted()
@@ -1241,8 +1291,10 @@ class RedisGateway(AbstractRedisGateway):
         Walks the same recovery path as ``_wait_for_claim`` but without
         gating on the interrupt flag, so a soft shutdown can flush
         ambiguous-claim state that would otherwise be dropped on process
-        exit (AA-05-F2). Returns True if no pending ids remain; False if
-        the deadline fired or transient Redis errors prevented full drain.
+        exit (AA-05-F2). No-visibility-timeout recoveries are returned to
+        the pending queue before their claim ids are cleared. Returns True
+        if no pending ids remain; False if the deadline fired or Redis
+        errors prevented full drain.
         """
         with self._drain_pending_claim_ids_lock:
             self._last_drain_error = None
@@ -1258,11 +1310,8 @@ class RedisGateway(AbstractRedisGateway):
         deadline_monotonic: float | None,
     ) -> bool:
         """Recover every in-memory pending claim id for ``processing_queue``."""
-        if self._message_visibility_timeout_seconds is not None:
-            recover = self._recover_pending_visibility_timeout_claim
-        else:
-            recover = self._recover_pending_non_visibility_timeout_claim
-        skipped_transient: set[str] = set()
+        has_visibility_timeout = self._message_visibility_timeout_seconds is not None
+        skipped_unresolved: set[str] = set()
         last_error: BaseException | None = None
         while True:
             # ``>=`` (not ``>``) makes ``timeout=0`` deterministically take
@@ -1278,7 +1327,7 @@ class RedisGateway(AbstractRedisGateway):
                     return True
                 recovering = self._recovering_claim_ids.setdefault(processing_queue, set())
                 claim_id = next(
-                    (cid for cid in pending if cid not in recovering and cid not in skipped_transient),
+                    (cid for cid in pending if cid not in recovering and cid not in skipped_unresolved),
                     None,
                 )
                 if claim_id is None:
@@ -1287,8 +1336,34 @@ class RedisGateway(AbstractRedisGateway):
             clear = False
             try:
                 try:
-                    recover(processing_queue, claim_id, deadline_monotonic=deadline_monotonic)
-                    clear = True
+                    if has_visibility_timeout:
+                        self._recover_pending_visibility_timeout_claim(
+                            processing_queue,
+                            claim_id,
+                            deadline_monotonic=deadline_monotonic,
+                        )
+                        clear = True
+                        continue
+
+                    recovered_claim = self._recover_pending_non_visibility_timeout_claim(
+                        processing_queue,
+                        claim_id,
+                        deadline_monotonic=deadline_monotonic,
+                    )
+                    if recovered_claim is None:
+                        clear = True
+                    elif self._return_recovered_non_visibility_timeout_claim_to_pending(
+                        processing_queue,
+                        recovered_claim,
+                        claim_id,
+                        deadline_monotonic=deadline_monotonic,
+                    ):
+                        clear = True
+                    else:
+                        last_error = RuntimeError(
+                            f"drain recovered claim {claim_id!r} but message was not present in processing queue"
+                        )
+                        skipped_unresolved.add(claim_id)
                 except _DrainDeadlineExceeded:
                     last_error = TimeoutError("drain pending-claim recovery deadline expired")
                     break
@@ -1301,7 +1376,7 @@ class RedisGateway(AbstractRedisGateway):
                         claim_id,
                         type(exc).__name__,
                     )
-                    skipped_transient.add(claim_id)
+                    skipped_unresolved.add(claim_id)
             finally:
                 self._finish_pending_claim_recovery(processing_queue, claim_id, clear=clear)
         with self._pending_claim_ids_lock:

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/redis_message_queue/asyncio/_redis_gateway.py

@@ -29,6 +29,7 @@ from redis_message_queue._config import (
     REMOVE_MESSAGE_LUA_SCRIPT,
     REMOVE_MESSAGE_WITH_LEASE_TOKEN_LUA_SCRIPT,
     RENEW_MESSAGE_LEASE_LUA_SCRIPT,
+    RETURN_MESSAGE_TO_PENDING_LUA_SCRIPT,
     _ChainedInterrupt,
     build_retry_strategy,
     is_redis_retryable_exception,
@@ -78,6 +79,8 @@ _VISIBILITY_TIMEOUT_POLL_INTERVAL_SECONDS = 0.25
 _PENDING_OVERLOAD_INITIAL_BACKOFF_SECONDS = 0.010
 _PENDING_OVERLOAD_MAX_BACKOFF_SECONDS = 0.500
 _CLAIM_STORE_FAILED_LUA_SENTINEL_BYTES = CLAIM_STORE_FAILED_LUA_SENTINEL.encode("utf-8")
+_PENDING_QUEUE_SUFFIX = "pending"
+_PROCESSING_QUEUE_SUFFIX = "processing"
 
 
 class _DrainDeadlineExceeded(Exception):
@@ -275,8 +278,8 @@ class RedisGateway(AbstractRedisGateway):
 
     async def _emit_event(
         self,
-        operation: EventOperation,
-        outcome: EventOutcome,
+        operation: EventOperation | str,
+        outcome: EventOutcome | str,
         *,
         message_id: str | None = None,
         claim_id: str | None = None,
@@ -302,7 +305,7 @@ class RedisGateway(AbstractRedisGateway):
 
     async def _emit_repeated_event(
         self,
-        operation: EventOperation,
+        operation: EventOperation | str,
         attempts: list[_MessageAttemptEvent],
         *,
         destination_queue: str | None = None,
@@ -1206,6 +1209,53 @@ class RedisGateway(AbstractRedisGateway):
         _raise_if_drain_deadline_expired(deadline_monotonic)
         return ClaimedMessage(stored_message=stored_message, lease_token=lease_token)
 
+    def _pending_queue_from_processing_queue(self, processing_queue: str) -> str:
+        if not processing_queue.endswith(_PROCESSING_QUEUE_SUFFIX):
+            raise RuntimeError(f"cannot derive pending queue key from processing queue {processing_queue!r}")
+        return f"{processing_queue.removesuffix(_PROCESSING_QUEUE_SUFFIX)}{_PENDING_QUEUE_SUFFIX}"
+
+    async def _return_recovered_non_visibility_timeout_claim_to_pending(
+        self,
+        processing_queue: str,
+        stored_message: MessageData,
+        claim_id: str,
+        *,
+        deadline_monotonic: float | None,
+    ) -> bool:
+        pending_queue = self._pending_queue_from_processing_queue(processing_queue)
+        operation_id = uuid.uuid4().hex
+        operation_result_key = self._operation_result_key(processing_queue, operation_id)
+
+        try:
+            _raise_if_drain_deadline_expired(deadline_monotonic)
+            result = await _call_with_drain_deadline(
+                lambda: self._eval(
+                    RETURN_MESSAGE_TO_PENDING_LUA_SCRIPT,
+                    5,
+                    processing_queue,
+                    pending_queue,
+                    self._claim_result_ids_key(processing_queue),
+                    self._claim_result_backrefs_key(processing_queue),
+                    operation_result_key,
+                    stored_message,
+                    claim_id,
+                    self._operation_result_ttl_ms(),
+                ),
+                deadline_monotonic=deadline_monotonic,
+            )
+            _raise_if_drain_deadline_expired(deadline_monotonic)
+            return bool(_coerce_lua_count(result))
+        except RedisMessageQueueError as exc:
+            _set_exception_context(
+                exc,
+                queue=processing_queue,
+                message_id=extract_stored_message_id(stored_message),
+                operation="drain",
+            )
+            raise
+        finally:
+            await self._delete_operation_result_key(operation_result_key)
+
     def _is_interrupted(self, is_interrupted: BaseGracefulInterruptHandler | None = None) -> bool:
         return (self._interrupt is not None and self._interrupt.is_interrupted()) or (
             is_interrupted is not None and is_interrupted.is_interrupted()
@@ -1222,8 +1272,10 @@ class RedisGateway(AbstractRedisGateway):
         Kept separate from the sync implementation per AF11 (sync/async
         gateways stay duplicated). Walks the same recovery path as
         ``_wait_for_claim`` but without gating on the interrupt flag so a
-        soft shutdown can flush ambiguous-claim state. Returns True if no
-        pending ids remain; False on deadline expiry or transient errors.
+        soft shutdown can flush ambiguous-claim state. No-visibility-timeout
+        recoveries are returned to the pending queue before their claim ids
+        are cleared. Returns True if no pending ids remain; False on deadline
+        expiry or Redis errors.
         """
         async with self._drain_pending_claim_ids_lock:
             self._last_drain_error = None
@@ -1239,11 +1291,8 @@ class RedisGateway(AbstractRedisGateway):
         deadline_monotonic: float | None,
     ) -> bool:
         """Recover every in-memory pending claim id for ``processing_queue``."""
-        if self._message_visibility_timeout_seconds is not None:
-            recover = self._recover_pending_visibility_timeout_claim
-        else:
-            recover = self._recover_pending_non_visibility_timeout_claim
-        skipped_transient: set[str] = set()
+        has_visibility_timeout = self._message_visibility_timeout_seconds is not None
+        skipped_unresolved: set[str] = set()
         loop = asyncio.get_running_loop()
         last_error: BaseException | None = None
         while True:
@@ -1258,7 +1307,7 @@ class RedisGateway(AbstractRedisGateway):
                     return True
                 recovering = self._recovering_claim_ids.setdefault(processing_queue, set())
                 claim_id = next(
-                    (cid for cid in pending if cid not in recovering and cid not in skipped_transient),
+                    (cid for cid in pending if cid not in recovering and cid not in skipped_unresolved),
                     None,
                 )
                 if claim_id is None:
@@ -1267,8 +1316,34 @@ class RedisGateway(AbstractRedisGateway):
             clear = False
             try:
                 try:
-                    await recover(processing_queue, claim_id, deadline_monotonic=deadline_monotonic)
-                    clear = True
+                    if has_visibility_timeout:
+                        await self._recover_pending_visibility_timeout_claim(
+                            processing_queue,
+                            claim_id,
+                            deadline_monotonic=deadline_monotonic,
+                        )
+                        clear = True
+                        continue
+
+                    recovered_claim = await self._recover_pending_non_visibility_timeout_claim(
+                        processing_queue,
+                        claim_id,
+                        deadline_monotonic=deadline_monotonic,
+                    )
+                    if recovered_claim is None:
+                        clear = True
+                    elif await self._return_recovered_non_visibility_timeout_claim_to_pending(
+                        processing_queue,
+                        recovered_claim,
+                        claim_id,
+                        deadline_monotonic=deadline_monotonic,
+                    ):
+                        clear = True
+                    else:
+                        last_error = RuntimeError(
+                            f"drain recovered claim {claim_id!r} but message was not present in processing queue"
+                        )
+                        skipped_unresolved.add(claim_id)
                 except _DrainDeadlineExceeded:
                     last_error = TimeoutError("drain pending-claim recovery deadline expired")
                     break
@@ -1281,7 +1356,7 @@ class RedisGateway(AbstractRedisGateway):
                         claim_id,
                         type(exc).__name__,
                     )
-                    skipped_transient.add(claim_id)
+                    skipped_unresolved.add(claim_id)
             finally:
                 self._finish_pending_claim_recovery(processing_queue, claim_id, clear=clear)
         with self._pending_claim_ids_lock:

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/redis_message_queue/asyncio/redis_message_queue.py

@@ -11,7 +11,6 @@ from typing import AsyncIterator, Awaitable, Callable, Literal, Optional, TypeVa
 import redis.asyncio
 import redis.exceptions
 
-from redis_message_queue._callable_utils import is_async_callable
 from redis_message_queue._config import (
     DEFAULT_PENDING_OVERLOAD_BLOCK_TIMEOUT_SECONDS,
     validate_dedup_configuration,
@@ -332,8 +331,12 @@ def _derive_dead_letter_queue(name: str, key_separator: str) -> str:
     return f"{name}{key_separator}{_AUTO_DEAD_LETTER_QUEUE_SUFFIX}"
 
 
-def _bind_dead_letter_gateway_to_queue(gateway: AbstractRedisGateway, queue_pending_key: str) -> None:
-    """Reject reuse of a dead-letter-enabled gateway across different queues.
+def _bind_dead_letter_gateway_to_queue(
+    gateway: AbstractRedisGateway,
+    queue_pending_key: str,
+    queue_processing_key: str,
+) -> None:
+    """Validate and bind a dead-letter-enabled gateway to this queue.
 
     The check is not thread-safe: constructing ``RedisMessageQueue`` instances
     concurrently on multiple threads with the same DLQ-enabled gateway can
@@ -346,6 +349,12 @@ def _bind_dead_letter_gateway_to_queue(gateway: AbstractRedisGateway, queue_pend
     if max_delivery_count is None:
         return
 
+    if gateway.dead_letter_queue in (queue_pending_key, queue_processing_key):
+        raise ConfigurationError(
+            "'dead_letter_queue' must be distinct from the queue's pending and processing Redis keys. "
+            "Use a separate Redis list key for poison messages."
+        )
+
     bound_pending_key = getattr(gateway, _GATEWAY_BOUND_PENDING_QUEUE_ATTR, None)
     if bound_pending_key is None:
         setattr(gateway, _GATEWAY_BOUND_PENDING_QUEUE_ATTR, queue_pending_key)
@@ -451,7 +460,7 @@ class _LeaseHeartbeat:
                 stacklevel=2,
             )
 
-    async def _emit(self, operation: EventOperation, outcome: EventOutcome, **kwargs: object) -> None:
+    async def _emit(self, operation: EventOperation | str, outcome: EventOutcome | str, **kwargs: object) -> None:
         if self._emit_event is not None:
             await self._emit_event(operation, outcome, **kwargs)
 
@@ -597,6 +606,11 @@ class RedisMessageQueue:
         auto-derived dead-letter queue. Set it to ``None`` for unlimited
         redelivery.
 
+        When ``gateway=`` is supplied, queue-level defaults are not transferred
+        to the gateway. Configure lease, dead-letter, and backpressure settings
+        such as ``message_visibility_timeout_seconds``, ``max_delivery_count``,
+        and ``max_pending_length`` on the gateway itself.
+
         ``deduplication=True`` requires ``get_deduplication_key`` to be a
         callable that returns a non-empty string. Use a stable logical ID for
         the deduplication keyspace.
@@ -636,11 +650,11 @@ class RedisMessageQueue:
         ``GracefulInterruptHandler()`` for prompt Ctrl-C / termination handling
         in polling waits. ``on_heartbeat_failure`` is a zero-argument callable
         or coroutine callable invoked when lease renewal fails. ``on_event`` is
-        telemetry only: an async callback receiving best-effort QueueEvent
-        lifecycle notifications. Callback failures are logged and converted to
-        RuntimeWarning without influencing ack/nack or any other message
-        outcome. Do not use it for correctness-critical callbacks or follow-up
-        writes.
+        telemetry only: a callable returning an awaitable and receiving
+        best-effort QueueEvent lifecycle notifications. Callback failures are
+        logged and converted to RuntimeWarning without influencing ack/nack or
+        any other message outcome. Do not use it for correctness-critical
+        callbacks or follow-up writes.
         """
         self.key = QueueKeyManager(name, key_separator=key_separator)
         if not isinstance(deduplication, bool):
@@ -742,8 +756,6 @@ class RedisMessageQueue:
             )
         if on_event is not None and not callable(on_event):
             raise TypeError(f"'on_event' must be callable, got {type(on_event).__name__}.")
-        if on_event is not None and not is_async_callable(on_event):
-            raise TypeError("'on_event' must be an async callable.")
         self._queue_name = name
         self._on_event = on_event
         # Queue-local soft-drain flag. See sync queue ``_draining`` docstring
@@ -825,7 +837,7 @@ class RedisMessageQueue:
                     "'max_pending_length' cannot be provided alongside 'gateway'."
                     " Configure publish backpressure on the gateway directly instead."
                 )
-            _bind_dead_letter_gateway_to_queue(gateway, self.key.pending)
+            _bind_dead_letter_gateway_to_queue(gateway, self.key.pending, self.key.processing)
             self._max_delivery_count = None
             self._redis = gateway
         elif client is None:
@@ -867,8 +879,8 @@ class RedisMessageQueue:
 
     async def _emit_event(
         self,
-        operation: EventOperation,
-        outcome: EventOutcome,
+        operation: EventOperation | str,
+        outcome: EventOutcome | str,
         *,
         message_id: str | None = None,
         claim_id: str | None = None,
@@ -908,6 +920,18 @@ class RedisMessageQueue:
                     "Use an async callable or return an awaitable."
                 )
             await result
+        except asyncio.CancelledError as exc:
+            current_task = asyncio.current_task()
+            if current_task is not None and current_task.cancelling() > 0:
+                raise
+            logger.exception("on_event callback raised an exception")
+            with warnings.catch_warnings():
+                warnings.simplefilter("always", RuntimeWarning)
+                warnings.warn(
+                    f"on_event callback raised {type(exc).__name__}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
         except Exception as exc:
             logger.exception("on_event callback raised an exception")
             with warnings.catch_warnings():
@@ -1402,7 +1426,9 @@ class RedisMessageQueue:
         calls raise ``QueueDrainedError``. It then awaits the gateway's
         pending-claim-id recovery loop. Returns ``True`` if all pending claim
         ids were recovered, ``False`` if the deadline fired or a transient
-        Redis error left ids pending.
+        Redis error left ids pending. In no-visibility-timeout queues,
+        recovered messages are returned to pending before the claim id is
+        cleared.
 
         Unlike ``asyncio.CancelledError`` (hard-abort, leaves messages
         claimed for VT-reclaim), ``aclose()`` is the explicit-drain

{redis_message_queue-8.2.1 → redis_message_queue-8.2.3}/redis_message_queue/redis_message_queue.py

@@ -1,3 +1,4 @@
+import asyncio
 import hashlib
 import inspect
 import logging
@@ -273,8 +274,12 @@ def _derive_dead_letter_queue(name: str, key_separator: str) -> str:
     return f"{name}{key_separator}{_AUTO_DEAD_LETTER_QUEUE_SUFFIX}"
 
 
-def _bind_dead_letter_gateway_to_queue(gateway: AbstractRedisGateway, queue_pending_key: str) -> None:
-    """Reject reuse of a dead-letter-enabled gateway across different queues.
+def _bind_dead_letter_gateway_to_queue(
+    gateway: AbstractRedisGateway,
+    queue_pending_key: str,
+    queue_processing_key: str,
+) -> None:
+    """Validate and bind a dead-letter-enabled gateway to this queue.
 
     The check is not thread-safe: constructing ``RedisMessageQueue`` instances
     concurrently on multiple threads with the same DLQ-enabled gateway can
@@ -287,6 +292,12 @@ def _bind_dead_letter_gateway_to_queue(gateway: AbstractRedisGateway, queue_pend
     if max_delivery_count is None:
         return
 
+    if gateway.dead_letter_queue in (queue_pending_key, queue_processing_key):
+        raise ConfigurationError(
+            "'dead_letter_queue' must be distinct from the queue's pending and processing Redis keys. "
+            "Use a separate Redis list key for poison messages."
+        )
+
     bound_pending_key = getattr(gateway, _GATEWAY_BOUND_PENDING_QUEUE_ATTR, None)
     if bound_pending_key is None:
         setattr(gateway, _GATEWAY_BOUND_PENDING_QUEUE_ATTR, queue_pending_key)
@@ -407,7 +418,7 @@ class _LeaseHeartbeat:
                 stacklevel=2,
             )
 
-    def _emit(self, operation: EventOperation, outcome: EventOutcome, **kwargs: object) -> None:
+    def _emit(self, operation: EventOperation | str, outcome: EventOutcome | str, **kwargs: object) -> None:
         if self._emit_event is not None:
             self._emit_event(operation, outcome, **kwargs)
 
@@ -549,6 +560,11 @@ class RedisMessageQueue:
         auto-derived dead-letter queue. Set it to ``None`` for unlimited
         redelivery.
 
+        When ``gateway=`` is supplied, queue-level defaults are not transferred
+        to the gateway. Configure lease, dead-letter, and backpressure settings
+        such as ``message_visibility_timeout_seconds``, ``max_delivery_count``,
+        and ``max_pending_length`` on the gateway itself.
+
         ``deduplication=True`` requires ``get_deduplication_key`` to be a
         callable that returns a non-empty string. Use a stable logical ID for
         the deduplication keyspace.
@@ -786,7 +802,7 @@ class RedisMessageQueue:
                     "'max_pending_length' cannot be provided alongside 'gateway'."
                     " Configure publish backpressure on the gateway directly instead."
                 )
-            _bind_dead_letter_gateway_to_queue(gateway, self.key.pending)
+            _bind_dead_letter_gateway_to_queue(gateway, self.key.pending, self.key.processing)
             self._max_delivery_count = None
             self._redis = gateway
         elif client is None:
@@ -827,8 +843,8 @@ class RedisMessageQueue:
 
     def _emit_event(
         self,
-        operation: EventOperation,
-        outcome: EventOutcome,
+        operation: EventOperation | str,
+        outcome: EventOutcome | str,
         *,
         message_id: str | None = None,
         claim_id: str | None = None,
@@ -868,6 +884,15 @@ class RedisMessageQueue:
                     "'on_event' returned an awaitable; use the async RedisMessageQueue "
                     "from redis_message_queue.asyncio instead"
                 )
+        except asyncio.CancelledError as exc:
+            logger.exception("on_event callback raised an exception")
+            with warnings.catch_warnings():
+                warnings.simplefilter("always", RuntimeWarning)
+                warnings.warn(
+                    f"on_event callback raised {type(exc).__name__}",
+                    RuntimeWarning,
+                    stacklevel=2,
+                )
         except Exception as exc:
             logger.exception("on_event callback raised an exception")
             with warnings.catch_warnings():
@@ -1031,8 +1056,9 @@ class RedisMessageQueue:
         does not inspect handler return values; if your handler returns a
         coroutine or other awaitable, the awaitable can be dropped while the
         message is acked. Use ``redis_message_queue.asyncio.RedisMessageQueue``
-        for async handlers. An ergonomic callback API that detects this is
-        planned for v8.1.
+        for async handlers. For sync callback-style handlers, use
+        ``process_message_callback(handler)`` so awaitable returns are detected
+        before acking.
 
         If the process is killed mid-handler, the claimed message and lease
         metadata remain in Redis until a later consumer claim triggers
@@ -1358,7 +1384,8 @@ class RedisMessageQueue:
         calls raise ``QueueDrainedError``. It then walks the gateway's
         in-memory ``_pending_claim_ids`` to recover any ambiguous claims that
         an interrupt-aware shutdown would otherwise drop on process exit
-        (AA-05-F2).
+        (AA-05-F2). In no-visibility-timeout queues, recovered messages are
+        returned to pending before the claim id is cleared.
 
         ``timeout`` bounds the pending-claim-id recovery loop in seconds;
         ``None`` waits indefinitely, ``0`` skips the loop entirely. The