redis-message-queue 3.0.0__tar.gz → 3.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/PKG-INFO +5 -4
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/README.md +4 -3
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/pyproject.toml +1 -1
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_abstract_redis_gateway.py +8 -4
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_redis_gateway.py +1 -5
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/asyncio/_abstract_redis_gateway.py +8 -4
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/asyncio/_redis_gateway.py +1 -5
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/asyncio/redis_message_queue.py +17 -14
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/interrupt_handler/_implementation.py +7 -2
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/redis_message_queue.py +19 -15
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/LICENSE +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/__init__.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_callable_utils.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_config.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_queue_key_manager.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_redis_cluster.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_stored_message.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/asyncio/__init__.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/interrupt_handler/__init__.py +0 -0
- {redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/interrupt_handler/_interface.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: redis-message-queue
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.1.0
|
|
4
4
|
Summary: Python message queuing with Redis and message deduplication
|
|
5
5
|
License-File: LICENSE
|
|
6
6
|
Author: Elijas
|
|
@@ -16,7 +16,7 @@ Description-Content-Type: text/markdown
|
|
|
16
16
|
|
|
17
17
|
# redis-message-queue
|
|
18
18
|
|
|
19
|
-
[](https://pypi.org/project/redis-message-queue)
|
|
20
20
|
[](https://pypistats.org/packages/redis-message-queue)
|
|
21
21
|
[](LICENSE)
|
|
22
22
|
[](https://github.com/Elijas/redis-message-queue/issues)
|
|
@@ -234,7 +234,7 @@ queue = RedisMessageQueue("q", gateway=gateway)
|
|
|
234
234
|
|
|
235
235
|
The retry knobs configure an internal `tenacity` strategy: exponential
|
|
236
236
|
backoff with jitter, retry on transient Redis errors only, capped at
|
|
237
|
-
`retry_budget_seconds`. Setting `retry_budget_seconds=0` disables retry
|
|
237
|
+
`retry_budget_seconds`. The budget is wall-clock time from the first attempt (including attempt duration), not inter-attempt delay; a single attempt that takes longer than the budget results in zero retries. Setting `retry_budget_seconds=0` disables retry
|
|
238
238
|
entirely (single attempt; exceptions propagate). The library uses
|
|
239
239
|
`retry_budget_seconds` to size the operation-result cache TTL automatically,
|
|
240
240
|
so the previous footgun of an over-long retry budget out-living the cache
|
|
@@ -299,8 +299,9 @@ await client.aclose()
|
|
|
299
299
|
- **Timed waits use polling claim loops.** To make claims recoverable after ambiguous connection drops, `wait_for_message_and_move()` uses idempotent Lua claim polling instead of raw blocking list-move commands. This adds a small polling cadence during timed waits.
|
|
300
300
|
- **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
|
|
301
301
|
- **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
|
|
302
|
+
- **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
|
|
302
303
|
- **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
|
|
303
|
-
- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk.
|
|
304
|
+
- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
|
|
304
305
|
|
|
305
306
|
For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
|
|
306
307
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# redis-message-queue
|
|
2
2
|
|
|
3
|
-
[](https://pypi.org/project/redis-message-queue)
|
|
4
4
|
[](https://pypistats.org/packages/redis-message-queue)
|
|
5
5
|
[](LICENSE)
|
|
6
6
|
[](https://github.com/Elijas/redis-message-queue/issues)
|
|
@@ -218,7 +218,7 @@ queue = RedisMessageQueue("q", gateway=gateway)
|
|
|
218
218
|
|
|
219
219
|
The retry knobs configure an internal `tenacity` strategy: exponential
|
|
220
220
|
backoff with jitter, retry on transient Redis errors only, capped at
|
|
221
|
-
`retry_budget_seconds`. Setting `retry_budget_seconds=0` disables retry
|
|
221
|
+
`retry_budget_seconds`. The budget is wall-clock time from the first attempt (including attempt duration), not inter-attempt delay; a single attempt that takes longer than the budget results in zero retries. Setting `retry_budget_seconds=0` disables retry
|
|
222
222
|
entirely (single attempt; exceptions propagate). The library uses
|
|
223
223
|
`retry_budget_seconds` to size the operation-result cache TTL automatically,
|
|
224
224
|
so the previous footgun of an over-long retry budget out-living the cache
|
|
@@ -283,8 +283,9 @@ await client.aclose()
|
|
|
283
283
|
- **Timed waits use polling claim loops.** To make claims recoverable after ambiguous connection drops, `wait_for_message_and_move()` uses idempotent Lua claim polling instead of raw blocking list-move commands. This adds a small polling cadence during timed waits.
|
|
284
284
|
- **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
|
|
285
285
|
- **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
|
|
286
|
+
- **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
|
|
286
287
|
- **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
|
|
287
|
-
- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk.
|
|
288
|
+
- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
|
|
288
289
|
|
|
289
290
|
For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
|
|
290
291
|
|
|
@@ -11,13 +11,17 @@ class AbstractRedisGateway(ABC):
|
|
|
11
11
|
gateways MUST uphold the same behavioral contracts documented on each method
|
|
12
12
|
to avoid phantom heartbeats, undetected lease conflicts, or silent data loss.
|
|
13
13
|
|
|
14
|
-
Gateways that support visibility timeouts (lease-based claiming)
|
|
14
|
+
Gateways that support visibility timeouts (lease-based claiming) MUST expose
|
|
15
15
|
a ``message_visibility_timeout_seconds`` property (int or None). This is not
|
|
16
16
|
abstract because it is configuration rather than protocol, but it is required
|
|
17
17
|
when the queue is configured with ``heartbeat_interval_seconds``.
|
|
18
|
-
Lease-capable custom gateways
|
|
19
|
-
|
|
20
|
-
gateway as a non-lease implementation.
|
|
18
|
+
Lease-capable custom gateways MUST expose this property; omitting it
|
|
19
|
+
silently disables heartbeat validation and lease-token safety checks,
|
|
20
|
+
causing the queue to treat the gateway as a non-lease implementation.
|
|
21
|
+
|
|
22
|
+
Gateways that wrap a Redis Cluster client should expose an
|
|
23
|
+
``is_redis_cluster`` property returning ``True`` so the queue can apply
|
|
24
|
+
hash-tag validation at construction time.
|
|
21
25
|
|
|
22
26
|
Concurrency
|
|
23
27
|
-----------
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_redis_gateway.py
RENAMED
|
@@ -589,7 +589,7 @@ class RedisGateway(AbstractRedisGateway):
|
|
|
589
589
|
return f"{processing_queue}{_OPERATION_RESULT_SUFFIX}:{lease_token}:{operation_id}"
|
|
590
590
|
|
|
591
591
|
def _publish_operation_result_ttl_ms(self) -> str:
|
|
592
|
-
return str(max(self._message_deduplication_log_ttl_seconds, 3600) * 1000)
|
|
592
|
+
return str(max(self._message_deduplication_log_ttl_seconds, 3600, self._retry_budget_seconds + 180) * 1000)
|
|
593
593
|
|
|
594
594
|
def _operation_result_ttl_ms(self) -> str:
|
|
595
595
|
# Floor is derived from the configured retry budget so the cached
|
|
@@ -685,8 +685,6 @@ class RedisGateway(AbstractRedisGateway):
|
|
|
685
685
|
claim_result_key = self._claim_result_key(processing_queue, claim_id)
|
|
686
686
|
cached_claim = self._redis_client.get(claim_result_key)
|
|
687
687
|
if cached_claim is None:
|
|
688
|
-
if self._is_interrupted():
|
|
689
|
-
return None
|
|
690
688
|
cached_claim = self._redis_client.hget(self._claim_result_ids_key(processing_queue), claim_id)
|
|
691
689
|
if cached_claim is None:
|
|
692
690
|
return None
|
|
@@ -701,8 +699,6 @@ class RedisGateway(AbstractRedisGateway):
|
|
|
701
699
|
claim_result_key = self._claim_result_key(processing_queue, claim_id)
|
|
702
700
|
cached_claim = self._redis_client.get(claim_result_key)
|
|
703
701
|
if cached_claim is None:
|
|
704
|
-
if self._is_interrupted():
|
|
705
|
-
return None
|
|
706
702
|
cached_claim = self._redis_client.hget(self._claim_result_ids_key(processing_queue), claim_id)
|
|
707
703
|
if cached_claim is None:
|
|
708
704
|
return None
|
|
@@ -12,13 +12,17 @@ class AbstractRedisGateway(ABC):
|
|
|
12
12
|
documented on each method to avoid phantom heartbeats, undetected lease conflicts,
|
|
13
13
|
or silent data loss.
|
|
14
14
|
|
|
15
|
-
Gateways that support visibility timeouts (lease-based claiming)
|
|
15
|
+
Gateways that support visibility timeouts (lease-based claiming) MUST expose
|
|
16
16
|
a ``message_visibility_timeout_seconds`` property (int or None). This is not
|
|
17
17
|
abstract because it is configuration rather than protocol, but it is required
|
|
18
18
|
when the queue is configured with ``heartbeat_interval_seconds``.
|
|
19
|
-
Lease-capable custom gateways
|
|
20
|
-
|
|
21
|
-
gateway as a non-lease implementation.
|
|
19
|
+
Lease-capable custom gateways MUST expose this property; omitting it
|
|
20
|
+
silently disables heartbeat validation and lease-token safety checks,
|
|
21
|
+
causing the queue to treat the gateway as a non-lease implementation.
|
|
22
|
+
|
|
23
|
+
Gateways that wrap a Redis Cluster client should expose an
|
|
24
|
+
``is_redis_cluster`` property returning ``True`` so the queue can apply
|
|
25
|
+
hash-tag validation at construction time.
|
|
22
26
|
|
|
23
27
|
Concurrency
|
|
24
28
|
-----------
|
|
@@ -590,7 +590,7 @@ class RedisGateway(AbstractRedisGateway):
|
|
|
590
590
|
return f"{processing_queue}{_OPERATION_RESULT_SUFFIX}:{lease_token}:{operation_id}"
|
|
591
591
|
|
|
592
592
|
def _publish_operation_result_ttl_ms(self) -> str:
|
|
593
|
-
return str(max(self._message_deduplication_log_ttl_seconds, 3600) * 1000)
|
|
593
|
+
return str(max(self._message_deduplication_log_ttl_seconds, 3600, self._retry_budget_seconds + 180) * 1000)
|
|
594
594
|
|
|
595
595
|
def _operation_result_ttl_ms(self) -> str:
|
|
596
596
|
# Floor is derived from the configured retry budget so the cached
|
|
@@ -686,8 +686,6 @@ class RedisGateway(AbstractRedisGateway):
|
|
|
686
686
|
claim_result_key = self._claim_result_key(processing_queue, claim_id)
|
|
687
687
|
cached_claim = await self._redis_client.get(claim_result_key)
|
|
688
688
|
if cached_claim is None:
|
|
689
|
-
if self._is_interrupted():
|
|
690
|
-
return None
|
|
691
689
|
cached_claim = await self._redis_client.hget(self._claim_result_ids_key(processing_queue), claim_id)
|
|
692
690
|
if cached_claim is None:
|
|
693
691
|
return None
|
|
@@ -702,8 +700,6 @@ class RedisGateway(AbstractRedisGateway):
|
|
|
702
700
|
claim_result_key = self._claim_result_key(processing_queue, claim_id)
|
|
703
701
|
cached_claim = await self._redis_client.get(claim_result_key)
|
|
704
702
|
if cached_claim is None:
|
|
705
|
-
if self._is_interrupted():
|
|
706
|
-
return None
|
|
707
703
|
cached_claim = await self._redis_client.hget(self._claim_result_ids_key(processing_queue), claim_id)
|
|
708
704
|
if cached_claim is None:
|
|
709
705
|
return None
|
|
@@ -20,6 +20,17 @@ logger = logging.getLogger(__name__)
|
|
|
20
20
|
_T = TypeVar("_T")
|
|
21
21
|
_GATEWAY_BOUND_PENDING_QUEUE_ATTR = "_rmq_bound_pending_queue"
|
|
22
22
|
|
|
23
|
+
_STALE_LEASE_ACK_WARNING = (
|
|
24
|
+
"Message cleanup after successful processing was a no-op: "
|
|
25
|
+
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
26
|
+
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
27
|
+
)
|
|
28
|
+
_STALE_LEASE_NACK_WARNING = (
|
|
29
|
+
"Message cleanup after failed processing was a no-op: "
|
|
30
|
+
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
31
|
+
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
32
|
+
)
|
|
33
|
+
|
|
23
34
|
|
|
24
35
|
class _TaskBaseException(Exception):
|
|
25
36
|
def __init__(self, original: BaseException):
|
|
@@ -110,10 +121,10 @@ def _validate_heartbeat_interval_seconds(
|
|
|
110
121
|
"'heartbeat_interval_seconds' requires a configured visibility timeout."
|
|
111
122
|
)
|
|
112
123
|
raise ValueError(require_visibility_timeout_message)
|
|
113
|
-
if heartbeat_interval_seconds
|
|
124
|
+
if heartbeat_interval_seconds >= visibility_timeout_seconds / 2:
|
|
114
125
|
raise ValueError(
|
|
115
|
-
"'heartbeat_interval_seconds' must be
|
|
116
|
-
f"({heartbeat_interval_seconds}
|
|
126
|
+
"'heartbeat_interval_seconds' must be less than half of 'visibility_timeout_seconds' "
|
|
127
|
+
f"({heartbeat_interval_seconds} >= {visibility_timeout_seconds / 2})"
|
|
117
128
|
)
|
|
118
129
|
return heartbeat_interval_seconds
|
|
119
130
|
|
|
@@ -385,7 +396,6 @@ class RedisMessageQueue:
|
|
|
385
396
|
raise TypeError(f"'gateway' must be an AbstractRedisGateway, got {type(gateway).__name__}")
|
|
386
397
|
gateway_visibility_timeout_seconds = _get_optional_gateway_visibility_timeout_seconds(gateway)
|
|
387
398
|
self._requires_claimed_message = gateway_visibility_timeout_seconds is not None
|
|
388
|
-
_bind_dead_letter_gateway_to_queue(gateway, self.key.pending)
|
|
389
399
|
_validate_cluster_configuration(self.key, gateway=gateway)
|
|
390
400
|
if heartbeat_interval_seconds is not None:
|
|
391
401
|
gateway_visibility_timeout_seconds = _get_gateway_visibility_timeout_seconds(gateway)
|
|
@@ -402,6 +412,7 @@ class RedisMessageQueue:
|
|
|
402
412
|
"'max_delivery_count' cannot be provided alongside 'gateway'."
|
|
403
413
|
" Configure 'max_delivery_count' and 'dead_letter_queue' on the gateway directly instead."
|
|
404
414
|
)
|
|
415
|
+
_bind_dead_letter_gateway_to_queue(gateway, self.key.pending)
|
|
405
416
|
self._redis = gateway
|
|
406
417
|
elif client is None:
|
|
407
418
|
raise ValueError("Either 'client' or 'gateway' must be provided.")
|
|
@@ -535,11 +546,7 @@ class RedisMessageQueue:
|
|
|
535
546
|
self._remove_processed_message(stored_message, lease_token)
|
|
536
547
|
)
|
|
537
548
|
if lease_token is not None and not applied:
|
|
538
|
-
logger.warning(
|
|
539
|
-
"Message cleanup after failed processing was a no-op: "
|
|
540
|
-
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
541
|
-
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
542
|
-
)
|
|
549
|
+
logger.warning(_STALE_LEASE_NACK_WARNING)
|
|
543
550
|
except BaseException:
|
|
544
551
|
logger.exception("Failed to clean up message from processing queue")
|
|
545
552
|
raise
|
|
@@ -555,11 +562,7 @@ class RedisMessageQueue:
|
|
|
555
562
|
self._remove_processed_message(stored_message, lease_token)
|
|
556
563
|
)
|
|
557
564
|
if lease_token is not None and not applied:
|
|
558
|
-
logger.warning(
|
|
559
|
-
"Message cleanup after successful processing was a no-op: "
|
|
560
|
-
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
561
|
-
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
562
|
-
)
|
|
565
|
+
logger.warning(_STALE_LEASE_ACK_WARNING)
|
|
563
566
|
finished_without_error = True
|
|
564
567
|
finally:
|
|
565
568
|
if lease_heartbeat is not None:
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import os
|
|
2
2
|
import signal
|
|
3
|
+
import sys
|
|
3
4
|
from typing import Iterable
|
|
4
5
|
|
|
5
6
|
from redis_message_queue.interrupt_handler._interface import (
|
|
@@ -70,6 +71,7 @@ class GracefulInterruptHandler(BaseGracefulInterruptHandler):
|
|
|
70
71
|
raise ValueError(
|
|
71
72
|
f"Signal {sig.name} already has a non-default handler installed."
|
|
72
73
|
" GracefulInterruptHandler refuses to replace existing handlers."
|
|
74
|
+
" If running inside asyncio.run(), create the handler before asyncio.run() starts."
|
|
73
75
|
)
|
|
74
76
|
self._interrupted = False
|
|
75
77
|
self._verbose = verbose
|
|
@@ -91,6 +93,9 @@ class GracefulInterruptHandler(BaseGracefulInterruptHandler):
|
|
|
91
93
|
return
|
|
92
94
|
os.kill(os.getpid(), signum)
|
|
93
95
|
return
|
|
94
|
-
if self._verbose:
|
|
95
|
-
print(f"Received signal: {signal.strsignal(signum)}")
|
|
96
96
|
self._interrupted = True
|
|
97
|
+
if self._verbose:
|
|
98
|
+
try:
|
|
99
|
+
print(f"Received signal: {signal.strsignal(signum)}", file=sys.stderr)
|
|
100
|
+
except Exception:
|
|
101
|
+
pass
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/redis_message_queue.py
RENAMED
|
@@ -20,6 +20,17 @@ from redis_message_queue.interrupt_handler import BaseGracefulInterruptHandler
|
|
|
20
20
|
logger = logging.getLogger(__name__)
|
|
21
21
|
_GATEWAY_BOUND_PENDING_QUEUE_ATTR = "_rmq_bound_pending_queue"
|
|
22
22
|
|
|
23
|
+
_STALE_LEASE_ACK_WARNING = (
|
|
24
|
+
"Message cleanup after successful processing was a no-op: "
|
|
25
|
+
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
26
|
+
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
27
|
+
)
|
|
28
|
+
_STALE_LEASE_NACK_WARNING = (
|
|
29
|
+
"Message cleanup after failed processing was a no-op: "
|
|
30
|
+
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
31
|
+
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
32
|
+
)
|
|
33
|
+
|
|
23
34
|
|
|
24
35
|
def _validate_heartbeat_interval_seconds(
|
|
25
36
|
heartbeat_interval_seconds: int | float | None,
|
|
@@ -45,10 +56,10 @@ def _validate_heartbeat_interval_seconds(
|
|
|
45
56
|
"'heartbeat_interval_seconds' requires a configured visibility timeout."
|
|
46
57
|
)
|
|
47
58
|
raise ValueError(require_visibility_timeout_message)
|
|
48
|
-
if heartbeat_interval_seconds
|
|
59
|
+
if heartbeat_interval_seconds >= visibility_timeout_seconds / 2:
|
|
49
60
|
raise ValueError(
|
|
50
|
-
"'heartbeat_interval_seconds' must be
|
|
51
|
-
f"({heartbeat_interval_seconds}
|
|
61
|
+
"'heartbeat_interval_seconds' must be less than half of 'visibility_timeout_seconds' "
|
|
62
|
+
f"({heartbeat_interval_seconds} >= {visibility_timeout_seconds / 2})"
|
|
52
63
|
)
|
|
53
64
|
return heartbeat_interval_seconds
|
|
54
65
|
|
|
@@ -338,7 +349,6 @@ class RedisMessageQueue:
|
|
|
338
349
|
raise TypeError(f"'gateway' must be an AbstractRedisGateway, got {type(gateway).__name__}")
|
|
339
350
|
gateway_visibility_timeout_seconds = _get_optional_gateway_visibility_timeout_seconds(gateway)
|
|
340
351
|
self._requires_claimed_message = gateway_visibility_timeout_seconds is not None
|
|
341
|
-
_bind_dead_letter_gateway_to_queue(gateway, self.key.pending)
|
|
342
352
|
_validate_cluster_configuration(self.key, gateway=gateway)
|
|
343
353
|
if heartbeat_interval_seconds is not None:
|
|
344
354
|
gateway_visibility_timeout_seconds = _get_gateway_visibility_timeout_seconds(gateway)
|
|
@@ -355,6 +365,7 @@ class RedisMessageQueue:
|
|
|
355
365
|
"'max_delivery_count' cannot be provided alongside 'gateway'."
|
|
356
366
|
" Configure 'max_delivery_count' and 'dead_letter_queue' on the gateway directly instead."
|
|
357
367
|
)
|
|
368
|
+
_bind_dead_letter_gateway_to_queue(gateway, self.key.pending)
|
|
358
369
|
self._redis = gateway
|
|
359
370
|
elif client is None:
|
|
360
371
|
raise ValueError("Either 'client' or 'gateway' must be provided.")
|
|
@@ -391,7 +402,8 @@ class RedisMessageQueue:
|
|
|
391
402
|
``TypeError`` to avoid silent ``json.dumps`` coercion that would
|
|
392
403
|
collapse distinct keys into the same dedup key (e.g. ``{1: "x"}``
|
|
393
404
|
vs ``{"1": "x"}``). Only top-level keys are validated; nested
|
|
394
|
-
dicts follow ``json.dumps`` defaults.
|
|
405
|
+
dicts follow ``json.dumps`` defaults (e.g. nested non-string keys
|
|
406
|
+
are silently coerced: integer keys become strings).
|
|
395
407
|
"""
|
|
396
408
|
if not isinstance(message, (str, dict)):
|
|
397
409
|
raise TypeError(f"'message' must be a str or dict, got {type(message).__name__}")
|
|
@@ -492,11 +504,7 @@ class RedisMessageQueue:
|
|
|
492
504
|
else:
|
|
493
505
|
applied = self._remove_processed_message(stored_message, lease_token)
|
|
494
506
|
if lease_token is not None and not applied:
|
|
495
|
-
logger.warning(
|
|
496
|
-
"Message cleanup after failed processing was a no-op: "
|
|
497
|
-
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
498
|
-
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
499
|
-
)
|
|
507
|
+
logger.warning(_STALE_LEASE_NACK_WARNING)
|
|
500
508
|
except BaseException:
|
|
501
509
|
logger.exception("Failed to clean up message from processing queue")
|
|
502
510
|
raise
|
|
@@ -508,11 +516,7 @@ class RedisMessageQueue:
|
|
|
508
516
|
else:
|
|
509
517
|
applied = self._remove_processed_message(stored_message, lease_token)
|
|
510
518
|
if lease_token is not None and not applied:
|
|
511
|
-
logger.warning(
|
|
512
|
-
"Message cleanup after successful processing was a no-op: "
|
|
513
|
-
"the lease expired and the message was likely reclaimed by another consumer. "
|
|
514
|
-
"This is expected at-least-once delivery behavior under visibility timeout."
|
|
515
|
-
)
|
|
519
|
+
logger.warning(_STALE_LEASE_ACK_WARNING)
|
|
516
520
|
finally:
|
|
517
521
|
if lease_heartbeat is not None:
|
|
518
522
|
lease_heartbeat.stop()
|
|
File without changes
|
|
File without changes
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_callable_utils.py
RENAMED
|
File without changes
|
|
File without changes
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_queue_key_manager.py
RENAMED
|
File without changes
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_redis_cluster.py
RENAMED
|
File without changes
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/_stored_message.py
RENAMED
|
File without changes
|
{redis_message_queue-3.0.0 → redis_message_queue-3.1.0}/redis_message_queue/asyncio/__init__.py
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|