redis-message-queue 6.0.1__tar.gz → 7.0.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/PKG-INFO +302 -34
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/README.md +300 -32
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/pyproject.toml +2 -2
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/__init__.py +2 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_abstract_redis_gateway.py +7 -6
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_config.py +157 -23
- redis_message_queue-7.0.1/redis_message_queue/_event.py +66 -0
- redis_message_queue-7.0.1/redis_message_queue/_exceptions.py +67 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_queue_key_manager.py +27 -1
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_redis_cluster.py +9 -3
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_redis_gateway.py +52 -6
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/asyncio/__init__.py +2 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/asyncio/_abstract_redis_gateway.py +7 -6
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/asyncio/_redis_gateway.py +52 -6
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/asyncio/redis_message_queue.py +92 -40
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/interrupt_handler/_implementation.py +16 -3
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/redis_message_queue.py +81 -39
- redis_message_queue-6.0.1/redis_message_queue/_event.py +0 -39
- redis_message_queue-6.0.1/redis_message_queue/_exceptions.py +0 -36
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/LICENSE +0 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_callable_utils.py +0 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/_stored_message.py +0 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/interrupt_handler/__init__.py +0 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/interrupt_handler/_interface.py +0 -0
- {redis_message_queue-6.0.1 → redis_message_queue-7.0.1}/redis_message_queue/py.typed +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: redis-message-queue
|
|
3
|
-
Version:
|
|
3
|
+
Version: 7.0.1
|
|
4
4
|
Summary: Python message queuing with Redis and message deduplication
|
|
5
5
|
License: MIT
|
|
6
6
|
License-File: LICENSE
|
|
@@ -17,7 +17,7 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
17
17
|
Classifier: Programming Language :: Python :: 3.14
|
|
18
18
|
Classifier: Topic :: Software Development :: Libraries
|
|
19
19
|
Classifier: Topic :: System :: Distributed Computing
|
|
20
|
-
Requires-Dist: redis (>=5.0.0)
|
|
20
|
+
Requires-Dist: redis (>=5.0.0,<8.0.0)
|
|
21
21
|
Requires-Dist: tenacity (>=8.1.0)
|
|
22
22
|
Project-URL: Homepage, https://github.com/Elijas/redis-message-queue
|
|
23
23
|
Project-URL: Issues, https://github.com/Elijas/redis-message-queue/issues
|
|
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
|
|
|
26
26
|
|
|
27
27
|
# redis-message-queue
|
|
28
28
|
|
|
29
|
-
[](https://pypi.org/project/redis-message-queue)
|
|
30
30
|
[](https://pypistats.org/packages/redis-message-queue)
|
|
31
31
|
[](LICENSE)
|
|
32
32
|
[](https://github.com/Elijas/redis-message-queue/issues)
|
|
@@ -37,46 +37,50 @@ Description-Content-Type: text/markdown
|
|
|
37
37
|
**Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
|
|
38
38
|
|
|
39
39
|
```bash
|
|
40
|
-
pip install "redis-message-queue>=
|
|
40
|
+
pip install "redis-message-queue>=7.0.0,<8.0.0"
|
|
41
41
|
```
|
|
42
42
|
|
|
43
43
|
Requires Redis server >= 6.2.
|
|
44
44
|
|
|
45
45
|
## Quickstart
|
|
46
46
|
|
|
47
|
-
|
|
47
|
+
Redis must be running locally first: use `redis-server` or
|
|
48
|
+
`docker run -p 6379:6379 redis:7`.
|
|
48
49
|
|
|
49
50
|
```python
|
|
50
51
|
from redis import Redis
|
|
51
52
|
from redis_message_queue import RedisMessageQueue
|
|
52
53
|
|
|
53
54
|
client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
|
|
54
|
-
queue = RedisMessageQueue("
|
|
55
|
-
|
|
56
|
-
queue.
|
|
57
|
-
|
|
58
|
-
|
|
55
|
+
queue = RedisMessageQueue("quickstart", client=client, deduplication=True)
|
|
56
|
+
queue.publish("hello")
|
|
57
|
+
with queue.process_message() as message:
|
|
58
|
+
if message is not None:
|
|
59
|
+
print(f"got {message}")
|
|
60
|
+
# Expected output: got hello
|
|
59
61
|
```
|
|
60
62
|
|
|
61
|
-
|
|
63
|
+
`RedisMessageQueue` itself is not a context manager. Use
|
|
64
|
+
`with queue.process_message() as message:` for each message.
|
|
65
|
+
|
|
66
|
+
### Async quickstart
|
|
62
67
|
|
|
63
68
|
```python
|
|
64
|
-
|
|
65
|
-
from
|
|
69
|
+
import asyncio
|
|
70
|
+
from redis.asyncio import Redis
|
|
71
|
+
from redis_message_queue.asyncio import RedisMessageQueue
|
|
66
72
|
|
|
67
|
-
|
|
68
|
-
|
|
73
|
+
async def main():
|
|
74
|
+
client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
|
|
75
|
+
queue = RedisMessageQueue("quickstart", client=client, deduplication=True)
|
|
76
|
+
await queue.publish("hello")
|
|
77
|
+
async with queue.process_message() as message:
|
|
78
|
+
print(f"got {message}")
|
|
79
|
+
await client.aclose()
|
|
69
80
|
|
|
70
|
-
|
|
71
|
-
with queue.process_message() as message:
|
|
72
|
-
if message is not None:
|
|
73
|
-
print(f"Processing: {message}")
|
|
74
|
-
# Auto-acknowledged on success; cleaned up on exception
|
|
81
|
+
asyncio.run(main()) # Expected output: got hello
|
|
75
82
|
```
|
|
76
83
|
|
|
77
|
-
`RedisMessageQueue` itself is not a context manager. Use
|
|
78
|
-
`with queue.process_message() as message:` for each message.
|
|
79
|
-
|
|
80
84
|
## Why redis-message-queue
|
|
81
85
|
|
|
82
86
|
**The problem:** You're sending messages between services or workers and need guarantees. Simple Redis LPUSH/BRPOP loses messages on crashes, doesn't deduplicate, and gives you no visibility into what succeeded or failed.
|
|
@@ -124,6 +128,29 @@ queue = RedisMessageQueue(
|
|
|
124
128
|
queue = RedisMessageQueue("q", client=client, deduplication=False)
|
|
125
129
|
```
|
|
126
130
|
|
|
131
|
+
#### Custom dedup key callable must return a non-empty, high-cardinality, tenant-scoped string
|
|
132
|
+
|
|
133
|
+
When `get_deduplication_key` is a callable, it is called once per publish and
|
|
134
|
+
must return a `str` that uniquely represents the deduplication scope for that
|
|
135
|
+
message. Returning `None` or `""` raises `ConfigurationError` at publish time;
|
|
136
|
+
returning a non-`str` value raises `TypeError`.
|
|
137
|
+
|
|
138
|
+
Use stable, high-cardinality keys that include any tenant or account boundary
|
|
139
|
+
needed by your system:
|
|
140
|
+
|
|
141
|
+
```python
|
|
142
|
+
queue = RedisMessageQueue(
|
|
143
|
+
"orders",
|
|
144
|
+
client=client,
|
|
145
|
+
deduplication=True,
|
|
146
|
+
get_deduplication_key=lambda msg: f"{msg['tenant_id']}:{msg['order_id']}",
|
|
147
|
+
)
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
Avoid fallback patterns such as `lambda msg: msg.get("order_id", "")`.
|
|
151
|
+
Missing fields should fail loudly instead of collapsing unrelated messages into
|
|
152
|
+
one deduplication key.
|
|
153
|
+
|
|
127
154
|
### Success and failure tracking
|
|
128
155
|
|
|
129
156
|
```python
|
|
@@ -260,6 +287,22 @@ consumer whose claim request Redis executes next. There is no round-robin,
|
|
|
260
287
|
equal-share, or starvation-freedom guarantee; faster consumers can receive more
|
|
261
288
|
than 1/N of messages.
|
|
262
289
|
|
|
290
|
+
### If you need stronger ordering or fairness guarantees
|
|
291
|
+
|
|
292
|
+
- **Strict queue-wide processing order** — use a single consumer per queue.
|
|
293
|
+
Multiple consumers will interleave handler completions.
|
|
294
|
+
- **Per-key processing order** — partition by key into multiple queues
|
|
295
|
+
(`queue_<hash(key) % N>`), and consume each partition with a single consumer.
|
|
296
|
+
- **Equal-share / round-robin fairness across consumers** — choose a different
|
|
297
|
+
scheduler. This queue does not guarantee that any individual consumer makes
|
|
298
|
+
forward progress at any specific rate.
|
|
299
|
+
- **Cross-batch ordering after reclaim** — accept that reclaimed messages will
|
|
300
|
+
reappear after newer un-reclaimed messages have been consumed. If your handler
|
|
301
|
+
must observe original publish order, persist that order in the payload (for
|
|
302
|
+
example, a sequence number set by the producer). For clock-related operator
|
|
303
|
+
detail behind reclaim behavior, see
|
|
304
|
+
[production readiness R11](docs/production-readiness.md#r11-redis-clock-dependencies).
|
|
305
|
+
|
|
263
306
|
### Dead-letter queue
|
|
264
307
|
|
|
265
308
|
```python
|
|
@@ -310,14 +353,14 @@ There are three distinct shutdown shapes; pick the one that matches your runtime
|
|
|
310
353
|
|---|---|---|---|
|
|
311
354
|
| **Flag-based soft drain** (`GracefulInterruptHandler`) | First SIGINT/SIGTERM flips a flag | Runs to completion | Drained on the next claim call, not on signal arrival |
|
|
312
355
|
| **Async task cancellation** (`asyncio.CancelledError`) | Framework cancels the worker task (Uvicorn/K8s SIGTERM in many setups) | **Hard abort** — message stays in `processing`; with VT it is reclaimed at deadline expiry, without VT it is orphaned | Not drained |
|
|
313
|
-
| **Explicit drain** (`drain()` / `aclose()`) | You call the method | Caller's responsibility to let it finish (drain does **not** cancel) | Drained synchronously via the gateway recovery path |
|
|
356
|
+
| **Explicit drain** (`drain()` / `aclose()`) | You call the method | Caller's responsibility to let it finish (drain does **not** cancel) | Drained synchronously via the gateway recovery path; new publishes are refused |
|
|
314
357
|
|
|
315
358
|
Use `drain()` / `aclose()` to bridge K8s `preStop` / SIGTERM grace windows without
|
|
316
359
|
relying on signal interception:
|
|
317
360
|
|
|
318
361
|
```python
|
|
319
362
|
# sync — in your SIGTERM handler or preStop hook
|
|
320
|
-
queue.drain(timeout=25) # refuses new claims, recovers pending claim IDs
|
|
363
|
+
queue.drain(timeout=25) # refuses new publishes/claims, recovers pending claim IDs
|
|
321
364
|
worker_thread.join() # wait for in-flight process_message to finish
|
|
322
365
|
|
|
323
366
|
# async — same shape
|
|
@@ -326,12 +369,19 @@ await worker_task # task observes ``_draining`` and exits its loop
|
|
|
326
369
|
```
|
|
327
370
|
|
|
328
371
|
`drain()` / `aclose()` set a queue-local flag so subsequent `process_message()`
|
|
329
|
-
calls yield `None` immediately
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
372
|
+
calls yield `None` immediately and subsequent `publish()` calls raise
|
|
373
|
+
`QueueDrainedError("queue is drained")`. Drain also gates the publish path:
|
|
374
|
+
if a publish is already inside the queue instance's publish path, drain waits
|
|
375
|
+
for that publish to finish before it returns; publishes that arrive after the
|
|
376
|
+
drained flag is set are rejected. The drained state is local to that Python
|
|
377
|
+
queue object and is not written to Redis, so constructing a fresh
|
|
378
|
+
`RedisMessageQueue(...)` over the same keys remains usable.
|
|
379
|
+
|
|
380
|
+
Drain does not cancel in-flight handlers — the caller must arrange handler
|
|
381
|
+
exit through normal thread/task coordination. Returns `True` if all in-memory
|
|
382
|
+
pending claim IDs were recovered within the timeout; `False` if the deadline
|
|
383
|
+
fired or transient Redis errors left claim IDs pending (call again to retry).
|
|
384
|
+
`timeout=0` reports current state without attempting recovery.
|
|
335
385
|
|
|
336
386
|
> **Heartbeat caveat (best-effort stop):** when `heartbeat_interval_seconds` is
|
|
337
387
|
> set, the heartbeat sidecar's `stop()` is bounded but not strictly quiescent —
|
|
@@ -472,6 +522,32 @@ Notes:
|
|
|
472
522
|
message. Do not call `fork()` from inside active message handlers unless the
|
|
473
523
|
child exits without using the inherited queue/client.
|
|
474
524
|
|
|
525
|
+
#### Forking after constructing GracefulInterruptHandler
|
|
526
|
+
|
|
527
|
+
If your application constructed `GracefulInterruptHandler` in the parent process
|
|
528
|
+
before `os.fork()` (for example, via module import in a pre-fork app server),
|
|
529
|
+
forked children cannot construct a fresh handler for the same signal because the
|
|
530
|
+
inherited signal table still routes to the parent-process handler.
|
|
531
|
+
|
|
532
|
+
In each child process, call `parent_handler.reset()` before constructing a fresh
|
|
533
|
+
handler:
|
|
534
|
+
|
|
535
|
+
```python
|
|
536
|
+
def worker_main():
|
|
537
|
+
# Inherited handler from parent - reset it.
|
|
538
|
+
if shared.interrupt_handler is not None:
|
|
539
|
+
shared.interrupt_handler.reset()
|
|
540
|
+
|
|
541
|
+
# Now safe to construct a fresh handler for this child.
|
|
542
|
+
interrupt = GracefulInterruptHandler()
|
|
543
|
+
queue = RedisMessageQueue("jobs", client=redis.Redis(), interrupt=interrupt)
|
|
544
|
+
...
|
|
545
|
+
```
|
|
546
|
+
|
|
547
|
+
Alternatively, defer all construction (handler and queue) to inside
|
|
548
|
+
`worker_main()` and pass `--no-preload` (or equivalent) to your app server. That
|
|
549
|
+
avoids the parent-construct hazard entirely.
|
|
550
|
+
|
|
475
551
|
### Redis memory sizing for deduplication and replay metadata
|
|
476
552
|
|
|
477
553
|
When deduplication is enabled, each distinct dedup key creates one Redis string
|
|
@@ -530,6 +606,7 @@ Package logs remain diagnostic; use `on_event` rather than log parsing for
|
|
|
530
606
|
metrics.
|
|
531
607
|
|
|
532
608
|
```python
|
|
609
|
+
from opentelemetry import trace
|
|
533
610
|
from prometheus_client import Counter
|
|
534
611
|
from redis_message_queue import QueueEvent, RedisMessageQueue
|
|
535
612
|
|
|
@@ -538,22 +615,112 @@ events_total = Counter(
|
|
|
538
615
|
"redis-message-queue lifecycle events",
|
|
539
616
|
["queue", "operation", "outcome", "exception_type"],
|
|
540
617
|
)
|
|
618
|
+
SPAN_SINK_TRUSTED = False
|
|
541
619
|
|
|
542
620
|
def observe(event: QueueEvent) -> None:
|
|
543
621
|
events_total.labels(
|
|
544
622
|
event.queue, event.operation, event.outcome, event.exception_type or ""
|
|
545
623
|
).inc()
|
|
624
|
+
if event.error is not None and SPAN_SINK_TRUSTED:
|
|
625
|
+
trace.get_current_span().record_exception(event.error)
|
|
546
626
|
|
|
547
627
|
queue = RedisMessageQueue("jobs", client=client, on_event=observe)
|
|
548
628
|
```
|
|
549
629
|
|
|
630
|
+
#### ⚠ Secrets in `event.error`
|
|
631
|
+
|
|
632
|
+
`event.error` is the actual exception object — it retains the exception
|
|
633
|
+
message, `__cause__` chain, and traceback. These can contain sensitive content:
|
|
634
|
+
Redis credentials in connection-error messages, message payloads in handler
|
|
635
|
+
exceptions, environment values in stack-frame locals.
|
|
636
|
+
|
|
637
|
+
When exporting to telemetry sinks (OpenTelemetry, Sentry, Datadog), prefer the
|
|
638
|
+
redaction-friendly `event.exception_type` for metrics and labels. Use
|
|
639
|
+
`event.error` for full structured error data ONLY if your sink is
|
|
640
|
+
trust-equivalent to your application logs and is access-controlled.
|
|
641
|
+
|
|
642
|
+
Recommended pattern:
|
|
643
|
+
|
|
644
|
+
```python
|
|
645
|
+
def on_event(event: QueueEvent) -> None:
|
|
646
|
+
# Metric labels — always safe (just the exception class name)
|
|
647
|
+
metric_counter.labels(
|
|
648
|
+
operation=event.operation,
|
|
649
|
+
outcome=event.outcome,
|
|
650
|
+
exception_type=event.exception_type or "none",
|
|
651
|
+
).inc()
|
|
652
|
+
|
|
653
|
+
# Full exception — only if your span sink is trusted
|
|
654
|
+
if event.error is not None and SPAN_SINK_TRUSTED:
|
|
655
|
+
span.record_exception(event.error)
|
|
656
|
+
```
|
|
657
|
+
|
|
658
|
+
#### Event dispatch context
|
|
659
|
+
|
|
660
|
+
Callbacks fire inline:
|
|
661
|
+
|
|
662
|
+
- **Sync queue:** the callback runs in the caller's thread. It sees
|
|
663
|
+
contextvars, the OpenTelemetry current span, and structlog contextvars bound
|
|
664
|
+
by the caller.
|
|
665
|
+
- **Async queue:** the callback is awaited in the current asyncio task. It has
|
|
666
|
+
the same contextvars, span, and structlog visibility.
|
|
667
|
+
- **Sync heartbeat:** heartbeat events fire from a separate
|
|
668
|
+
`threading.Thread`. That thread does not inherit caller contextvars or the
|
|
669
|
+
caller's OpenTelemetry current span. Use `event.message_id` and
|
|
670
|
+
`event.lease_token_hash` for correlation.
|
|
671
|
+
- **Async heartbeat:** heartbeat events fire from an asyncio task. The task
|
|
672
|
+
copies the context present when the heartbeat was started, so contextvars and
|
|
673
|
+
OpenTelemetry spans bound at handler entry are visible.
|
|
674
|
+
|
|
675
|
+
#### Event timing vs. Redis commit
|
|
676
|
+
|
|
677
|
+
Most events are post-commit, emitted after the Redis command or Lua script
|
|
678
|
+
returned: `publish/success`, `publish_dedup_hit`, `claim/success`,
|
|
679
|
+
`claim_empty`, `claim_reclaim`, `ack`, `nack`, `completed`, `dlq`,
|
|
680
|
+
`lease_renew`, `trim_failed`, and `stale_lease_*`.
|
|
681
|
+
|
|
682
|
+
Pre-commit and mid-flight exceptions:
|
|
683
|
+
|
|
684
|
+
- `failed/failure` fires after the handler raises but before failed-queue
|
|
685
|
+
cleanup completes. Use `nack` for cleanup-commit metrics; use `failed` for
|
|
686
|
+
handler-exception attribution.
|
|
687
|
+
- `retry_attempt/failure` and `retry_exhausted` fire on the claim-loop retry
|
|
688
|
+
path. The first Redis attempt may or may not have committed.
|
|
689
|
+
- `publish/failure`, `claim/failure`, and `cleanup_failed/failure` follow
|
|
690
|
+
exceptions. Under an ambiguous lost response, Redis may have committed
|
|
691
|
+
despite the exception. Treat them as "operation did not succeed from the
|
|
692
|
+
caller's perspective", not "Redis did not commit".
|
|
693
|
+
|
|
694
|
+
#### Intentionally silent paths
|
|
695
|
+
|
|
696
|
+
The following operations have no `on_event` surface by design:
|
|
697
|
+
|
|
698
|
+
- **B1 Cluster `pcall` cleanup failure:** three lease-aware Lua scripts wrap a
|
|
699
|
+
data-derived `DEL` in `redis.pcall(...)` and ignore the result. This
|
|
700
|
+
preserves queue safety on Cluster `CROSSSLOT` rejection but cannot be
|
|
701
|
+
observed through `on_event`. Operators watching key-TTL behavior or Redis
|
|
702
|
+
slow logs can detect orphans.
|
|
703
|
+
- **VT claim-store OOM compensation:** if the visibility-timeout Lua script
|
|
704
|
+
cannot store the claim result, it removes the message from processing, pushes
|
|
705
|
+
it back to pending, and returns `false`. Python translates that into
|
|
706
|
+
`claim_empty/skipped`, the same shape as an empty poll. This is intentional
|
|
707
|
+
fail-safe behavior; the message is not lost.
|
|
708
|
+
- **`drain()` / `close()` / `aclose()` lifecycle:** explicit shutdown
|
|
709
|
+
operations do not emit lifecycle events. Pending-claim-drain recovery work
|
|
710
|
+
counts as `claim_reclaim` events when reached.
|
|
711
|
+
- **Non-claim-loop retry attempts:** tenacity retries in deduplicated publish,
|
|
712
|
+
ack/remove, move-to-completed/failed, and lease renewal collapse into the
|
|
713
|
+
terminal operation's failure event. There is no per-attempt event for those
|
|
714
|
+
paths.
|
|
715
|
+
|
|
550
716
|
The public exception hierarchy is rooted at `RedisMessageQueueError`.
|
|
551
717
|
Configuration value/combinations raise `ConfigurationError` (also a
|
|
552
718
|
`ValueError`), custom gateway contract violations raise `GatewayContractError`
|
|
553
719
|
(also a `TypeError`), and Lua `redis.error_reply(...)` failures raise
|
|
554
720
|
`LuaScriptError` (also a redis-py `ResponseError`). Publish overload raises
|
|
555
|
-
`QueueBackpressureError
|
|
556
|
-
|
|
721
|
+
`QueueBackpressureError`; publish after explicit drain raises
|
|
722
|
+
`QueueDrainedError`. `CleanupFailedError` and `RetryBudgetExhaustedError` are
|
|
723
|
+
reserved categories for cleanup and retry surfaces.
|
|
557
724
|
|
|
558
725
|
## Known limitations
|
|
559
726
|
|
|
@@ -564,13 +731,111 @@ are reserved categories for cleanup and retry surfaces.
|
|
|
564
731
|
- **Cluster detection uses `isinstance(client, RedisCluster)`.** Wrapped or instrumented cluster clients that delegate without inheriting will bypass hash-tag validation. Custom gateways should set `is_redis_cluster = True` explicitly.
|
|
565
732
|
- **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
|
|
566
733
|
- **Non-ASCII payloads use ~2x storage.** The default `ensure_ascii=True` in JSON serialization encodes non-ASCII characters as `\uXXXX` escape sequences. This is a deliberate compatibility choice.
|
|
567
|
-
- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH` by default, or a single non-idempotent Lua enqueue when `max_pending_length` is set: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice.
|
|
734
|
+
- **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` or `redis.asyncio.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH` by default, or a single non-idempotent Lua enqueue when `max_pending_length` is set: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. redis-py 6.0+ changed the default standalone `Redis()` / `redis.asyncio.Redis()` retry policy from `None` (no retry) to a 3-attempt `ExponentialWithJitterBackoff`; pass `retry=None` explicitly if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent enqueue path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
|
|
735
|
+
|
|
736
|
+
```python
|
|
737
|
+
import redis
|
|
738
|
+
from redis_message_queue import RedisMessageQueue
|
|
739
|
+
|
|
740
|
+
# Strict at-most-once for non-dedup messages: disable redis-py's
|
|
741
|
+
# default 3-retry policy explicitly.
|
|
742
|
+
client = redis.Redis(retry=None)
|
|
743
|
+
queue = RedisMessageQueue("jobs", client=client)
|
|
744
|
+
```
|
|
745
|
+
|
|
746
|
+
```python
|
|
747
|
+
import redis.asyncio as redis
|
|
748
|
+
from redis_message_queue.asyncio import RedisMessageQueue
|
|
749
|
+
|
|
750
|
+
# Strict at-most-once for non-dedup messages: disable redis-py's
|
|
751
|
+
# default 3-retry policy explicitly.
|
|
752
|
+
client = redis.Redis(retry=None)
|
|
753
|
+
queue = RedisMessageQueue("jobs", client=client)
|
|
754
|
+
```
|
|
568
755
|
- **Redis Cluster default retry can stack with this library's retry budget.** In redis-py 6.0+, `RedisCluster()` constructs a default `ExponentialWithJitterBackoff` retry below this library's `retry_budget_seconds`. If you need a single retry surface, pass `retry=Retry(NoBackoff(), 0)` to the cluster client or reduce `retry_budget_seconds` to account for the lower-level retry window.
|
|
569
756
|
|
|
570
757
|
For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
|
|
571
758
|
|
|
572
759
|
## Upgrading
|
|
573
760
|
|
|
761
|
+
### v6 to v7 migration
|
|
762
|
+
|
|
763
|
+
v7.0.0 has four breaking changes to check during upgrade.
|
|
764
|
+
|
|
765
|
+
**AC-02: queue event operation/outcome types are `StrEnum` members.**
|
|
766
|
+
Runtime string comparisons keep working because `StrEnum` subclasses `str`,
|
|
767
|
+
but type-strict code should replace old `Literal[...]` annotations and raw
|
|
768
|
+
string constants with enum members.
|
|
769
|
+
|
|
770
|
+
Before:
|
|
771
|
+
|
|
772
|
+
```python
|
|
773
|
+
from typing import Literal
|
|
774
|
+
|
|
775
|
+
QueueOperation = Literal["publish", "claim", "ack"]
|
|
776
|
+
|
|
777
|
+
def record(operation: QueueOperation) -> None:
|
|
778
|
+
if operation == "publish":
|
|
779
|
+
print("published")
|
|
780
|
+
```
|
|
781
|
+
|
|
782
|
+
After:
|
|
783
|
+
|
|
784
|
+
```python
|
|
785
|
+
from redis_message_queue import EventOperation
|
|
786
|
+
|
|
787
|
+
def record(operation: EventOperation) -> None:
|
|
788
|
+
if operation is EventOperation.PUBLISH:
|
|
789
|
+
print("published")
|
|
790
|
+
```
|
|
791
|
+
|
|
792
|
+
**AC-03: drained queue instances refuse new publishes.** After
|
|
793
|
+
`queue.drain()` / `queue.close()` (sync) or `await queue.drain()` /
|
|
794
|
+
`await queue.aclose()` (async), the same queue instance rejects `publish()`
|
|
795
|
+
with `QueueDrainedError("queue is drained")`.
|
|
796
|
+
|
|
797
|
+
This state is queue-local and process-local; it is not stored in Redis. If a
|
|
798
|
+
producer must continue publishing after a worker has drained, use a separate
|
|
799
|
+
`RedisMessageQueue(...)` instance for that producer lifecycle. During
|
|
800
|
+
shutdown, catch `QueueDrainedError` only at boundaries where late publishes are
|
|
801
|
+
expected and safe to drop or reschedule.
|
|
802
|
+
|
|
803
|
+
```python
|
|
804
|
+
from redis_message_queue import QueueDrainedError
|
|
805
|
+
|
|
806
|
+
try:
|
|
807
|
+
queue.publish("late shutdown audit event")
|
|
808
|
+
except QueueDrainedError:
|
|
809
|
+
# The queue instance already began draining; drop or reschedule elsewhere.
|
|
810
|
+
pass
|
|
811
|
+
```
|
|
812
|
+
|
|
813
|
+
**AC-09: unsafe `drop_oldest` combinations now fail at construction.** These
|
|
814
|
+
configurations raise `ConfigurationError` before the queue or gateway is
|
|
815
|
+
created:
|
|
816
|
+
|
|
817
|
+
- `pending_overload_policy="drop_oldest"` with `max_pending_length=None`:
|
|
818
|
+
`drop_oldest requires max_pending_length to be set. Use a positive
|
|
819
|
+
max_pending_length to define what can be dropped, or use
|
|
820
|
+
pending_overload_policy='raise' or 'block' for unbounded queues.`
|
|
821
|
+
- `pending_overload_policy="drop_oldest"` with deduplication enabled or
|
|
822
|
+
`get_deduplication_key` configured:
|
|
823
|
+
`'pending_overload_policy=drop_oldest' cannot be used with deduplication
|
|
824
|
+
because dropped messages leave their deduplication keys in Redis, causing
|
|
825
|
+
future publishes of the same payload to be silently suppressed. Use 'raise'
|
|
826
|
+
or 'block' for deduplicated queues, or disable deduplication if 'drop_oldest'
|
|
827
|
+
is required.`
|
|
828
|
+
- `pending_overload_policy="drop_oldest"` with `max_delivery_count` set:
|
|
829
|
+
`drop_oldest is incompatible with max_delivery_count (set
|
|
830
|
+
max_delivery_count=None or pick another policy to avoid silent loss of
|
|
831
|
+
pending DLQ candidates). Use pending_overload_policy='raise' or 'block' when
|
|
832
|
+
dead-letter handling is required.`
|
|
833
|
+
|
|
834
|
+
**AC-16: redis-py is capped below 8.0.0.** The package dependency is
|
|
835
|
+
`redis>=5.0.0,<8.0.0` until redis-py 8 RESP3-default behavior is verified.
|
|
836
|
+
Users on redis-py 7.x and earlier are unaffected. If you installed a redis-py
|
|
837
|
+
8.0.0 beta explicitly, downgrade with `pip install "redis<8.0.0"`.
|
|
838
|
+
|
|
574
839
|
### Configuration changes on live queues
|
|
575
840
|
|
|
576
841
|
> **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
|
|
@@ -596,6 +861,9 @@ v6.0.0 is a non-breaking-defaults release that adds new public APIs. v5 code con
|
|
|
596
861
|
- `max_pending_length=N` caps pending-list depth; with `pending_overload_policy="raise"` (default) producers see `QueueBackpressureError` when the cap is hit; `"block"` waits up to `pending_overload_block_timeout_seconds`; `"drop_oldest"` evicts silently, so use it only when data loss is acceptable.
|
|
597
862
|
- `queue.drain(timeout=...)` (sync) and `await queue.aclose(timeout=...)` (async) are explicit graceful-shutdown hooks. They refuse new claims and recover pending claim IDs but do not cancel in-flight handlers; join or await your worker separately.
|
|
598
863
|
- `on_event=callback` receives a `QueueEvent` dataclass for every publish/claim/ack/reclaim/dedup/cleanup lifecycle event. Use it for metrics, tracing, and structured logging. See [`examples/production/observability.py`](examples/production/observability.py) for the adapter pattern.
|
|
864
|
+
- See [`examples/production/backpressure.py`](examples/production/backpressure.py) and [`examples/production/graceful_shutdown.py`](examples/production/graceful_shutdown.py) for sync production patterns, with async siblings under [`examples/production/asyncio/`](examples/production/asyncio/).
|
|
865
|
+
|
|
866
|
+
> When using a pre-fork app server (gunicorn `--preload`, uvicorn workers that import the app at master startup), call `make_queue()` from your worker startup hook - NOT at module import. See [Fork safety](#fork-safety-and-pre-fork-servers) for why.
|
|
599
867
|
|
|
600
868
|
**New constructor rejections:**
|
|
601
869
|
|