redis-message-queue 3.1.2__tar.gz → 6.0.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (27) hide show
  1. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/PKG-INFO +279 -20
  2. redis_message_queue-6.0.0/README.md +589 -0
  3. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/pyproject.toml +12 -1
  4. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/__init__.py +20 -0
  5. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_abstract_redis_gateway.py +70 -9
  6. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_config.py +163 -27
  7. redis_message_queue-6.0.0/redis_message_queue/_event.py +39 -0
  8. redis_message_queue-6.0.0/redis_message_queue/_exceptions.py +36 -0
  9. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_queue_key_manager.py +11 -4
  10. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_redis_cluster.py +9 -4
  11. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_redis_gateway.py +255 -16
  12. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_stored_message.py +31 -3
  13. redis_message_queue-6.0.0/redis_message_queue/asyncio/__init__.py +35 -0
  14. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/asyncio/_abstract_redis_gateway.py +70 -9
  15. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/asyncio/_redis_gateway.py +253 -19
  16. redis_message_queue-6.0.0/redis_message_queue/asyncio/redis_message_queue.py +1115 -0
  17. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/interrupt_handler/_implementation.py +9 -3
  18. redis_message_queue-6.0.0/redis_message_queue/py.typed +0 -0
  19. redis_message_queue-6.0.0/redis_message_queue/redis_message_queue.py +1077 -0
  20. redis_message_queue-3.1.2/README.md +0 -330
  21. redis_message_queue-3.1.2/redis_message_queue/asyncio/__init__.py +0 -6
  22. redis_message_queue-3.1.2/redis_message_queue/asyncio/redis_message_queue.py +0 -634
  23. redis_message_queue-3.1.2/redis_message_queue/redis_message_queue.py +0 -584
  24. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/LICENSE +0 -0
  25. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/_callable_utils.py +0 -0
  26. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/interrupt_handler/__init__.py +0 -0
  27. {redis_message_queue-3.1.2 → redis_message_queue-6.0.0}/redis_message_queue/interrupt_handler/_interface.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: redis-message-queue
3
- Version: 3.1.2
3
+ Version: 6.0.0
4
4
  Summary: Python message queuing with Redis and message deduplication
5
5
  License: MIT
6
6
  License-File: LICENSE
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
26
26
 
27
27
  # redis-message-queue
28
28
 
29
- [![PyPI Version](https://img.shields.io/badge/v3.1.2-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
29
+ [![PyPI Version](https://img.shields.io/badge/v6.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
30
30
  [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
31
31
  [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
32
32
  [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -37,7 +37,7 @@ Description-Content-Type: text/markdown
37
37
  **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
38
38
 
39
39
  ```bash
40
- pip install "redis-message-queue>=3.0.0,<4.0.0"
40
+ pip install "redis-message-queue>=6.0.0,<7.0.0"
41
41
  ```
42
42
 
43
43
  Requires Redis server >= 6.2.
@@ -50,7 +50,7 @@ Requires Redis server >= 6.2.
50
50
  from redis import Redis
51
51
  from redis_message_queue import RedisMessageQueue
52
52
 
53
- client = Redis.from_url("redis://localhost:6379/0")
53
+ client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
54
54
  queue = RedisMessageQueue("my_queue", client=client, deduplication=True)
55
55
 
56
56
  queue.publish("order:1234") # returns True
@@ -74,6 +74,9 @@ while True:
74
74
  # Auto-acknowledged on success; cleaned up on exception
75
75
  ```
76
76
 
77
+ `RedisMessageQueue` itself is not a context manager. Use
78
+ `with queue.process_message() as message:` for each message.
79
+
77
80
  ## Why redis-message-queue
78
81
 
79
82
  **The problem:** You're sending messages between services or workers and need guarantees. Simple Redis LPUSH/BRPOP loses messages on crashes, doesn't deduplicate, and gives you no visibility into what succeeded or failed.
@@ -97,8 +100,8 @@ All features are optional and can be enabled or disabled as needed.
97
100
 
98
101
  | Configuration | Delivery guarantee |
99
102
  |---|---|
100
- | Default (no visibility timeout) | **At-most-once** — a consumer crash loses the in-flight message |
101
- | With `visibility_timeout_seconds` | **At-least-once** — expired messages are reclaimed and redelivered |
103
+ | Default (`visibility_timeout_seconds=300`) | **At-least-once** — expired messages are reclaimed and redelivered |
104
+ | With `visibility_timeout_seconds=None` | **At-most-once** — a consumer crash loses the in-flight message |
102
105
 
103
106
  See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-timeout) for details and tradeoffs.
104
107
 
@@ -107,7 +110,7 @@ See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-tim
107
110
  ### Deduplication
108
111
 
109
112
  ```python
110
- # Default: deduplicate by full message content (1-hour TTL)
113
+ # Default: deduplicate by SHA-256 hash of canonical message content (1-hour TTL)
111
114
  queue = RedisMessageQueue("q", client=client, deduplication=True)
112
115
 
113
116
  # Custom dedup key (e.g., deduplicate by order ID only)
@@ -131,7 +134,8 @@ queue = RedisMessageQueue(
131
134
  )
132
135
  ```
133
136
 
134
- To prevent unbounded growth, cap the queue lengths:
137
+ Completed and failed tracking queues are capped at 1,000 entries by default
138
+ when enabled. Override the caps when you need a different retention window:
135
139
 
136
140
  ```python
137
141
  queue = RedisMessageQueue(
@@ -144,6 +148,45 @@ queue = RedisMessageQueue(
144
148
  ```
145
149
 
146
150
  When set, `LTRIM` is called after each message is moved to the completed/failed queue. This is best-effort cleanup — if the trim fails, the queue is slightly longer until the next successful trim.
151
+ Pass `max_completed_length=None` or `max_failed_length=None` explicitly if you
152
+ want unbounded tracking queues.
153
+
154
+ ### Publish backpressure
155
+
156
+ By default, the pending queue is unbounded (`max_pending_length=None`), matching
157
+ the v5 behavior. Set `max_pending_length` when producers can outrun consumers
158
+ and Redis memory must fail closed before the broker is exhausted:
159
+
160
+ ```python
161
+ queue = RedisMessageQueue(
162
+ "q",
163
+ client=client,
164
+ max_pending_length=100_000,
165
+ pending_overload_policy="raise", # "raise", "drop_oldest", or "block"
166
+ )
167
+ ```
168
+
169
+ The built-in Redis path checks pending depth and enqueues in the same Lua script,
170
+ so concurrent publishers cannot race above the configured cap. Overload policies:
171
+
172
+ - `raise` raises `QueueBackpressureError` and leaves the pending list unchanged.
173
+ - `drop_oldest` removes the oldest pending message (`RPOP`) before enqueueing the
174
+ new message. This is silent data loss by design; deduplication markers for
175
+ dropped messages are not removed, so a dropped duplicate may still be
176
+ suppressed until its dedup TTL expires.
177
+ - `block` retries the atomic check until space opens or
178
+ `pending_overload_block_timeout_seconds` elapses (default: 1.0), then raises
179
+ `QueueBackpressureError`.
180
+
181
+ These limits apply only to the pending list at publish time. They do not cap
182
+ messages already in `processing`, dead-letter queues, deduplication keys, or
183
+ replay metadata. `max_completed_length` and `max_failed_length` only bound the
184
+ completed/failed history lists. Size pending payload memory separately from the
185
+ dedup/replay metadata described in
186
+ [Redis memory sizing](#redis-memory-sizing-for-deduplication-and-replay-metadata).
187
+
188
+ When using `gateway=`, configure backpressure on the gateway directly, for
189
+ example `RedisGateway(redis_client=client, max_pending_length=100_000)`.
147
190
 
148
191
  ### Crash recovery with visibility timeout
149
192
 
@@ -180,6 +223,43 @@ The callback is **advisory** — it may fire briefly after a successful `process
180
223
 
181
224
  Without a visibility timeout, messages already moved to `processing` remain there indefinitely after a consumer crash and are not redelivered, even if the crash happened before your handler started running.
182
225
 
226
+ ### Ordering and multi-consumer fairness
227
+
228
+ The built-in queue is a shared-pull Redis list. Successful publishes push to the
229
+ left side of the pending list, and claims pop from the right side, so Redis
230
+ grants claims in enqueue order in the no-failure path.
231
+
232
+ This is a claim-order guarantee only. It is not a completion-order guarantee:
233
+ multiple consumers process concurrently, handlers can run for different
234
+ durations, and younger messages can finish before older messages.
235
+
236
+ With `visibility_timeout_seconds` enabled, expired messages from `processing`
237
+ are reclaimed before fresh pending work on the next consumer poll. A reclaimed
238
+ message may be delivered after younger messages were already processed, and may
239
+ be processed concurrently with a stale original handler if that handler keeps
240
+ running after its lease expires.
241
+
242
+ Expired reclaims are ordered by lease deadline within one reclaim batch.
243
+ `CLAIM_MESSAGE_WITH_VISIBILITY_TIMEOUT_LUA_SCRIPT` selects expired leases with
244
+ `ZRANGEBYSCORE ... LIMIT 0, 100` to bound Redis Lua execution time. When more
245
+ than 100 messages expire together, the next poll can append a later reclaim
246
+ batch at the claimable end of the pending list ahead of leftovers from the
247
+ previous batch, so cross-batch redelivery order is not guaranteed.
248
+
249
+ `max_delivery_count` can skip over poison messages during a claim poll by moving
250
+ over-limit messages to the dead-letter queue and returning a later pending
251
+ message. Deduplication is publish-side only: duplicate publishes are not
252
+ enqueued and therefore do not occupy a queue position.
253
+
254
+ Handler exceptions are not retries: the default behavior removes the message
255
+ from `processing`, or moves it to the failed queue when enabled. Redelivery is
256
+ for crash, stall, or stale-lease paths where cleanup does not complete.
257
+
258
+ Multiple consumers contend for the same queue. The next message goes to the
259
+ consumer whose claim request Redis executes next. There is no round-robin,
260
+ equal-share, or starvation-freedom guarantee; faster consumers can receive more
261
+ than 1/N of messages.
262
+
183
263
  ### Dead-letter queue
184
264
 
185
265
  ```python
@@ -191,14 +271,14 @@ queue = RedisMessageQueue(
191
271
  )
192
272
  ```
193
273
 
194
- When a message has been delivered more than `max_delivery_count` times (due to consumer crashes causing visibility-timeout reclaim), it is automatically routed to a dead-letter queue (`{name}::dead_letter`) instead of being redelivered. This prevents poison messages from cycling indefinitely.
274
+ When a message has been delivered more than `max_delivery_count` times (due to consumer crashes causing visibility-timeout reclaim), it is automatically routed to a dead-letter queue (`{name}::dlq`) instead of being redelivered. `max_delivery_count` defaults to `10` on the built-in `client=` path, with the DLQ name auto-derived from the queue name. This prevents poison messages from cycling indefinitely.
195
275
 
196
276
  Notes:
197
277
  - requires `visibility_timeout_seconds` to be set (poison messages are only a concern with VT reclaim)
198
278
  - the delivery count is tracked per-message in a Redis HASH and cleaned up on successful ack or move to completed/failed
199
279
  - the delivery count increments when Redis grants the claim/lease, not when your handler begins running. If a process exits after Redis claims a message, that claim still counts toward `max_delivery_count`
200
280
  - `max_delivery_count=1` means the message is delivered once; any reclaim routes it to the dead-letter queue
201
- - without `max_delivery_count`, messages are redelivered indefinitely (existing behavior)
281
+ - set `max_delivery_count=None` explicitly for unlimited redelivery
202
282
  - dead-lettered messages contain the **raw payload** only — the internal envelope (which carries a per-delivery UUID) is stripped before pushing to the DLQ, consistent with how completed/failed queues store messages. Two identical payloads dead-lettered separately are indistinguishable in the DLQ
203
283
 
204
284
  ### Graceful shutdown
@@ -224,10 +304,46 @@ while not interrupt.is_interrupted():
224
304
  > (for example, a second Ctrl+C raises `KeyboardInterrupt`). If you need multiple
225
305
  > shutdown hooks, use a single handler and fan out in your own code.
226
306
 
307
+ There are three distinct shutdown shapes; pick the one that matches your runtime:
308
+
309
+ | Shape | Trigger | In-flight handler | Pending claim IDs |
310
+ |---|---|---|---|
311
+ | **Flag-based soft drain** (`GracefulInterruptHandler`) | First SIGINT/SIGTERM flips a flag | Runs to completion | Drained on the next claim call, not on signal arrival |
312
+ | **Async task cancellation** (`asyncio.CancelledError`) | Framework cancels the worker task (Uvicorn/K8s SIGTERM in many setups) | **Hard abort** — message stays in `processing`; with VT it is reclaimed at deadline expiry, without VT it is orphaned | Not drained |
313
+ | **Explicit drain** (`drain()` / `aclose()`) | You call the method | Caller's responsibility to let it finish (drain does **not** cancel) | Drained synchronously via the gateway recovery path |
314
+
315
+ Use `drain()` / `aclose()` to bridge K8s `preStop` / SIGTERM grace windows without
316
+ relying on signal interception:
317
+
318
+ ```python
319
+ # sync — in your SIGTERM handler or preStop hook
320
+ queue.drain(timeout=25) # refuses new claims, recovers pending claim IDs
321
+ worker_thread.join() # wait for in-flight process_message to finish
322
+
323
+ # async — same shape
324
+ await queue.aclose(timeout=25)
325
+ await worker_task # task observes ``_draining`` and exits its loop
326
+ ```
327
+
328
+ `drain()` / `aclose()` set a queue-local flag so subsequent `process_message()`
329
+ calls yield `None` immediately. They do not cancel in-flight handlers — the
330
+ caller must arrange handler exit through normal thread/task coordination.
331
+ Returns `True` if all in-memory pending claim IDs were recovered within the
332
+ timeout; `False` if the deadline fired or transient Redis errors left claim
333
+ IDs pending (call again to retry). `timeout=0` reports current state without
334
+ attempting recovery.
335
+
336
+ > **Heartbeat caveat (best-effort stop):** when `heartbeat_interval_seconds` is
337
+ > set, the heartbeat sidecar's `stop()` is bounded but not strictly quiescent —
338
+ > a slow renewal in flight when `process_message` exits may still write to
339
+ > Redis after the caller believes shutdown is complete. The renewal is bounded
340
+ > by the configured visibility timeout and the lease token check on the Redis
341
+ > side, but plan for a small post-shutdown overlap rather than instant quiesce.
342
+
227
343
  ### Custom gateway
228
344
 
229
345
  ```python
230
- from redis_message_queue._redis_gateway import RedisGateway
346
+ from redis_message_queue import RedisGateway
231
347
 
232
348
  # Tune retry budget, dedup TTL, or wait interval
233
349
  gateway = RedisGateway(
@@ -244,17 +360,17 @@ queue = RedisMessageQueue("q", gateway=gateway)
244
360
 
245
361
  The retry knobs configure an internal `tenacity` strategy: exponential
246
362
  backoff with jitter, retry on transient Redis errors only, capped at
247
- `retry_budget_seconds`. The budget is wall-clock time from the first attempt (including attempt duration), not inter-attempt delay; a single attempt that takes longer than the budget results in zero retries. Setting `retry_budget_seconds=0` disables retry
363
+ `retry_budget_seconds`. The budget is monotonic elapsed time from the first attempt (including attempt duration), not inter-attempt delay; it is unaffected by Python-host NTP jumps. A single attempt that takes longer than the budget results in zero retries. Setting `retry_budget_seconds=0` disables retry
248
364
  entirely (single attempt; exceptions propagate). The library uses
249
365
  `retry_budget_seconds` to size the operation-result cache TTL automatically,
250
366
  so the previous footgun of an over-long retry budget out-living the cache
251
367
  and producing misleading "cleanup was a no-op" warnings is now structurally
252
- impossible. Note: tenacity may allow one additional attempt beyond the budget if the budget check passes at attempt start total wall-clock time can exceed `retry_budget_seconds` by the duration of that final attempt.
368
+ impossible. Note: tenacity may allow one additional attempt beyond the budget if the budget check passes at attempt start, so total monotonic elapsed time can exceed `retry_budget_seconds` by the duration of that final attempt.
253
369
 
254
370
  To plug in a different retry library (`backoff`, `asyncstdlib.retry`, or your
255
371
  own logic) or fundamentally different semantics, subclass
256
- `AbstractRedisGateway` from `redis_message_queue._abstract_redis_gateway`
257
- (or `redis_message_queue.asyncio._abstract_redis_gateway`) and override the
372
+ `AbstractRedisGateway` from `redis_message_queue` (or
373
+ `redis_message_queue.asyncio` for the async sibling) and override the
258
374
  operation methods directly.
259
375
 
260
376
  If your custom gateway uses visibility timeouts, it must expose a public
@@ -285,6 +401,16 @@ Use a separate gateway instance per queue when `max_delivery_count` is enabled.
285
401
  Dead-letter routing is gateway-scoped, so reusing the same gateway across different
286
402
  queues is rejected.
287
403
 
404
+ If you use Redis Sentinel, pass the Redis client returned by
405
+ `sentinel.master_for(name)` to `client=` or `RedisGateway(redis_client=...)`, not
406
+ the `sentinel` object itself.
407
+
408
+ ### Connection pool sizing
409
+
410
+ Each queue with `heartbeat_interval_seconds` set uses up to 2 simultaneous
411
+ connections: one for the main operation and one for heartbeat renewal. Size Redis
412
+ client pools with `max_connections >= 2 * number_of_queues + headroom`.
413
+
288
414
  ## Async API
289
415
 
290
416
  Replace the import to use the async variant — the API is identical:
@@ -293,6 +419,11 @@ Replace the import to use the async variant — the API is identical:
293
419
  from redis_message_queue.asyncio import RedisMessageQueue
294
420
  ```
295
421
 
422
+ The sync and async classes intentionally share names. In modules that use both,
423
+ alias the imports explicitly, for example
424
+ `from redis_message_queue import RedisMessageQueue as SyncRedisMessageQueue` and
425
+ `from redis_message_queue.asyncio import RedisMessageQueue as AsyncRedisMessageQueue`.
426
+
296
427
  All examples work the same way. Remember to close the connection when done:
297
428
 
298
429
  ```python
@@ -303,18 +434,138 @@ client = redis.Redis()
303
434
  await client.aclose()
304
435
  ```
305
436
 
437
+ For the sync Redis client, call `client.close()` during application shutdown when
438
+ you own the client lifecycle.
439
+
440
+ ## Production notes
441
+
442
+ ### Fork safety and pre-fork servers
443
+
444
+ Construct Redis clients and `RedisMessageQueue` instances after a process forks.
445
+ This is the recommended pattern for `multiprocessing`, `ProcessPoolExecutor`,
446
+ and pre-fork servers such as gunicorn with `--preload`.
447
+
448
+ ```python
449
+ def worker_main():
450
+ client = redis.Redis()
451
+ queue = RedisMessageQueue("jobs", client=client)
452
+ ...
453
+ ```
454
+
455
+ Avoid constructing a queue/client in a parent process and then using that same
456
+ object in forked children, especially if the parent has already run any Redis
457
+ command. The queue stores the user-provided Redis client and process-local
458
+ claim-recovery state. Inherited Redis sockets can corrupt the Redis protocol if
459
+ two processes use the same file descriptor.
460
+
461
+ Notes:
462
+
463
+ - The sync redis-py pooled client attempts to reset its connection pool after
464
+ fork, but this does not apply to every client shape.
465
+ - The built-in sync gateway rejects `redis.Redis(single_connection_client=True)`
466
+ because that mode pins one socket instead of using the pool.
467
+ - Do not share `redis.asyncio.Redis` or async queues across fork; create or
468
+ reconnect them in the child process.
469
+ - If you use `GracefulInterruptHandler`, create it in the worker process after
470
+ fork so signal ownership is local to that worker.
471
+ - The heartbeat sidecar is lazy and starts only while processing a leased
472
+ message. Do not call `fork()` from inside active message handlers unless the
473
+ child exits without using the inherited queue/client.
474
+
475
+ ### Redis memory sizing for deduplication and replay metadata
476
+
477
+ When deduplication is enabled, each distinct dedup key creates one Redis string
478
+ for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The default
479
+ dedup key is a SHA-256 hash of the canonical message payload, so distinct
480
+ payloads are distinct keys. Size Redis for:
481
+
482
+ ```text
483
+ peak_unique_publish_rate_per_second
484
+ * message_deduplication_log_ttl_seconds
485
+ * bytes_per_dedup_key
486
+ ```
487
+
488
+ Use 200 bytes per dedup key as a conservative starting point for short queue
489
+ names, then validate with `MEMORY USAGE` in your Redis version. Example:
490
+ 1,000 unique messages/s * 3,600s * 200 B ~= 720 MB for dedup markers alone.
491
+ A 24h dedup window at the same rate is 86.4M keys, or roughly 17 GB before
492
+ message payload lists, lease metadata, completed/failed queues, and allocator
493
+ fragmentation.
494
+
495
+ Operation-result replay keys are normally deleted after a successful call, but
496
+ may live until their TTL after ambiguous connection drops or failed cleanup
497
+ deletes. With visibility timeouts, active claims also store replay metadata
498
+ until ack or reclaim. Without visibility timeouts, abandoned claims leave
499
+ `claim_result_ids` and `claim_result_backrefs` fields until the message is
500
+ acked or manually cleaned.
501
+
502
+ `max_completed_length` and `max_failed_length` only bound the completed/failed
503
+ lists. They do not bound deduplication keys or replay metadata.
504
+
505
+ Avoid sharing queue Redis DBs with unrelated high-cardinality workloads. If
506
+ idempotency matters, prefer explicit capacity planning and `noeviction` with
507
+ alerts over LRU/random eviction policies: evicting dedup/replay keys before
508
+ their TTL can weaken duplicate suppression and retry result replay.
509
+
510
+ ## Observability
511
+
512
+ Queue instances accept an optional `on_event` callback for metrics, tracing, or
513
+ structured logging. The sync queue expects a regular callable; the async queue
514
+ expects an async callable:
515
+
516
+ ```python
517
+ from redis_message_queue import QueueEvent, RedisMessageQueue
518
+
519
+ def on_event(event: QueueEvent) -> None:
520
+ ...
521
+
522
+ queue = RedisMessageQueue("jobs", client=client, on_event=on_event)
523
+ ```
524
+
525
+ Events cover publish, dedup hits, claim/empty polls, reclaim, ack/nack,
526
+ completed/failed cleanup, DLQ moves, heartbeat renewal, stale leases, cleanup
527
+ and trim failures, and retry attempts. Callback exceptions are logged and
528
+ reported with `RuntimeWarning`, but never propagate into queue operations.
529
+ Package logs remain diagnostic; use `on_event` rather than log parsing for
530
+ metrics.
531
+
532
+ ```python
533
+ from prometheus_client import Counter
534
+ from redis_message_queue import QueueEvent, RedisMessageQueue
535
+
536
+ events_total = Counter(
537
+ "rmq_events_total",
538
+ "redis-message-queue lifecycle events",
539
+ ["queue", "operation", "outcome", "exception_type"],
540
+ )
541
+
542
+ def observe(event: QueueEvent) -> None:
543
+ events_total.labels(
544
+ event.queue, event.operation, event.outcome, event.exception_type or ""
545
+ ).inc()
546
+
547
+ queue = RedisMessageQueue("jobs", client=client, on_event=observe)
548
+ ```
549
+
550
+ The public exception hierarchy is rooted at `RedisMessageQueueError`.
551
+ Configuration value/combinations raise `ConfigurationError` (also a
552
+ `ValueError`), custom gateway contract violations raise `GatewayContractError`
553
+ (also a `TypeError`), and Lua `redis.error_reply(...)` failures raise
554
+ `LuaScriptError` (also a redis-py `ResponseError`). Publish overload raises
555
+ `QueueBackpressureError`. `CleanupFailedError` and `RetryBudgetExhaustedError`
556
+ are reserved categories for cleanup and retry surfaces.
557
+
306
558
  ## Known limitations
307
559
 
308
- - **No metrics or observability hooks.** The library logs warnings (stale leases, heartbeat failures, transient errors) via Python's `logging` module but does not expose callbacks, event hooks, or metric counters. To monitor queue health, inspect the underlying Redis keys directly or parse log output.
309
560
  - **Timed waits use polling claim loops.** To make claims recoverable after ambiguous connection drops, `wait_for_message_and_move()` uses idempotent Lua claim polling instead of raw blocking list-move commands. This adds a small polling cadence during timed waits.
310
561
  - **Redis Lua is atomic, not rollback-transactional.** The built-in scripts now preflight queue key types and fail closed on `WRONGTYPE` before mutating queue state, but Redis does not undo earlier writes if a later script command fails for another reason (for example `OOM` under severe memory pressure).
311
562
  - **Batch reclaim limit of 100.** The visibility-timeout reclaim Lua script processes at most 100 expired messages per consumer poll. Under extreme backlog this may delay recovery, but prevents any single poll from blocking Redis.
312
563
  - **Claim-attempt loop limit of 100 per poll.** The VT claim Lua script attempts at most 100 LMOVE+delivery-count checks per invocation. Under pathological conditions (>100 consecutive poison messages in pending), a single poll returns no message even though non-poison messages exist deeper in the queue. Subsequent polls drain the poison batch 100 at a time.
313
- - **Default dedup key is the full message.** Without a custom `get_deduplication_key`, the entire serialized message becomes a Redis key name for dedup tracking. For large messages (>1KB), provide a custom key function to avoid excessive Redis memory usage.
314
564
  - **Cluster detection uses `isinstance(client, RedisCluster)`.** Wrapped or instrumented cluster clients that delegate without inheriting will bypass hash-tag validation. Custom gateways should set `is_redis_cluster = True` explicitly.
315
565
  - **Redis Cluster requires hash tags.** The built-in queue uses multiple Redis keys per operation. Wrap the queue name in hash tags (for example `{myqueue}`) so every generated key lands in the same slot. When you pass a Redis Cluster client to the built-in queue/gateway path, incompatible names are rejected early.
316
566
  - **Non-ASCII payloads use ~2x storage.** The default `ensure_ascii=True` in JSON serialization encodes non-ASCII characters as `\uXXXX` escape sequences. This is a deliberate compatibility choice.
317
- - **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH`: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent `LPUSH` path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
567
+ - **Client-side `Retry` can duplicate non-deduplicated publishes.** If you construct your `redis.Redis` client with `retry=Retry(...)`, redis-py retries `ConnectionError` / `TimeoutError` at the connection layer — *below* this library. Idempotent operations (deduplicated `publish()`, lease-scoped cleanup) are safe because their Lua scripts replay the original result. `add_message()` (used by `publish()` when `deduplication=False`) is a bare `LPUSH` by default, or a single non-idempotent Lua enqueue when `max_pending_length` is set: this library deliberately does not retry it, but a client-level `Retry` will, and if the server executed the command before the response was lost the message is enqueued twice. Leave `retry=None` (the default) if you need strict at-most-once semantics for non-deduplicated publishes, or accept the duplication risk. More broadly, any non-idempotent enqueue path is vulnerable if the connection drops after server execution but before the client receives the response; all other built-in operations (deduplicated publish, lease-scoped ack/move, lease renewal) use replay markers and are safe under client-level `Retry`.
568
+ - **Redis Cluster default retry can stack with this library's retry budget.** In redis-py 6.0+, `RedisCluster()` constructs a default `ExponentialWithJitterBackoff` retry below this library's `retry_budget_seconds`. If you need a single retry surface, pass `retry=Retry(NoBackoff(), 0)` to the cluster client or reduce `retry_budget_seconds` to account for the lower-level retry window.
318
569
 
319
570
  For a full analysis, see [docs/production-readiness.md](docs/production-readiness.md).
320
571
 
@@ -324,9 +575,17 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
324
575
 
325
576
  > **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
326
577
 
327
- - **Do not change `key_separator` on a live queue.** All existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing separators.
328
- - **Do not switch from no-VT to VT with messages in processing.** Messages claimed by non-VT consumers have no lease deadline entries. VT-enabled consumers cannot reclaim them. Drain the processing queue first.
578
+ - **Do not change `name` or `key_separator` on a live queue.** Both settings define the Redis key namespace. Existing Redis keys become invisible to the new key scheme. Drain the queue completely before changing either value.
579
+ - **Do not rename `dead_letter_queue` on a live queue.** Existing DLQ records stay in the old list, while new failures route to the new list. Inspect or drain the old DLQ manually before switching names.
580
+ - **Do not toggle visibility timeout in either direction with messages in processing.** Messages claimed by non-VT consumers have no lease metadata, so VT-enabled consumers cannot reclaim them. Disabling VT later orphans existing lease deadline, lease token, and delivery count metadata and removes crash recovery for those in-flight messages. Drain the processing queue first.
329
581
  - **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
582
+ - **Changing `max_delivery_count` from a number to `None` leaves delivery metadata behind.** The delivery count hash continues to exist but is no longer consulted. Use this only after draining or after planning manual cleanup of the delivery-count hash.
583
+ - **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching between the default hash, explicit `None`, or a custom key function.
584
+ - **Disabling `deduplication` has a retention-window overlap.** Existing dedup records remain in Redis until their TTL expires, but new publishes bypass them. Republishes that would have been suppressed under the old setting can enqueue during that window.
585
+ - **Disabling `enable_failed_queue` stops recording handler failures.** Existing failed entries remain in Redis, but new failures are removed from `processing` without being appended to the failed queue. If `max_delivery_count=None` is also set, repeated handler failures can be dropped with no DLQ or failed-queue record; see [Dead-letter queue](#dead-letter-queue).
586
+ - **Lowering `max_completed_length` or `max_failed_length` trims existing history.** The next completed or failed move calls `LTRIM`, so changing `None` to `N` or lowering `N` can immediately reduce historical entries to the new cap.
587
+ - **Do not switch sync and async gateway instances mid-process while claims are active.** Redis state is compatible across deploys, but each gateway instance keeps its own pending claim-recovery IDs. In-flight claim recovery state does not transfer between instances.
588
+ - **Switching between `gateway=` and `client=` can retarget the DLQ.** The built-in `client=` path derives the DLQ from the queue name. If a custom gateway used a different `dead_letter_queue`, switching paths has the same orphaning impact as renaming the DLQ.
330
589
 
331
590
  ### v2 to v3 migration
332
591