redis-message-queue 7.0.0__tar.gz → 8.0.0__tar.gz

{redis_message_queue-7.0.0 → redis_message_queue-8.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 7.0.0
+Version: 8.0.0
 Summary: Python message queuing with Redis and message deduplication
 License: MIT
 License-File: LICENSE
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v7.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v8.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -37,46 +37,60 @@ Description-Content-Type: text/markdown
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=6.0.0,<7.0.0"
+pip install "redis-message-queue>=8.0.0,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
 
 ## Quickstart
 
-### Publish messages
+Redis must be running locally first: use `redis-server` or
+`docker run -p 6379:6379 redis:7`.
 
 ```python
 from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
 client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
-queue = RedisMessageQueue("my_queue", client=client, deduplication=True)
-
-queue.publish("order:1234")           # returns True
-queue.publish("order:1234")           # returns False (deduplicated)
-queue.publish({"user": "alice"})      # dicts work too
+queue = RedisMessageQueue(
+    "quickstart",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: msg["id"],
+)
+queue.publish({"id": "msg-1", "text": "hello"})
+with queue.process_message() as message:
+    if message is not None:
+        print(f"got {message['text']}")
+# Expected output: got hello
 ```
 
-### Consume messages
+`RedisMessageQueue` itself is not a context manager. Use
+`with queue.process_message() as message:` for each message.
 
-```python
-from redis import Redis
-from redis_message_queue import RedisMessageQueue
+### Async quickstart
 
-client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
-queue = RedisMessageQueue("my_queue", client=client)
+```python
+import asyncio
+from redis.asyncio import Redis
+from redis_message_queue.asyncio import RedisMessageQueue
 
-while True:
-    with queue.process_message() as message:
-        if message is not None:
-            print(f"Processing: {message}")
-            # Auto-acknowledged on success; cleaned up on exception
+async def main():
+    client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
+    queue = RedisMessageQueue(
+        "quickstart",
+        client=client,
+        deduplication=True,
+        get_deduplication_key=lambda msg: msg["id"],
+    )
+    await queue.publish({"id": "msg-1", "text": "hello"})
+    async with queue.process_message() as message:
+        print(f"got {message['text']}")
+    await client.aclose()
+
+asyncio.run(main())  # Expected output: got hello
 ```
 
-`RedisMessageQueue` itself is not a context manager. Use
-`with queue.process_message() as message:` for each message.
-
 ## Why redis-message-queue
 
 **The problem:** You're sending messages between services or workers and need guarantees. Simple Redis LPUSH/BRPOP loses messages on crashes, doesn't deduplicate, and gives you no visibility into what succeeded or failed.
@@ -85,7 +99,7 @@ while True:
 
 | Feature | Details |
 |---------|---------|
-| **Deduplicated publish** | Lua-scripted atomic SET NX + LPUSH prevents duplicate enqueues within a configurable TTL window (default: 1 hour), even with producer retries. Supports custom key functions for content-based deduplication. Note: deduplication is publish-side only and does not prevent duplicate *delivery* under at-least-once visibility-timeout reclaim |
+| **Deduplicated publish** | Lua-scripted atomic SET NX + LPUSH prevents duplicate enqueues within a configurable TTL window (default: 1 hour), even with producer retries. Requires an explicit `get_deduplication_key` callable so your application defines what counts as a duplicate. Note: deduplication is publish-side only and does not prevent duplicate *delivery* under at-least-once visibility-timeout reclaim |
 | **Visibility-timeout redelivery** | Crashed or stalled consumers' messages are reclaimed and redelivered when a visibility timeout is configured |
 | **Success & failure logs** | Optional completed/failed queues for auditing and reprocessing, with configurable max length to prevent unbounded growth |
 | **Dead-letter queue** | Poison messages that exceed a configurable delivery count are automatically routed to a dead-letter queue instead of being redelivered indefinitely |
@@ -110,10 +124,7 @@ See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-tim
 ### Deduplication
 
 ```python
-# Default: deduplicate by SHA-256 hash of canonical message content (1-hour TTL)
-queue = RedisMessageQueue("q", client=client, deduplication=True)
-
-# Custom dedup key (e.g., deduplicate by order ID only)
+# Deduplicate by order ID for a 1-hour TTL
 queue = RedisMessageQueue(
     "q", client=client,
     deduplication=True,
@@ -124,6 +135,30 @@ queue = RedisMessageQueue(
 queue = RedisMessageQueue("q", client=client, deduplication=False)
 ```
 
+#### Dedup key callable must return a non-empty, high-cardinality, tenant-scoped string
+
+When `deduplication=True`, `get_deduplication_key` is required. The callable is
+called once per publish and must return a `str` that uniquely represents the
+deduplication scope for that message. Returning `None` or `""` raises
+`ConfigurationError` at publish time; returning a non-`str` value raises
+`TypeError`.
+
+Use stable, high-cardinality keys that include any tenant or account boundary
+needed by your system:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: f"{msg['tenant_id']}:{msg['order_id']}",
+)
+```
+
+Avoid fallback patterns such as `lambda msg: msg.get("order_id", "")`.
+Missing fields should fail loudly instead of collapsing unrelated messages into
+one deduplication key.
+
 ### Success and failure tracking
 
 ```python
@@ -524,9 +559,9 @@ avoids the parent-construct hazard entirely.
 ### Redis memory sizing for deduplication and replay metadata
 
 When deduplication is enabled, each distinct dedup key creates one Redis string
-for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The default
-dedup key is a SHA-256 hash of the canonical message payload, so distinct
-payloads are distinct keys. Size Redis for:
+for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The dedup
+key is whatever your `get_deduplication_key` callable returns, so choose a
+short, stable logical ID and size Redis for:
 
 ```text
 peak_unique_publish_rate_per_second
@@ -588,17 +623,46 @@ events_total = Counter(
     "redis-message-queue lifecycle events",
     ["queue", "operation", "outcome", "exception_type"],
 )
+SPAN_SINK_TRUSTED = False
 
 def observe(event: QueueEvent) -> None:
     events_total.labels(
         event.queue, event.operation, event.outcome, event.exception_type or ""
     ).inc()
-    if event.error is not None:
+    if event.error is not None and SPAN_SINK_TRUSTED:
         trace.get_current_span().record_exception(event.error)
 
 queue = RedisMessageQueue("jobs", client=client, on_event=observe)
 ```
 
+#### ⚠ Secrets in `event.error`
+
+`event.error` is the actual exception object — it retains the exception
+message, `__cause__` chain, and traceback. These can contain sensitive content:
+Redis credentials in connection-error messages, message payloads in handler
+exceptions, environment values in stack-frame locals.
+
+When exporting to telemetry sinks (OpenTelemetry, Sentry, Datadog), prefer the
+redaction-friendly `event.exception_type` for metrics and labels. Use
+`event.error` for full structured error data ONLY if your sink is
+trust-equivalent to your application logs and is access-controlled.
+
+Recommended pattern:
+
+```python
+def on_event(event: QueueEvent) -> None:
+    # Metric labels — always safe (just the exception class name)
+    metric_counter.labels(
+        operation=event.operation,
+        outcome=event.outcome,
+        exception_type=event.exception_type or "none",
+    ).inc()
+
+    # Full exception — only if your span sink is trusted
+    if event.error is not None and SPAN_SINK_TRUSTED:
+        span.record_exception(event.error)
+```
+
 #### Event dispatch context
 
 Callbacks fire inline:
@@ -702,12 +766,87 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 
 ## Upgrading
 
+### v7 to v8 migration
+
+v8.0.0 removes implicit dedup key generation. Deduplication is opt-in and
+`deduplication=True` now requires `get_deduplication_key`. If you were relying
+on v7's automatic content hash, provide the equivalent callable explicitly.
+
+Before:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+)
+```
+
+After, prefer a stable logical ID:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: msg["order_id"],
+)
+```
+
+To preserve v7 content-hash behavior mechanically:
+
+```python
+import hashlib
+import json
+
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: hashlib.sha256(
+        json.dumps(msg, sort_keys=True).encode()
+    ).hexdigest(),
+)
+```
+
+If you do not need publish-side deduplication, omit `deduplication` or set
+`deduplication=False`.
+
 ### v6 to v7 migration
 
-v7.0.0 changes explicit drain shutdown semantics. After `queue.drain()` /
-`queue.close()` (sync) or `await queue.drain()` / `await queue.aclose()`
-(async), the same queue instance rejects `publish()` with
-`QueueDrainedError("queue is drained")`.
+v7.0.0 has four breaking changes to check during upgrade.
+
+**AC-02: queue event operation/outcome types are `StrEnum` members.**
+Runtime string comparisons keep working because `StrEnum` subclasses `str`,
+but type-strict code should replace old `Literal[...]` annotations and raw
+string constants with enum members.
+
+Before:
+
+```python
+from typing import Literal
+
+QueueOperation = Literal["publish", "claim", "ack"]
+
+def record(operation: QueueOperation) -> None:
+    if operation == "publish":
+        print("published")
+```
+
+After:
+
+```python
+from redis_message_queue import EventOperation
+
+def record(operation: EventOperation) -> None:
+    if operation is EventOperation.PUBLISH:
+        print("published")
+```
+
+**AC-03: drained queue instances refuse new publishes.** After
+`queue.drain()` / `queue.close()` (sync) or `await queue.drain()` /
+`await queue.aclose()` (async), the same queue instance rejects `publish()`
+with `QueueDrainedError("queue is drained")`.
 
 This state is queue-local and process-local; it is not stored in Redis. If a
 producer must continue publishing after a worker has drained, use a separate
@@ -715,6 +854,42 @@ producer must continue publishing after a worker has drained, use a separate
 shutdown, catch `QueueDrainedError` only at boundaries where late publishes are
 expected and safe to drop or reschedule.
 
+```python
+from redis_message_queue import QueueDrainedError
+
+try:
+    queue.publish("late shutdown audit event")
+except QueueDrainedError:
+    # The queue instance already began draining; drop or reschedule elsewhere.
+    pass
+```
+
+**AC-09: unsafe `drop_oldest` combinations now fail at construction.** These
+configurations raise `ConfigurationError` before the queue or gateway is
+created:
+
+- `pending_overload_policy="drop_oldest"` with `max_pending_length=None`:
+  `drop_oldest requires max_pending_length to be set. Use a positive
+  max_pending_length to define what can be dropped, or use
+  pending_overload_policy='raise' or 'block' for unbounded queues.`
+- `pending_overload_policy="drop_oldest"` with deduplication enabled or
+  `get_deduplication_key` configured:
+  `'pending_overload_policy=drop_oldest' cannot be used with deduplication
+  because dropped messages leave their deduplication keys in Redis, causing
+  future publishes of the same payload to be silently suppressed. Use 'raise'
+  or 'block' for deduplicated queues, or disable deduplication if 'drop_oldest'
+  is required.`
+- `pending_overload_policy="drop_oldest"` with `max_delivery_count` set:
+  `drop_oldest is incompatible with max_delivery_count (set
+  max_delivery_count=None or pick another policy to avoid silent loss of
+  pending DLQ candidates). Use pending_overload_policy='raise' or 'block' when
+  dead-letter handling is required.`
+
+**AC-16: redis-py is capped below 8.0.0.** The package dependency is
+`redis>=5.0.0,<8.0.0` until redis-py 8 RESP3-default behavior is verified.
+Users on redis-py 7.x and earlier are unaffected. If you installed a redis-py
+8.0.0 beta explicitly, downgrade with `pip install "redis<8.0.0"`.
+
 ### Configuration changes on live queues
 
 > **Warning:** These changes are destructive on live queues. Drain the queue completely before applying them.
@@ -724,7 +899,7 @@ expected and safe to drop or reschedule.
 - **Do not toggle visibility timeout in either direction with messages in processing.** Messages claimed by non-VT consumers have no lease metadata, so VT-enabled consumers cannot reclaim them. Disabling VT later orphans existing lease deadline, lease token, and delivery count metadata and removes crash recovery for those in-flight messages. Drain the processing queue first.
 - **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
 - **Changing `max_delivery_count` from a number to `None` leaves delivery metadata behind.** The delivery count hash continues to exist but is no longer consulted. Use this only after draining or after planning manual cleanup of the delivery-count hash.
-- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching between the default hash, explicit `None`, or a custom key function.
+- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching key functions.
 - **Disabling `deduplication` has a retention-window overlap.** Existing dedup records remain in Redis until their TTL expires, but new publishes bypass them. Republishes that would have been suppressed under the old setting can enqueue during that window.
 - **Disabling `enable_failed_queue` stops recording handler failures.** Existing failed entries remain in Redis, but new failures are removed from `processing` without being appended to the failed queue. If `max_delivery_count=None` is also set, repeated handler failures can be dropped with no DLQ or failed-queue record; see [Dead-letter queue](#dead-letter-queue).
 - **Lowering `max_completed_length` or `max_failed_length` trims existing history.** The next completed or failed move calls `LTRIM`, so changing `None` to `N` or lowering `N` can immediately reduce historical entries to the new cap.