redis-message-queue 7.0.1__tar.gz → 8.0.0__tar.gz

{redis_message_queue-7.0.1 → redis_message_queue-8.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: redis-message-queue
-Version: 7.0.1
+Version: 8.0.0
 Summary: Python message queuing with Redis and message deduplication
 License: MIT
 License-File: LICENSE
@@ -26,7 +26,7 @@ Description-Content-Type: text/markdown
 
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v7.0.1-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v8.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -37,7 +37,7 @@ Description-Content-Type: text/markdown
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=7.0.0,<8.0.0"
+pip install "redis-message-queue>=8.0.0,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
@@ -52,11 +52,16 @@ from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
 client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
-queue = RedisMessageQueue("quickstart", client=client, deduplication=True)
-queue.publish("hello")
+queue = RedisMessageQueue(
+    "quickstart",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: msg["id"],
+)
+queue.publish({"id": "msg-1", "text": "hello"})
 with queue.process_message() as message:
     if message is not None:
-        print(f"got {message}")
+        print(f"got {message['text']}")
 # Expected output: got hello
 ```
 
@@ -72,10 +77,15 @@ from redis_message_queue.asyncio import RedisMessageQueue
 
 async def main():
     client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
-    queue = RedisMessageQueue("quickstart", client=client, deduplication=True)
-    await queue.publish("hello")
+    queue = RedisMessageQueue(
+        "quickstart",
+        client=client,
+        deduplication=True,
+        get_deduplication_key=lambda msg: msg["id"],
+    )
+    await queue.publish({"id": "msg-1", "text": "hello"})
     async with queue.process_message() as message:
-        print(f"got {message}")
+        print(f"got {message['text']}")
     await client.aclose()
 
 asyncio.run(main())  # Expected output: got hello
@@ -89,7 +99,7 @@ asyncio.run(main())  # Expected output: got hello
 
 | Feature | Details |
 |---------|---------|
-| **Deduplicated publish** | Lua-scripted atomic SET NX + LPUSH prevents duplicate enqueues within a configurable TTL window (default: 1 hour), even with producer retries. Supports custom key functions for content-based deduplication. Note: deduplication is publish-side only and does not prevent duplicate *delivery* under at-least-once visibility-timeout reclaim |
+| **Deduplicated publish** | Lua-scripted atomic SET NX + LPUSH prevents duplicate enqueues within a configurable TTL window (default: 1 hour), even with producer retries. Requires an explicit `get_deduplication_key` callable so your application defines what counts as a duplicate. Note: deduplication is publish-side only and does not prevent duplicate *delivery* under at-least-once visibility-timeout reclaim |
 | **Visibility-timeout redelivery** | Crashed or stalled consumers' messages are reclaimed and redelivered when a visibility timeout is configured |
 | **Success & failure logs** | Optional completed/failed queues for auditing and reprocessing, with configurable max length to prevent unbounded growth |
 | **Dead-letter queue** | Poison messages that exceed a configurable delivery count are automatically routed to a dead-letter queue instead of being redelivered indefinitely |
@@ -114,10 +124,7 @@ See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-tim
 ### Deduplication
 
 ```python
-# Default: deduplicate by SHA-256 hash of canonical message content (1-hour TTL)
-queue = RedisMessageQueue("q", client=client, deduplication=True)
-
-# Custom dedup key (e.g., deduplicate by order ID only)
+# Deduplicate by order ID for a 1-hour TTL
 queue = RedisMessageQueue(
     "q", client=client,
     deduplication=True,
@@ -128,12 +135,13 @@ queue = RedisMessageQueue(
 queue = RedisMessageQueue("q", client=client, deduplication=False)
 ```
 
-#### Custom dedup key callable must return a non-empty, high-cardinality, tenant-scoped string
+#### Dedup key callable must return a non-empty, high-cardinality, tenant-scoped string
 
-When `get_deduplication_key` is a callable, it is called once per publish and
-must return a `str` that uniquely represents the deduplication scope for that
-message. Returning `None` or `""` raises `ConfigurationError` at publish time;
-returning a non-`str` value raises `TypeError`.
+When `deduplication=True`, `get_deduplication_key` is required. The callable is
+called once per publish and must return a `str` that uniquely represents the
+deduplication scope for that message. Returning `None` or `""` raises
+`ConfigurationError` at publish time; returning a non-`str` value raises
+`TypeError`.
 
 Use stable, high-cardinality keys that include any tenant or account boundary
 needed by your system:
@@ -551,9 +559,9 @@ avoids the parent-construct hazard entirely.
 ### Redis memory sizing for deduplication and replay metadata
 
 When deduplication is enabled, each distinct dedup key creates one Redis string
-for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The default
-dedup key is a SHA-256 hash of the canonical message payload, so distinct
-payloads are distinct keys. Size Redis for:
+for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The dedup
+key is whatever your `get_deduplication_key` callable returns, so choose a
+short, stable logical ID and size Redis for:
 
 ```text
 peak_unique_publish_rate_per_second
@@ -758,6 +766,52 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 
 ## Upgrading
 
+### v7 to v8 migration
+
+v8.0.0 removes implicit dedup key generation. Deduplication is opt-in and
+`deduplication=True` now requires `get_deduplication_key`. If you were relying
+on v7's automatic content hash, provide the equivalent callable explicitly.
+
+Before:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+)
+```
+
+After, prefer a stable logical ID:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: msg["order_id"],
+)
+```
+
+To preserve v7 content-hash behavior mechanically:
+
+```python
+import hashlib
+import json
+
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: hashlib.sha256(
+        json.dumps(msg, sort_keys=True).encode()
+    ).hexdigest(),
+)
+```
+
+If you do not need publish-side deduplication, omit `deduplication` or set
+`deduplication=False`.
+
 ### v6 to v7 migration
 
 v7.0.0 has four breaking changes to check during upgrade.
@@ -845,7 +899,7 @@ Users on redis-py 7.x and earlier are unaffected. If you installed a redis-py
 - **Do not toggle visibility timeout in either direction with messages in processing.** Messages claimed by non-VT consumers have no lease metadata, so VT-enabled consumers cannot reclaim them. Disabling VT later orphans existing lease deadline, lease token, and delivery count metadata and removes crash recovery for those in-flight messages. Drain the processing queue first.
 - **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
 - **Changing `max_delivery_count` from a number to `None` leaves delivery metadata behind.** The delivery count hash continues to exist but is no longer consulted. Use this only after draining or after planning manual cleanup of the delivery-count hash.
-- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching between the default hash, explicit `None`, or a custom key function.
+- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching key functions.
 - **Disabling `deduplication` has a retention-window overlap.** Existing dedup records remain in Redis until their TTL expires, but new publishes bypass them. Republishes that would have been suppressed under the old setting can enqueue during that window.
 - **Disabling `enable_failed_queue` stops recording handler failures.** Existing failed entries remain in Redis, but new failures are removed from `processing` without being appended to the failed queue. If `max_delivery_count=None` is also set, repeated handler failures can be dropped with no DLQ or failed-queue record; see [Dead-letter queue](#dead-letter-queue).
 - **Lowering `max_completed_length` or `max_failed_length` trims existing history.** The next completed or failed move calls `LTRIM`, so changing `None` to `N` or lowering `N` can immediately reduce historical entries to the new cap.

{redis_message_queue-7.0.1 → redis_message_queue-8.0.0}/README.md

@@ -1,6 +1,6 @@
 # redis-message-queue
 
-[![PyPI Version](https://img.shields.io/badge/v7.0.1-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
+[![PyPI Version](https://img.shields.io/badge/v8.0.0-version?color=43cd0f&style=flat&label=pypi)](https://pypi.org/project/redis-message-queue)
 [![PyPI Downloads](https://img.shields.io/pypi/dm/redis-message-queue?color=43cd0f&style=flat&label=downloads)](https://pypistats.org/packages/redis-message-queue)
 [![License: MIT](https://img.shields.io/badge/License-MIT-43cd0f.svg?style=flat&label=license)](LICENSE)
 [![Maintained: yes](https://img.shields.io/badge/yes-43cd0f.svg?style=flat&label=maintained)](https://github.com/Elijas/redis-message-queue/issues)
@@ -11,7 +11,7 @@
 **Lightweight Python message queuing with Redis and built-in publish-side deduplication.** Deduplicate publishes within a TTL window, with optional crash recovery — across any number of producers and consumers.
 
 ```bash
-pip install "redis-message-queue>=7.0.0,<8.0.0"
+pip install "redis-message-queue>=8.0.0,<9.0.0"
 ```
 
 Requires Redis server >= 6.2.
@@ -26,11 +26,16 @@ from redis import Redis
 from redis_message_queue import RedisMessageQueue
 
 client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
-queue = RedisMessageQueue("quickstart", client=client, deduplication=True)
-queue.publish("hello")
+queue = RedisMessageQueue(
+    "quickstart",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: msg["id"],
+)
+queue.publish({"id": "msg-1", "text": "hello"})
 with queue.process_message() as message:
     if message is not None:
-        print(f"got {message}")
+        print(f"got {message['text']}")
 # Expected output: got hello
 ```
 
@@ -46,10 +51,15 @@ from redis_message_queue.asyncio import RedisMessageQueue
 
 async def main():
     client = Redis.from_url("redis://localhost:6379/0", decode_responses=True)
-    queue = RedisMessageQueue("quickstart", client=client, deduplication=True)
-    await queue.publish("hello")
+    queue = RedisMessageQueue(
+        "quickstart",
+        client=client,
+        deduplication=True,
+        get_deduplication_key=lambda msg: msg["id"],
+    )
+    await queue.publish({"id": "msg-1", "text": "hello"})
     async with queue.process_message() as message:
-        print(f"got {message}")
+        print(f"got {message['text']}")
     await client.aclose()
 
 asyncio.run(main())  # Expected output: got hello
@@ -63,7 +73,7 @@ asyncio.run(main())  # Expected output: got hello
 
 | Feature | Details |
 |---------|---------|
-| **Deduplicated publish** | Lua-scripted atomic SET NX + LPUSH prevents duplicate enqueues within a configurable TTL window (default: 1 hour), even with producer retries. Supports custom key functions for content-based deduplication. Note: deduplication is publish-side only and does not prevent duplicate *delivery* under at-least-once visibility-timeout reclaim |
+| **Deduplicated publish** | Lua-scripted atomic SET NX + LPUSH prevents duplicate enqueues within a configurable TTL window (default: 1 hour), even with producer retries. Requires an explicit `get_deduplication_key` callable so your application defines what counts as a duplicate. Note: deduplication is publish-side only and does not prevent duplicate *delivery* under at-least-once visibility-timeout reclaim |
 | **Visibility-timeout redelivery** | Crashed or stalled consumers' messages are reclaimed and redelivered when a visibility timeout is configured |
 | **Success & failure logs** | Optional completed/failed queues for auditing and reprocessing, with configurable max length to prevent unbounded growth |
 | **Dead-letter queue** | Poison messages that exceed a configurable delivery count are automatically routed to a dead-letter queue instead of being redelivered indefinitely |
@@ -88,10 +98,7 @@ See [Crash recovery with visibility timeout](#crash-recovery-with-visibility-tim
 ### Deduplication
 
 ```python
-# Default: deduplicate by SHA-256 hash of canonical message content (1-hour TTL)
-queue = RedisMessageQueue("q", client=client, deduplication=True)
-
-# Custom dedup key (e.g., deduplicate by order ID only)
+# Deduplicate by order ID for a 1-hour TTL
 queue = RedisMessageQueue(
     "q", client=client,
     deduplication=True,
@@ -102,12 +109,13 @@ queue = RedisMessageQueue(
 queue = RedisMessageQueue("q", client=client, deduplication=False)
 ```
 
-#### Custom dedup key callable must return a non-empty, high-cardinality, tenant-scoped string
+#### Dedup key callable must return a non-empty, high-cardinality, tenant-scoped string
 
-When `get_deduplication_key` is a callable, it is called once per publish and
-must return a `str` that uniquely represents the deduplication scope for that
-message. Returning `None` or `""` raises `ConfigurationError` at publish time;
-returning a non-`str` value raises `TypeError`.
+When `deduplication=True`, `get_deduplication_key` is required. The callable is
+called once per publish and must return a `str` that uniquely represents the
+deduplication scope for that message. Returning `None` or `""` raises
+`ConfigurationError` at publish time; returning a non-`str` value raises
+`TypeError`.
 
 Use stable, high-cardinality keys that include any tenant or account boundary
 needed by your system:
@@ -525,9 +533,9 @@ avoids the parent-construct hazard entirely.
 ### Redis memory sizing for deduplication and replay metadata
 
 When deduplication is enabled, each distinct dedup key creates one Redis string
-for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The default
-dedup key is a SHA-256 hash of the canonical message payload, so distinct
-payloads are distinct keys. Size Redis for:
+for `message_deduplication_log_ttl_seconds` (default: 3600 seconds). The dedup
+key is whatever your `get_deduplication_key` callable returns, so choose a
+short, stable logical ID and size Redis for:
 
 ```text
 peak_unique_publish_rate_per_second
@@ -732,6 +740,52 @@ For a full analysis, see [docs/production-readiness.md](docs/production-readines
 
 ## Upgrading
 
+### v7 to v8 migration
+
+v8.0.0 removes implicit dedup key generation. Deduplication is opt-in and
+`deduplication=True` now requires `get_deduplication_key`. If you were relying
+on v7's automatic content hash, provide the equivalent callable explicitly.
+
+Before:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+)
+```
+
+After, prefer a stable logical ID:
+
+```python
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: msg["order_id"],
+)
+```
+
+To preserve v7 content-hash behavior mechanically:
+
+```python
+import hashlib
+import json
+
+queue = RedisMessageQueue(
+    "orders",
+    client=client,
+    deduplication=True,
+    get_deduplication_key=lambda msg: hashlib.sha256(
+        json.dumps(msg, sort_keys=True).encode()
+    ).hexdigest(),
+)
+```
+
+If you do not need publish-side deduplication, omit `deduplication` or set
+`deduplication=False`.
+
 ### v6 to v7 migration
 
 v7.0.0 has four breaking changes to check during upgrade.
@@ -819,7 +873,7 @@ Users on redis-py 7.x and earlier are unaffected. If you installed a redis-py
 - **Do not toggle visibility timeout in either direction with messages in processing.** Messages claimed by non-VT consumers have no lease metadata, so VT-enabled consumers cannot reclaim them. Disabling VT later orphans existing lease deadline, lease token, and delivery count metadata and removes crash recovery for those in-flight messages. Drain the processing queue first.
 - **Reducing `max_delivery_count` retroactively DLQs messages.** The delivery count hash persists across restarts. Messages whose accumulated count exceeds the new limit are immediately dead-lettered on next claim.
 - **Changing `max_delivery_count` from a number to `None` leaves delivery metadata behind.** The delivery count hash continues to exist but is no longer consulted. Use this only after draining or after planning manual cleanup of the delivery-count hash.
-- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching between the default hash, explicit `None`, or a custom key function.
+- **Changing `get_deduplication_key` changes the dedup keyspace.** Existing dedup records become inert for the duration of their TTL. Drain the queue or clear the old deduplication keys before switching key functions.
 - **Disabling `deduplication` has a retention-window overlap.** Existing dedup records remain in Redis until their TTL expires, but new publishes bypass them. Republishes that would have been suppressed under the old setting can enqueue during that window.
 - **Disabling `enable_failed_queue` stops recording handler failures.** Existing failed entries remain in Redis, but new failures are removed from `processing` without being appended to the failed queue. If `max_delivery_count=None` is also set, repeated handler failures can be dropped with no DLQ or failed-queue record; see [Dead-letter queue](#dead-letter-queue).
 - **Lowering `max_completed_length` or `max_failed_length` trims existing history.** The next completed or failed move calls `LTRIM`, so changing `None` to `N` or lowering `N` can immediately reduce historical entries to the new cap.

{redis_message_queue-7.0.1 → redis_message_queue-8.0.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "redis-message-queue"
-version = "7.0.1"
+version = "8.0.0"
 description = "Python message queuing with Redis and message deduplication"
 authors = ["Elijas <4084885+Elijas@users.noreply.github.com>"]
 readme = "README.md"

{redis_message_queue-7.0.1 → redis_message_queue-8.0.0}/redis_message_queue/_config.py

@@ -32,6 +32,11 @@ DEFAULT_PENDING_OVERLOAD_BLOCK_TIMEOUT_SECONDS = 1.0
 INTERRUPTIBLE_RETRY_SLEEP_POLL_SECONDS = 0.05
 PENDING_OVERLOAD_LUA_SENTINEL = -1
 PENDING_OVERLOAD_POLICIES = ("raise", "drop_oldest", "block")
+DEDUPLICATION_REQUIRES_KEY_MESSAGE = (
+    "deduplication=True requires get_deduplication_key (callable returning a non-empty str). "
+    "Pass a callable like `lambda msg: msg['id']` (recommended: a stable logical ID), "
+    "or set deduplication=False."
+)
 
 
 def is_redis_retryable_exception(exception):
@@ -301,6 +306,15 @@ def validate_gateway_parameters(
         )
 
 
+def validate_dedup_configuration(
+    *,
+    deduplication: bool,
+    get_deduplication_key: object,
+) -> None:
+    if deduplication and get_deduplication_key is None:
+        raise ConfigurationError(DEDUPLICATION_REQUIRES_KEY_MESSAGE)
+
+
 def validate_pending_backpressure_parameters(
     max_pending_length: int | None,
     pending_overload_policy: str,

{redis_message_queue-7.0.1 → redis_message_queue-8.0.0}/redis_message_queue/asyncio/redis_message_queue.py

@@ -15,6 +15,7 @@ import redis.exceptions
 from redis_message_queue._callable_utils import is_async_callable
 from redis_message_queue._config import (
     DEFAULT_PENDING_OVERLOAD_BLOCK_TIMEOUT_SECONDS,
+    validate_dedup_configuration,
     validate_pending_backpressure_parameters,
 )
 from redis_message_queue._event import EventOperation, EventOutcome, QueueEvent
@@ -232,16 +233,6 @@ def _derive_dead_letter_queue(name: str, key_separator: str) -> str:
     return f"{name}{key_separator}{_AUTO_DEAD_LETTER_QUEUE_SUFFIX}"
 
 
-def _canonical_bytes(message: str | dict) -> bytes:
-    if isinstance(message, dict):
-        return json.dumps(message, sort_keys=True, allow_nan=False).encode("utf-8")
-    return message.encode("utf-8")
-
-
-def _default_get_deduplication_key(message: str | dict) -> str:
-    return hashlib.sha256(_canonical_bytes(message)).hexdigest()
-
-
 def _bind_dead_letter_gateway_to_queue(gateway: AbstractRedisGateway, queue_pending_key: str) -> None:
     """Reject reuse of a dead-letter-enabled gateway across different queues.
 
@@ -461,7 +452,7 @@ class RedisMessageQueue:
         *,
         gateway: Optional[AbstractRedisGateway] = None,
         client: Optional[redis.asyncio.Redis] = None,
-        deduplication: bool = True,
+        deduplication: bool = False,
         enable_completed_queue: bool = False,
         enable_failed_queue: bool = False,
         visibility_timeout_seconds: int | None = _DEFAULT_VISIBILITY_TIMEOUT_SECONDS,
@@ -473,7 +464,7 @@ class RedisMessageQueue:
         pending_overload_policy: Literal["raise", "drop_oldest", "block"] = "raise",
         pending_overload_block_timeout_seconds: float = DEFAULT_PENDING_OVERLOAD_BLOCK_TIMEOUT_SECONDS,
         key_separator: str = "::",
-        get_deduplication_key: Optional[Callable[[str | dict], str]] = _default_get_deduplication_key,
+        get_deduplication_key: Optional[Callable[[str | dict], str]] = None,
         interrupt: BaseGracefulInterruptHandler | None = None,
         on_heartbeat_failure: Callable[[], Awaitable[None] | None] | None = None,
         on_event: Callable[[QueueEvent], Awaitable[None]] | None = None,
@@ -489,10 +480,9 @@ class RedisMessageQueue:
         auto-derived dead-letter queue. Set it to ``None`` for unlimited
         redelivery.
 
-        ``get_deduplication_key`` defaults to a SHA-256 hash of the canonical
-        message string. Passing ``None`` uses the literal serialized message as
-        the deduplication key; passing a callable or coroutine callable lets you
-        define a custom keyspace.
+        ``deduplication=True`` requires ``get_deduplication_key`` to be a
+        callable that returns a non-empty string. Use a stable logical ID for
+        the deduplication keyspace.
 
         ``max_pending_length`` defaults to ``None`` (unbounded). Set it to a
         positive integer to cap pending-list depth during publish.
@@ -575,15 +565,17 @@ class RedisMessageQueue:
                 raise ConfigurationError(
                     f"'visibility_timeout_seconds' must be positive when provided, got {visibility_timeout_seconds}"
                 )
-        get_deduplication_key_was_configured = (
-            get_deduplication_key is not None and get_deduplication_key is not _default_get_deduplication_key
-        )
+        get_deduplication_key_was_configured = get_deduplication_key is not None
         if get_deduplication_key is not None and not callable(get_deduplication_key):
             raise TypeError(
                 f"'get_deduplication_key' must be callable, got {type(get_deduplication_key).__name__}."
                 " Expected a function that takes the message (str | dict) and returns a str (or an awaitable thereof)."
                 " Example: get_deduplication_key=lambda msg: msg['user_id']"
             )
+        validate_dedup_configuration(
+            deduplication=deduplication,
+            get_deduplication_key=get_deduplication_key,
+        )
         validate_pending_backpressure_parameters(
             max_pending_length,
             pending_overload_policy,
@@ -812,13 +804,10 @@ class RedisMessageQueue:
             await self._emit_event("publish", "success", duration_ms=_duration_ms(started_at))
             return True
 
-        if self._get_deduplication_key is not None:
-            dedup_key = self._get_deduplication_key(message)
-            if inspect.isawaitable(dedup_key):
-                dedup_key = await dedup_key
-            dedup_key = validate_callable_deduplication_key(dedup_key, message)
-        else:
-            dedup_key = message_str
+        dedup_key = self._get_deduplication_key(message)
+        if inspect.isawaitable(dedup_key):
+            dedup_key = await dedup_key
+        dedup_key = validate_callable_deduplication_key(dedup_key, message)
         dedup_key = self.key.deduplication(dedup_key)
 
         try:

{redis_message_queue-7.0.1 → redis_message_queue-8.0.0}/redis_message_queue/redis_message_queue.py

@@ -16,6 +16,7 @@ from redis_message_queue._abstract_redis_gateway import AbstractRedisGateway
 from redis_message_queue._callable_utils import is_async_callable
 from redis_message_queue._config import (
     DEFAULT_PENDING_OVERLOAD_BLOCK_TIMEOUT_SECONDS,
+    validate_dedup_configuration,
     validate_pending_backpressure_parameters,
 )
 from redis_message_queue._event import EventOperation, EventOutcome, QueueEvent
@@ -166,16 +167,6 @@ def _derive_dead_letter_queue(name: str, key_separator: str) -> str:
     return f"{name}{key_separator}{_AUTO_DEAD_LETTER_QUEUE_SUFFIX}"
 
 
-def _canonical_bytes(message: str | dict) -> bytes:
-    if isinstance(message, dict):
-        return json.dumps(message, sort_keys=True, allow_nan=False).encode("utf-8")
-    return message.encode("utf-8")
-
-
-def _default_get_deduplication_key(message: str | dict) -> str:
-    return hashlib.sha256(_canonical_bytes(message)).hexdigest()
-
-
 def _bind_dead_letter_gateway_to_queue(gateway: AbstractRedisGateway, queue_pending_key: str) -> None:
     """Reject reuse of a dead-letter-enabled gateway across different queues.
 
@@ -402,7 +393,7 @@ class RedisMessageQueue:
         *,
         gateway: Optional[AbstractRedisGateway] = None,
         client: Optional[redis.Redis] = None,
-        deduplication: bool = True,
+        deduplication: bool = False,
         enable_completed_queue: bool = False,
         enable_failed_queue: bool = False,
         visibility_timeout_seconds: int | None = _DEFAULT_VISIBILITY_TIMEOUT_SECONDS,
@@ -414,7 +405,7 @@ class RedisMessageQueue:
         pending_overload_policy: Literal["raise", "drop_oldest", "block"] = "raise",
         pending_overload_block_timeout_seconds: float = DEFAULT_PENDING_OVERLOAD_BLOCK_TIMEOUT_SECONDS,
         key_separator: str = "::",
-        get_deduplication_key: Optional[Callable[[str | dict], str]] = _default_get_deduplication_key,
+        get_deduplication_key: Optional[Callable[[str | dict], str]] = None,
         interrupt: BaseGracefulInterruptHandler | None = None,
         on_heartbeat_failure: Callable[[], None] | None = None,
         on_event: Callable[[QueueEvent], None] | None = None,
@@ -430,10 +421,9 @@ class RedisMessageQueue:
         auto-derived dead-letter queue. Set it to ``None`` for unlimited
         redelivery.
 
-        ``get_deduplication_key`` defaults to a SHA-256 hash of the canonical
-        message string. Passing ``None`` uses the literal serialized message as
-        the deduplication key; passing a callable lets you define a custom
-        keyspace.
+        ``deduplication=True`` requires ``get_deduplication_key`` to be a
+        callable that returns a non-empty string. Use a stable logical ID for
+        the deduplication keyspace.
 
         ``max_pending_length`` defaults to ``None`` (unbounded). Set it to a
         positive integer to cap pending-list depth during publish.
@@ -515,9 +505,7 @@ class RedisMessageQueue:
                 raise ConfigurationError(
                     f"'visibility_timeout_seconds' must be positive when provided, got {visibility_timeout_seconds}"
                 )
-        get_deduplication_key_was_configured = (
-            get_deduplication_key is not None and get_deduplication_key is not _default_get_deduplication_key
-        )
+        get_deduplication_key_was_configured = get_deduplication_key is not None
         if get_deduplication_key is not None and not callable(get_deduplication_key):
             raise TypeError(
                 f"'get_deduplication_key' must be callable, got {type(get_deduplication_key).__name__}."
@@ -529,6 +517,10 @@ class RedisMessageQueue:
                 "'get_deduplication_key' is an async callable; "
                 "use the async RedisMessageQueue from redis_message_queue.asyncio instead"
             )
+        validate_dedup_configuration(
+            deduplication=deduplication,
+            get_deduplication_key=get_deduplication_key,
+        )
         validate_pending_backpressure_parameters(
             max_pending_length,
             pending_overload_policy,
@@ -766,22 +758,18 @@ class RedisMessageQueue:
             self._emit_event("publish", "success", duration_ms=_duration_ms(started_at))
             return True
 
-        if self._get_deduplication_key is not None:
-            dedup_key = self._get_deduplication_key(message)
-            if inspect.isawaitable(dedup_key):
-                is_coroutine = inspect.iscoroutine(dedup_key)
-                _close_or_cancel_awaitable(dedup_key)
-                if is_coroutine:
-                    raise TypeError(
-                        "'get_deduplication_key' returned a coroutine; "
-                        "use the async RedisMessageQueue for async callables"
-                    )
+        dedup_key = self._get_deduplication_key(message)
+        if inspect.isawaitable(dedup_key):
+            is_coroutine = inspect.iscoroutine(dedup_key)
+            _close_or_cancel_awaitable(dedup_key)
+            if is_coroutine:
                 raise TypeError(
-                    "'get_deduplication_key' returned an awaitable; use the async RedisMessageQueue for async callables"
+                    "'get_deduplication_key' returned a coroutine; use the async RedisMessageQueue for async callables"
                 )
-            dedup_key = validate_callable_deduplication_key(dedup_key, message)
-        else:
-            dedup_key = message_str
+            raise TypeError(
+                "'get_deduplication_key' returned an awaitable; use the async RedisMessageQueue for async callables"
+            )
+        dedup_key = validate_callable_deduplication_key(dedup_key, message)
         dedup_key = self.key.deduplication(dedup_key)
 
         try: