babelqueue 1.18.0__tar.gz → 1.19.0__tar.gz

{babelqueue-1.18.0 → babelqueue-1.19.0}/CHANGELOG.md

@@ -9,6 +9,52 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.19.0] - 2026-10-12
+
+### Changed
+- Tests only: the vendored conformance suite is now 1.3.0 (exact-case envelope keys, last-wins duplicates: new `invalid-wrong-case-job`, `wrong-case-keys-roundtrip` and `duplicate-keys-last-wins` fixtures). No production change; the Python core already behaved this way.
+- **Kafka: a poison record is dead-lettered wrapped (`dead_letter.reason: "poison"`), like the Go
+  reference (§6.5, ADR-0009).** A value that is not valid UTF-8, not valid JSON, not a JSON object, or has
+  a member of the wrong type (the exact set is listed in the next entry) used to reach `<topic>.dlq` as its raw
+  bytes. It is now written as a body that keeps its keys when it is exactly one JSON object, otherwise
+  `{"raw": "<body>"}` (bytes that are not UTF-8 are stored with U+FFFD substitutions), plus a
+  `dead_letter` block (`reason: "poison"`, `original_queue`, `attempts` from the `bq-attempts` header, `lang`);
+  the record key is preserved. A poison record is still never retried and is dropped when no DLQ is
+  configured (the `on_unknown_urn` strategy, `DELETE` included, no longer matters: see below). The version-gate rejection (`bq-schema-version`, §6.7) still
+  dead-letters its record raw. The wrap is applied when the runtime writes the DLQ record (not when the
+  record is read), so a poison record that `redrive` restores into the DLQ it popped it from stays byte
+  for byte, whatever that DLQ is named. `original_queue` is the `bq-original-topic` header when present
+  (else the consumed topic), like the Java, .NET and Node consumers; a body whose numbers cannot be
+  written back as valid JSON (e.g. `1e400`) is carried as `{"raw": ...}` instead of producing an invalid DLQ body.
+  A body that decodes but names no URN is not poison (unknown-URN path, as before). Consumers of the DLQ
+  that expected the raw bytes of a malformed record must now read the wrapped body.
+- **Kafka: the poison trigger set now equals what the Go reference's typed `Decode` rejects.** Behaviour
+  change; the old Python-only rules (a lenient `attempts` read, `Accepts`-style conformance checks) are gone.
+  A record is poison when it is not valid UTF-8, not valid JSON (an empty or whitespace-only body, a BOM,
+  trailing data, leading zeros, `NaN` / `Infinity`, single quotes), or not a JSON object (a bare `null` is
+  not poison: it is an empty envelope and takes the unknown-URN path), or when a member has the wrong type:
+  `job`, `trace_id`, `meta.id|queue|lang` and `dead_letter.reason|error|exception|original_queue|lang`
+  must be a string or null; `meta.schema_version`, `meta.created_at`, `attempts` and
+  `dead_letter.failed_at|attempts` must be an int64 integer literal or null (a string such as `"3"` or
+  `"2026"`, a fraction or exponent such as `3.0` / `1e2`, a bool, an array and an object are poison);
+  `data`, `meta` and `dead_letter` must be an object or null; every number inside `data` must be a finite
+  double. Member names are matched exactly as spelled (case-sensitive, as the contract schema and the core decoder read them; a `JOB` or `Meta` member is an unknown member, not a typed one) and every duplicate key is checked. A record Go decodes
+  is no longer poison, even when it is not a conformant envelope: a non-string `urn` alias, an absent
+  `data`, `"attempts": null` (read as 0 and written back as 0) and `meta.schema_version: 2` are routed by URN
+  like any other record (Go's consume path never calls `Accepts`). The Kafka transport no longer
+  delivers records under the shared non-numeric-`attempts` route; they are plain poison now.
+  `redrive` restores a poison-routed record into the DLQ unchanged instead of replaying or re-wrapping it,
+  even when its `job` still reads as a URN. The only Python-specific limit is JSON nested deeper than
+  the interpreter's recursion limit (about 1000 levels, Go allows 10000), which is treated as poison.
+- Known, intentional differences from the Go reference for a poison record: `dead_letter.attempts` is the `bq-attempts` header (Go writes 0) and `original_queue` is the `bq-original-topic` header when present (Go writes the consumed topic); a float literal such as `1.0` / `1e2` is re-formatted when the body is re-encoded. The wire envelope and the `dead_letter` shape are unchanged.
+- **Kafka: any decode failure is poison; the partition is never left to wedge.** Member names are matched case-sensitively, exactly as the core decoder reads them (no `encoding/json`-style folding, no `str.casefold()`); the Go reference's case-insensitive read is a known difference that contradicts the contract schema and is tightened separately. Whatever `EnvelopeCodec.decode` raises after the shape scan (e.g. `RecursionError` at the interpreter's limit) is now routed as poison instead of escaping `pop` (which left the offset uncommitted and the partition redelivered forever); the poison wrapper falls back to `{"raw": ...}` on `RecursionError` or a body that cannot be written as UTF-8. A lone surrogate escape (`"\ud800"`) that survives into a retry/DLQ write is substituted by U+FFFD, as Go does, instead of raising `UnicodeEncodeError`. Invalid UTF-8 remains poison (Go substitutes U+FFFD and accepts it). The wire envelope and the `dead_letter` shape are unchanged.
+- **Kafka: a body the core JSON parser refuses is poison, not an empty envelope.** The core decoder turns a `ValueError` from `json.loads` into an empty envelope; where the shape scan accepts a body that plain `json.loads` rejects (Python 3.11+: an unknown member holding an integer of more than 4300 digits), the record would have reached a handler with its content dropped. It is now dead-lettered as poison. No change to the wire envelope or the `dead_letter` shape.
+- **Kafka: a poison record is dead-lettered whatever `on_unknown_urn` is.** The runtime checked the `DELETE` strategy before it recognised a poison record, so with `on_unknown_urn="delete"` a poison record was acknowledged and silently dropped even with a DLQ configured, where the Go reference writes it to `<topic>.dlq` with `dead_letter.reason: "poison"` and commits. A poison delivery (`PoisonRoute`) is now settled before the strategy is consulted: written wrapped to the DLQ and committed under `fail`, `delete`, `dead_letter` and `release`, and dropped only when no DLQ is configured (unchanged). The version-gate path and every other transport keep their behaviour. No change to the wire envelope or the `dead_letter` shape.
+
+### Fixed
+- Kafka: a wrongly-typed `meta` or `meta.created_at` in a wrapped poison body no longer fails the write
+  that settles it (header projection and record timestamp tolerate it).
+
 ## [1.18.0] - 2026-10-11
 
 ### Added

{babelqueue-1.18.0 → babelqueue-1.19.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: babelqueue
-Version: 1.18.0
+Version: 1.19.0
 Summary: Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers.
 Project-URL: Homepage, https://babelqueue.com
 Project-URL: Source, https://github.com/BabelQueue/babelqueue-python

{babelqueue-1.18.0 → babelqueue-1.19.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.18.0"
+version = "1.19.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"

{babelqueue-1.18.0 → babelqueue-1.19.0}/src/babelqueue/__init__.py

@@ -41,7 +41,7 @@ from .transport import (
     Transport,
 )
 
-__version__ = "1.18.0"
+__version__ = "1.19.0"
 
 __all__ = [
     "BabelQueue",

babelqueue-1.19.0/src/babelqueue/_envelope_shape.py

@@ -0,0 +1,153 @@
+"""The poison trigger set of the Kafka transport (§6.5).
+
+A body is undecodable when a strict, typed decode into ``Envelope`` / ``Meta`` / ``DeadLetter`` (the
+Go reference's ``babelqueue.Decode``, except that member names are case-sensitive here) rejects it:
+
+* the body must be exactly one JSON value (no BOM, trailing tokens, ``NaN`` / ``Infinity``, malformed
+  numbers); a bare ``null`` decodes to an empty envelope, any other non-object root is rejected;
+* ``job``, ``trace_id``, ``meta.id|queue|lang`` and ``dead_letter.reason|error|exception|original_queue|lang``
+  are string-or-null;
+* ``attempts``, ``meta.schema_version|created_at`` and ``dead_letter.failed_at|attempts`` are
+  integer-or-null: an integer literal (no fraction or exponent) that fits ``int64``;
+* ``data``, ``meta`` and ``dead_letter`` are object-or-null, and a number inside ``data`` must be a
+  finite double;
+* member names are matched exactly as spelled (case-sensitive, as the contract schema and the core
+  decoder read them) and every occurrence of a duplicated key is checked; any other member (a
+  differently-cased name or a non-string ``urn`` alias included) is only checked for syntax.
+
+A well-typed but non-conformant envelope is not poison: Go's consume path never calls ``Accepts``.
+"""
+
+import json
+import math
+from typing import Any, Dict, List, Optional, Tuple
+
+_INT64_MIN = -(2**63)
+_INT64_MAX = 2**63 - 1
+#: Beyond this many digits an integer literal cannot fit an int64 or a double; it is kept as a
+#: sentinel so ``json`` never trips the interpreter's integer-string-length limit.
+_MAX_INT_DIGITS = 4000
+
+
+class _Pairs(List[Tuple[str, Any]]):
+    """A decoded JSON object as its ``(key, value)`` list, so duplicate keys survive."""
+
+
+class _HugeInt:
+    """An integer literal too large for any int64 or double."""
+
+
+_HUGE = _HugeInt()
+
+_STRING = "string"
+_INTEGER = "integer"
+_OBJECT = "object"
+_DATA = "data"
+
+_META = {
+    "id": _STRING,
+    "queue": _STRING,
+    "lang": _STRING,
+    "schema_version": _INTEGER,
+    "created_at": _INTEGER,
+}
+_DEAD_LETTER = {
+    "reason": _STRING,
+    "error": _STRING,
+    "exception": _STRING,
+    "original_queue": _STRING,
+    "lang": _STRING,
+    "failed_at": _INTEGER,
+    "attempts": _INTEGER,
+}
+_TOP = {
+    "job": _STRING,
+    "trace_id": _STRING,
+    "attempts": _INTEGER,
+    "data": _DATA,
+    "meta": _OBJECT,
+    "dead_letter": _OBJECT,
+}
+
+
+def _reject_constant(name: str) -> Any:
+    raise ValueError("invalid JSON constant " + name)
+
+
+def _parse_int(literal: str) -> Any:
+    return int(literal) if len(literal) <= _MAX_INT_DIGITS else _HUGE
+
+
+def _is_finite_number(value: Any) -> bool:
+    if value is _HUGE:
+        return False
+    if isinstance(value, float):
+        return math.isfinite(value)
+    if isinstance(value, int) and not isinstance(value, bool):
+        try:
+            return math.isfinite(float(value))
+        except OverflowError:
+            return False
+    return True
+
+
+def _all_numbers_finite(root: Any) -> bool:
+    stack = [root]
+    while stack:
+        value = stack.pop()
+        if isinstance(value, _Pairs):
+            stack.extend(item for _, item in value)
+        elif isinstance(value, list):
+            stack.extend(value)
+        elif not _is_finite_number(value):
+            return False
+    return True
+
+
+def _check_member(kind: str, name: str, value: Any) -> Optional[str]:
+    if value is None:
+        return None
+    if kind == _STRING:
+        return None if isinstance(value, str) else "member %r is not a string" % name
+    if kind == _INTEGER:
+        if type(value) is int and _INT64_MIN <= value <= _INT64_MAX:
+            return None
+        return "member %r is not an int64 integer" % name
+    if not isinstance(value, _Pairs):
+        return "member %r is not an object" % name
+    if kind == _DATA and not _all_numbers_finite(value):
+        return "member %r holds a number that is not a finite double" % name
+    return None
+
+
+def _check_object(pairs: _Pairs, kinds: Dict[str, str], prefix: str) -> Optional[str]:
+    for key, value in pairs:
+        kind = kinds.get(key)
+        if kind is None:
+            continue
+        failure = _check_member(kind, prefix + key, value)
+        if failure is not None:
+            return failure
+        if kind == _OBJECT and value is not None:
+            nested = _check_object(value, _META if key == "meta" else _DEAD_LETTER, key + ".")
+            if nested is not None:
+                return nested
+    return None
+
+
+def envelope_violation(text: str) -> Optional[str]:
+    """Why ``text`` is not a decodable BabelQueue body, or ``None`` when Go's typed decode accepts it."""
+    try:
+        parsed = json.loads(
+            text,
+            object_pairs_hook=_Pairs,
+            parse_int=_parse_int,
+            parse_constant=_reject_constant,
+        )
+    except (ValueError, RecursionError) as exc:
+        return "invalid JSON: " + str(exc)
+    if parsed is None:
+        return None  # a bare ``null`` is an empty envelope, not poison
+    if not isinstance(parsed, _Pairs):
+        return "the JSON root is not an object"
+    return _check_object(parsed, _TOP, "")

{babelqueue-1.18.0 → babelqueue-1.19.0}/src/babelqueue/app.py

@@ -41,6 +41,7 @@ from .replay import HEADER_REPLAY_BYPASS, _replay_scope
 from .routing import UnknownUrnStrategy
 from .transport import (
     HeaderPublisher,
+    PoisonRoute,
     ReceivedMessage,
     Redeliverer,
     Rewinder,
@@ -476,6 +477,14 @@ class BabelQueue:
 
     def _route_unknown(self, urn: str, received: ReceivedMessage, envelope: Mapping[str, Any]) -> None:
         strategy = self.on_unknown_urn
+        # A Kafka poison record (§6.5: not valid UTF-8 / not a decodable body) is not an unknown URN: it
+        # never decoded into an envelope, so ``on_unknown_urn`` has no say. It is checked before the
+        # DELETE strategy and settled like the Go reference does -- dead-lettered (wrapped, reason
+        # ``poison``) and committed, or dropped only when no DLQ is configured. Other transports'
+        # version-gated deliveries keep their existing order below.
+        if isinstance(received.route_urn, PoisonRoute):
+            self._settle_version_gated(received)
+            return
         if strategy == UnknownUrnStrategy.DELETE:
             self._ack_settled(received)
             return

{babelqueue-1.18.0 → babelqueue-1.19.0}/src/babelqueue/kafka_transport.py

@@ -19,10 +19,18 @@ Settlement writes (retry republish, DLQ) go through ``publish_settle`` and wait
 delivery report (``delivery_timeout``); a failure raises ``KafkaDeliveryError`` and the offset is
 not committed. ``publish`` itself stays fire-and-forget. Those writes also carry the consumed
 record's key unchanged (``ReceivedMessage.key``; null stays null, empty stays empty, never
-synthesised). A record value that is not valid UTF-8 is handed to the runtime under a
-reserved route so its original bytes are dead-lettered raw (written back with no projected headers;
-with the DLQ off, or ``UnknownUrnStrategy.DELETE``, it is dropped and committed). The runtime logs
-it as a version-gated message because it reuses that settle path; the route in the log names it.
+synthesised). A poison record (§6.5: a value that is not valid UTF-8, not valid JSON, not a JSON
+object other than ``null``, or a member of the wrong type, member names matched exactly as spelled: what the Go
+reference's typed ``Decode`` rejects, see :mod:`babelqueue._envelope_shape`) is handed to the runtime under a
+reserved route and dead-lettered directly, never retried, *wrapped* like the Go reference: a body that
+is one JSON object keeps its keys, anything else is carried as ``{"raw": "<body>"}``, and a
+``dead_letter`` block with ``reason: "poison"`` is added (the key is preserved; with the DLQ off it is
+dropped and committed; ``on_unknown_urn``, ``DELETE`` included, has no say). The record is handed over raw under a
+``PoisonRoute`` and wrapped by ``publish_settle`` only when it is written to a queue other than the one
+it was read from, so a ``redrive`` restore into the DLQ it popped it from (any DLQ name) is byte for
+byte and never double-wraps. The version gate (§6.7) still
+dead-letters its record raw. The runtime logs a poison record as a version-gated message because it
+reuses that settle path; the route in the log names it.
 
 This implements §6 of the broker-bindings contract. The envelope is unchanged
 (``schema_version`` stays 1); Apache Kafka is purely additive.
@@ -34,15 +42,18 @@ or ``KafkaTransport(producer=..., consumer_factory=...)``.
 
 from __future__ import annotations
 
+import json
 import logging
 import re
 from typing import Any, Callable, Dict, List, Optional, Tuple
 from urllib.parse import quote
 
+from . import dead_letter
 from .codec import SCHEMA_VERSION, EnvelopeCodec, parse_envelope
+from ._envelope_shape import envelope_violation
 from .exceptions import BabelQueueError
 from .transport import (
-    NON_NUMERIC_ATTEMPTS_ROUTE,
+    PoisonRoute,
     ReceivedMessage,
     Transport,
     truncate_declared_version,
@@ -54,6 +65,8 @@ _logger = logging.getLogger("babelqueue.kafka")
 #: same set in every SDK (GR-5). Deliberately not ``str.strip()``, which also strips U+001C-U+001F,
 #: U+0085, U+00A0, U+2028, U+3000 ...
 _BLANK_ASCII_PATTERN = re.compile(r"[ \t\n\x0b\x0c\r]*")
+#: Every surrogate code point except U+DC80..U+DCFF (those are ``surrogateescape``d non-UTF-8 bytes).
+_LONE_SURROGATE = re.compile("[\ud800-\udbff\udc00-\udc7f\udd00-\udfff]")
 #: Prefix of the reserved ``route_urn`` a version-gated message is delivered under. It is not a
 #: ``urn:babel:`` URN, so no handler can legitimately own it; the runtime dead-letters it
 #: directly (never retried, never decoded; the DLQ record is the incoming raw body). The
@@ -67,6 +80,13 @@ UNSUPPORTED_SCHEMA_ROUTE_PREFIX = "babelqueue:unsupported-schema-version:"
 #: :meth:`KafkaTransport.publish` encodes back to the exact original bytes.
 NON_UTF8_ROUTE = UNSUPPORTED_SCHEMA_ROUTE_PREFIX + "non-utf8-payload"
 
+#: Reserved ``route_urn`` of a record whose value is not a decodable BabelQueue body (invalid JSON, a
+#: non-object root, a member of the wrong type). Like the gated routes it is dead-lettered directly and
+#: never retried, but the DLQ record is the *wrapped* poison body built by
+#: :meth:`KafkaTransport._poison_body` (``dead_letter.reason == "poison"``), not the raw bytes.
+POISON_ROUTE = UNSUPPORTED_SCHEMA_ROUTE_PREFIX + "poison"
+
+
 #: How long (seconds) a settle-path publish waits for the broker to confirm the write.
 DEFAULT_DELIVERY_TIMEOUT = 30.0
 
@@ -79,10 +99,13 @@ class KafkaDeliveryError(BabelQueueError):
 
 def _runtime_attempts(env: Any) -> int:
     """The envelope's ``attempts`` exactly as the runtime's retry path reads it
-    (``int(envelope.get("attempts", 0))``): an absent key is 0. Raises ``ValueError`` for every
-    value that read would choke on — null, ``""``, a list/dict, ``"three"``, ``inf``/``nan`` — and
-    for a bool, which is no counter at all although ``int()`` happens to accept it."""
+    (``int(envelope.get("attempts", 0))``): an absent key or a null is 0 (Go decodes a null integer
+    as zero). Raises ``ValueError`` for every value that read would choke on — ``""``, a list/dict,
+    ``"three"``, ``inf``/``nan`` — and for a bool, which is no counter at all although ``int()``
+    happens to accept it."""
     value = env.get("attempts", 0)
+    if value is None:
+        return 0
     if isinstance(value, bool):
         raise ValueError("attempts is a bool")
     try:
@@ -91,6 +114,11 @@ def _runtime_attempts(env: Any) -> int:
         raise ValueError("attempts is not an integer") from exc
 
 
+def _reject_json_constant(name: str) -> Any:
+    """``json.loads`` hook: ``NaN`` / ``Infinity`` are not JSON, so a body carrying them is undecodable."""
+    raise ValueError("invalid JSON constant " + name)
+
+
 def _brokers_from_url(url: str) -> str:
     rest = url.split("://", 1)[1] if "://" in url else url
     return rest.split("/", 1)[0] or "localhost:9092"
@@ -166,7 +194,11 @@ class KafkaTransport(Transport):
         env = parse_envelope(body)
         if not env:
             return []
-        meta = env.get("meta") or {}
+        meta = env.get("meta")
+        if not isinstance(meta, dict):
+            # A wrongly-typed ``meta`` (a poison body keeps its keys when dead-lettered, §6.5) has no
+            # header view; it must not fail the write that settles the record.
+            meta = {}
 
         headers: Headers = []
 
@@ -217,7 +249,7 @@ class KafkaTransport(Transport):
                 except (ValueError, TypeError):
                     pass
                 break
-        if attempts == body_attempts:
+        if attempts == body_attempts and env.get("attempts", 0) is not None:
             return body
         env["attempts"] = attempts
         return EnvelopeCodec.encode(env)
@@ -265,18 +297,89 @@ class KafkaTransport(Transport):
         return key_of() if callable(key_of) else None
 
     @staticmethod
-    def _has_non_numeric_attempts(body: str) -> bool:
-        """Whether the body is an envelope whose ``attempts`` the runtime's retry path could
-        not read (see ``_runtime_attempts``), so such a record could never be retried. An absent
-        ``attempts`` is a plain 0, not a poison."""
-        env = parse_envelope(body)
-        if not env or "attempts" not in env:
-            return False
+    def _decode_failure(text: str) -> Optional[str]:
+        """Why ``text`` is not a decodable BabelQueue body, or ``None`` when it is.
+
+        The core decoder is lenient (``decode`` returns ``{}`` for a malformed or non-object body),
+        so poison is detected explicitly, with the trigger set of the Go reference's typed
+        ``Decode`` but case-sensitive member names, as the core decoder reads them (see :mod:`babelqueue._envelope_shape`). A well-typed but non-conformant envelope
+        is not poison: it is routed by its URN, as in Go.
+
+        The shape scan keeps unknown members out of its way, but the core decoder parses the whole
+        body with plain ``json.loads`` and turns a ``ValueError`` into an EMPTY envelope (on Python
+        3.11+, e.g. an unknown member holding an integer of more than 4300 digits). Such a body is
+        poison here too, so content is never silently dropped on the way to a handler."""
+        violation = envelope_violation(text)
+        if violation is not None:
+            return violation
         try:
-            _runtime_attempts(env)
-        except ValueError:
-            return True
-        return False
+            json.loads(text)
+        except ValueError as exc:
+            return "the core decoder cannot parse the body: " + str(exc)
+        return None
+
+    @staticmethod
+    def _header_value(headers: Any, wanted: str) -> Optional[str]:
+        """The text of the first record header named ``wanted`` (``None`` when absent)."""
+        for key, value in headers or []:
+            name = key.decode("utf-8", "replace") if isinstance(key, (bytes, bytearray)) else key
+            if name != wanted:
+                continue
+            if isinstance(value, (bytes, bytearray)):
+                return bytes(value).decode("utf-8", "replace")
+            return None if value is None else str(value)
+        return None
+
+    @classmethod
+    def _header_attempts(cls, headers: Any) -> int:
+        """The ``bq-attempts`` header as a non-negative int (0 when absent or unparseable)."""
+        raw = cls._header_value(headers, "bq-attempts")
+        if raw is None:
+            return 0
+        try:
+            return max(int(raw), 0)
+        except (ValueError, TypeError):
+            return 0
+
+    @classmethod
+    def _poison_body(cls, text: str, cause: str, original_queue: str, attempts: int) -> str:
+        """The DLQ body of a poison record, in the form of the Go reference (§6.5, ADR-0009): a body
+        that is exactly one JSON object keeps its keys, anything else is carried as the string value
+        of ``raw`` (bytes that are not UTF-8 are stored with U+FFFD substitutions); a ``dead_letter``
+        block with ``reason: "poison"`` replaces any existing one. A body whose keys cannot be written
+        back as valid JSON (a number past the double range such as ``1e400``) falls back to ``raw``."""
+        # Bytes that are not UTF-8 are carried as surrogateescape code points; the DLQ body must be
+        # encodable, so they become U+FFFD (the Go reference stores the lossy conversion too).
+        text = text.encode("utf-8", "surrogateescape").decode("utf-8", "replace")
+        try:
+            parsed = json.loads(text, parse_constant=_reject_json_constant)
+        except (ValueError, RecursionError):
+            parsed = None
+        candidates: List[Dict[str, Any]] = []
+        if isinstance(parsed, dict):
+            parsed.pop("dead_letter", None)
+            candidates.append(parsed)
+        candidates.append({"raw": text})
+        for record in candidates:
+            annotated = dead_letter.annotate(
+                record,
+                "poison",
+                original_queue,
+                attempts,
+                error="Undecodable BabelQueue body: " + cause,
+                exception=BabelQueueError.__name__,
+            )
+            try:
+                serialised = json.dumps(
+                    annotated, ensure_ascii=False, separators=(",", ":"), allow_nan=False
+                )
+                # A lone surrogate (a ``\ud800`` escape in a key or value) serialises but cannot be
+                # written to the wire as UTF-8: fall back to the next candidate.
+                serialised.encode("utf-8")
+                return serialised
+            except (ValueError, RecursionError):
+                continue
+        raise AssertionError("unreachable: the raw fallback is always serialisable")  # pragma: no cover
 
     @staticmethod
     def _is_non_utf8(message: Any) -> bool:
@@ -308,6 +411,11 @@ class KafkaTransport(Transport):
         ``key`` is the consumed record's key (:attr:`ReceivedMessage.key`): it is written back
         unchanged, so a retry or DLQ record keeps the original key (``None`` stays a null key, an
         empty key stays empty, and no key is ever synthesised). §6."""
+        if isinstance(route_urn, PoisonRoute) and queue != route_urn.queue:
+            # The runtime dead-letters a poison record (§6.5): the DLQ body is the Go-form wrap.
+            body = self._poison_body(
+                body, route_urn.cause, route_urn.original_queue, route_urn.attempts
+            )
         kwargs = self._record_kwargs(body, raw_ok=True)
         if key is not None:
             kwargs["key"] = key
@@ -325,11 +433,22 @@ class KafkaTransport(Transport):
         except UnicodeEncodeError:
             if not raw_ok:
                 raise
-            return {"value": body.encode("utf-8", "surrogateescape"), "headers": []}
+            # Bytes that were not UTF-8 are carried as ``surrogateescape`` code points and restored
+            # byte for byte. A *lone* surrogate (a ``\\ud800`` JSON escape that survived the decode) is
+            # not restorable: it becomes U+FFFD, as the Go reference does when it parses the escape.
+            # Without this the settle write would raise and the record would be redelivered forever.
+            cleaned = _LONE_SURROGATE.sub("\ufffd", body)
+            return {"value": cleaned.encode("utf-8", "surrogateescape"), "headers": []}
         kwargs: Dict[str, Any] = {"value": body.encode("utf-8"), "headers": cls._projection(body)}
-        created_at = (parse_envelope(body).get("meta") or {}).get("created_at")
+        meta = parse_envelope(body).get("meta")
+        created_at = meta.get("created_at") if isinstance(meta, dict) else None
         if created_at:
-            kwargs["timestamp"] = int(created_at)
+            try:
+                kwargs["timestamp"] = int(created_at)
+            except (ValueError, TypeError, OverflowError):
+                # A poison body keeps its keys when wrapped (§6.5), so ``created_at`` can be anything:
+                # a record timestamp is optional and must not fail the write that settles it.
+                pass
         return kwargs
 
     def _produce_confirmed(self, producer: Any, queue: str, kwargs: Dict[str, Any]) -> None:
@@ -421,45 +540,49 @@ class KafkaTransport(Transport):
             )
             self._pending.setdefault(queue, []).append(message)
             return received
-        if self._is_non_utf8(message):
-            # The value cannot be decoded, so there is no envelope to retry or annotate: hand it
-            # over under the reserved route; the runtime dead-letters the raw bytes directly (and
-            # does not commit if that write is not confirmed) instead of letting ``pop`` raise.
-            _logger.warning(
-                "Kafka record on %r is not valid UTF-8; dead-lettering its raw bytes.", queue
-            )
-            received = ReceivedMessage(
-                body=self._payload(message),
-                queue=queue,
-                handle=message,
-                route_urn=NON_UTF8_ROUTE,
-                key=self._key(message),
-            )
-            self._pending.setdefault(queue, []).append(message)
-            return received
         payload = self._payload(message)
-        if self._has_non_numeric_attempts(payload):
-            # A foreign producer wrote a non-numeric ``attempts``. The runtime could not retry it
-            # (it needs an integer counter) and a failing handler would wedge the partition, so it
-            # is handed over under the reserved route like a non-UTF-8 record: dead-lettered raw,
-            # never given to a handler, and restored (not replayed) by ``redrive``.
-            _logger.warning(
-                "Kafka record on %r carries a non-numeric attempts; dead-lettering it raw.", queue
-            )
+        non_utf8 = self._is_non_utf8(message)
+        cause: Optional[str]
+        if non_utf8:
+            cause = "the value is not valid UTF-8"
+        else:
+            try:
+                cause = self._decode_failure(payload)
+            except Exception as exc:  # noqa: BLE001 - e.g. RecursionError on a pathologically nested body
+                cause = "the body could not be scanned (%s)" % type(exc).__name__
+        body = payload
+        if cause is None:
+            try:
+                body = self._reconcile(payload, headers)
+            except Exception as exc:  # noqa: BLE001 - any decode failure is poison, never a wedged partition
+                cause = "the body could not be decoded (%s)" % type(exc).__name__
+        if cause is not None:
+            # §6.5 poison: there is no envelope to retry or hand to a handler, so the record is handed
+            # over, raw, under a reserved route and dead-lettered directly (never retried; dropped when
+            # no DLQ is configured). The route carries what ``publish_settle`` needs to wrap the DLQ
+            # body like the Go reference (``dead_letter.reason == "poison"``, the original key); a
+            # restore to the queue the record was read from (``redrive``) stays byte for byte.
+            _logger.warning("Kafka record on %r is poison (%s); dead-lettering it.", queue, cause)
+            base = NON_UTF8_ROUTE if non_utf8 else POISON_ROUTE
             received = ReceivedMessage(
                 body=payload,
                 queue=queue,
                 handle=message,
-                route_urn=NON_NUMERIC_ATTEMPTS_ROUTE,
+                route_urn=PoisonRoute(
+                    base,
+                    queue,
+                    self._header_value(headers, "bq-original-topic") or queue,
+                    self._header_attempts(headers),
+                    cause,
+                ),
                 key=self._key(message),
             )
             self._pending.setdefault(queue, []).append(message)
             return received
-        body = self._reconcile(payload, headers)
         received = ReceivedMessage(body=body, queue=queue, handle=message, key=self._key(message))
-        # Track the record as in flight only once ``pop`` has built it. A decode error above
-        # escapes before this point and must not leave the record pending: the next ``pop`` would
-        # seek back to it and fail identically, wedging the partition forever.
+        # Track the record as in flight only once ``pop`` has built it. Any decode error above is
+        # routed as poison instead of escaping: an escape would leave the record uncommitted, so the
+        # next ``pop`` would seek back to it and fail identically, wedging the partition forever.
         self._pending.setdefault(queue, []).append(message)
         return received
 

{babelqueue-1.18.0 → babelqueue-1.19.0}/src/babelqueue/redrive.py

@@ -40,6 +40,7 @@ from .transport import (
     NON_NUMERIC_ATTEMPTS_ROUTE,
     BatchPopper,
     HeaderPublisher,
+    PoisonRoute,
     ReceivedMessage,
     Transport,
     publish_keeping_route,
@@ -97,12 +98,15 @@ def redrive(
                 break
             popped.append(message)
     batch: List[Tuple[ReceivedMessage, Optional[Dict[str, Any]]]] = [
-        (message, _decoded(message.body)) for message in popped
+        (message, None if isinstance(message.route_urn, PoisonRoute) else _decoded(message.body))
+        for message in popped
     ]
 
     result = RedriveResult()
     for message, envelope in batch:
-        # A record the transport flagged as non-numeric-``attempts`` is not redrivable either: a
+        # A record the transport flagged as poison (``PoisonRoute``: it never decoded into an envelope,
+        # whatever its ``job`` looks like) is restored to the DLQ as-is rather than replayed or re-wrapped,
+        # like the Go reference. A record the transport flagged as non-numeric-``attempts`` is not redrivable either: a
         # replay would reset it to a valid counter and quietly repair a foreign producer's bad
         # envelope, so it is restored to the DLQ like any other poison body.
         if envelope is None or message.route_urn == NON_NUMERIC_ATTEMPTS_ROUTE:

{babelqueue-1.18.0 → babelqueue-1.19.0}/src/babelqueue/transport.py

@@ -50,6 +50,33 @@ class ReceivedMessage:
     key: Optional[bytes] = None
 
 
+class PoisonRoute(str):
+    """The ``route_urn`` a poison record is delivered under (the reserved ``NON_UTF8_ROUTE`` / ``POISON_ROUTE`` of a transport that dead-letters undecodable records, today Kafka). It compares and hashes exactly like the
+    plain route string, so the runtime and ``redrive`` keep recognising the reserved routes, but it
+    also carries what is needed to build the wrapped DLQ body when the record is dead-lettered:
+    the consumed queue, the original queue, the ``bq-attempts`` header value and the failure cause.
+
+    The wrap is applied by :meth:`KafkaTransport.publish_settle`, and only when the record is written
+    to a queue other than the one it was consumed from. That is exactly the runtime's dead-letter
+    write; ``redrive`` restores a poison record to the DLQ it popped it from, which therefore stays
+    byte for byte, whatever that DLQ is named (``dead_letter_queue`` / ``dead_letter_suffix``)."""
+
+    queue: str
+    original_queue: str
+    attempts: int
+    cause: str
+
+    def __new__(
+        cls, route: str, queue: str, original_queue: str, attempts: int, cause: str
+    ) -> "PoisonRoute":
+        obj = super().__new__(cls, route)
+        obj.queue = queue
+        obj.original_queue = original_queue
+        obj.attempts = attempts
+        obj.cause = cause
+        return obj
+
+
 class Transport(ABC):
     """Minimal broker contract: publish a raw body, pop one, acknowledge it."""