babelqueue 1.17.0__tar.gz → 1.19.0__tar.gz

{babelqueue-1.17.0 → babelqueue-1.19.0}/CHANGELOG.md

@@ -9,6 +9,81 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.19.0] - 2026-10-12
+
+### Changed
+- Tests only: the vendored conformance suite is now 1.3.0 (exact-case envelope keys, last-wins duplicates: new `invalid-wrong-case-job`, `wrong-case-keys-roundtrip` and `duplicate-keys-last-wins` fixtures). No production change; the Python core already behaved this way.
+- **Kafka: a poison record is dead-lettered wrapped (`dead_letter.reason: "poison"`), like the Go
+  reference (§6.5, ADR-0009).** A value that is not valid UTF-8, not valid JSON, not a JSON object, or has
+  a member of the wrong type (the exact set is listed in the next entry) used to reach `<topic>.dlq` as its raw
+  bytes. It is now written as a body that keeps its keys when it is exactly one JSON object, otherwise
+  `{"raw": "<body>"}` (bytes that are not UTF-8 are stored with U+FFFD substitutions), plus a
+  `dead_letter` block (`reason: "poison"`, `original_queue`, `attempts` from the `bq-attempts` header, `lang`);
+  the record key is preserved. A poison record is still never retried and is dropped when no DLQ is
+  configured (the `on_unknown_urn` strategy, `DELETE` included, no longer matters: see below). The version-gate rejection (`bq-schema-version`, §6.7) still
+  dead-letters its record raw. The wrap is applied when the runtime writes the DLQ record (not when the
+  record is read), so a poison record that `redrive` restores into the DLQ it popped it from stays byte
+  for byte, whatever that DLQ is named. `original_queue` is the `bq-original-topic` header when present
+  (else the consumed topic), like the Java, .NET and Node consumers; a body whose numbers cannot be
+  written back as valid JSON (e.g. `1e400`) is carried as `{"raw": ...}` instead of producing an invalid DLQ body.
+  A body that decodes but names no URN is not poison (unknown-URN path, as before). Consumers of the DLQ
+  that expected the raw bytes of a malformed record must now read the wrapped body.
+- **Kafka: the poison trigger set now equals what the Go reference's typed `Decode` rejects.** Behaviour
+  change; the old Python-only rules (a lenient `attempts` read, `Accepts`-style conformance checks) are gone.
+  A record is poison when it is not valid UTF-8, not valid JSON (an empty or whitespace-only body, a BOM,
+  trailing data, leading zeros, `NaN` / `Infinity`, single quotes), or not a JSON object (a bare `null` is
+  not poison: it is an empty envelope and takes the unknown-URN path), or when a member has the wrong type:
+  `job`, `trace_id`, `meta.id|queue|lang` and `dead_letter.reason|error|exception|original_queue|lang`
+  must be a string or null; `meta.schema_version`, `meta.created_at`, `attempts` and
+  `dead_letter.failed_at|attempts` must be an int64 integer literal or null (a string such as `"3"` or
+  `"2026"`, a fraction or exponent such as `3.0` / `1e2`, a bool, an array and an object are poison);
+  `data`, `meta` and `dead_letter` must be an object or null; every number inside `data` must be a finite
+  double. Member names are matched exactly as spelled (case-sensitive, as the contract schema and the core decoder read them; a `JOB` or `Meta` member is an unknown member, not a typed one) and every duplicate key is checked. A record Go decodes
+  is no longer poison, even when it is not a conformant envelope: a non-string `urn` alias, an absent
+  `data`, `"attempts": null` (read as 0 and written back as 0) and `meta.schema_version: 2` are routed by URN
+  like any other record (Go's consume path never calls `Accepts`). The Kafka transport no longer
+  delivers records under the shared non-numeric-`attempts` route; they are plain poison now.
+  `redrive` restores a poison-routed record into the DLQ unchanged instead of replaying or re-wrapping it,
+  even when its `job` still reads as a URN. The only Python-specific limit is JSON nested deeper than
+  the interpreter's recursion limit (about 1000 levels, Go allows 10000), which is treated as poison.
+- Known, intentional differences from the Go reference for a poison record: `dead_letter.attempts` is the `bq-attempts` header (Go writes 0) and `original_queue` is the `bq-original-topic` header when present (Go writes the consumed topic); a float literal such as `1.0` / `1e2` is re-formatted when the body is re-encoded. The wire envelope and the `dead_letter` shape are unchanged.
+- **Kafka: any decode failure is poison; the partition is never left to wedge.** Member names are matched case-sensitively, exactly as the core decoder reads them (no `encoding/json`-style folding, no `str.casefold()`); the Go reference's case-insensitive read is a known difference that contradicts the contract schema and is tightened separately. Whatever `EnvelopeCodec.decode` raises after the shape scan (e.g. `RecursionError` at the interpreter's limit) is now routed as poison instead of escaping `pop` (which left the offset uncommitted and the partition redelivered forever); the poison wrapper falls back to `{"raw": ...}` on `RecursionError` or a body that cannot be written as UTF-8. A lone surrogate escape (`"\ud800"`) that survives into a retry/DLQ write is substituted by U+FFFD, as Go does, instead of raising `UnicodeEncodeError`. Invalid UTF-8 remains poison (Go substitutes U+FFFD and accepts it). The wire envelope and the `dead_letter` shape are unchanged.
+- **Kafka: a body the core JSON parser refuses is poison, not an empty envelope.** The core decoder turns a `ValueError` from `json.loads` into an empty envelope; where the shape scan accepts a body that plain `json.loads` rejects (Python 3.11+: an unknown member holding an integer of more than 4300 digits), the record would have reached a handler with its content dropped. It is now dead-lettered as poison. No change to the wire envelope or the `dead_letter` shape.
+- **Kafka: a poison record is dead-lettered whatever `on_unknown_urn` is.** The runtime checked the `DELETE` strategy before it recognised a poison record, so with `on_unknown_urn="delete"` a poison record was acknowledged and silently dropped even with a DLQ configured, where the Go reference writes it to `<topic>.dlq` with `dead_letter.reason: "poison"` and commits. A poison delivery (`PoisonRoute`) is now settled before the strategy is consulted: written wrapped to the DLQ and committed under `fail`, `delete`, `dead_letter` and `release`, and dropped only when no DLQ is configured (unchanged). The version-gate path and every other transport keep their behaviour. No change to the wire envelope or the `dead_letter` shape.
+
+### Fixed
+- Kafka: a wrongly-typed `meta` or `meta.created_at` in a wrapped poison body no longer fails the write
+  that settles it (header projection and record timestamp tolerate it).
+
+## [1.18.0] - 2026-10-11
+
+### Added
+- `BatchPopper` (optional transport capability, exported from `babelqueue`) with
+  `pop_batch(queue, max=0, timeout=1.0)`: take several deliveries at once and hold them un-settled.
+  `KafkaTransport` implements it; `redrive` uses it when a transport offers it.
+
+### Fixed
+- **`redrive` no longer loops forever (default `max=0`) or repeats records (`max>1`) on Kafka.**
+  `redrive` drains the DLQ before it processes it, but `KafkaTransport.pop` first seeks back to the
+  previous un-acked record, so every pop returned the first record again. The drain now takes the
+  batch through `pop_batch` (one rewind up front, none between the polls), and `KafkaTransport`
+  tracks several in-flight records: acks are committed in offset order, so the cumulative commit
+  never passes a record that is still unsettled, and an abandoned batch is rewound to its lowest
+  offset by the next `pop`. The restore writes are not re-read in the same call. Other transports
+  and the single-record `pop`/`ack`/`rewind` behaviour are unchanged. No wire change.
+  A record whose envelope carries an `attempts` the runtime cannot read as an integer (a foreign
+  producer: `null`, `""`, a list/object, a bool, `"three"`, `NaN`/`Infinity`; an absent key is just 0) no longer raises out
+  of `pop` or the restore publish. `KafkaTransport.pop` hands it over under a reserved poison route
+  (like a non-UTF-8 record), so the runtime dead-letters it raw without running a handler (a failing
+  handler can no longer wedge the partition) and `redrive` restores it to the DLQ instead of
+  replaying it. The redundant `bq-attempts` header is left out of such a write.
+- **`redrive` now carries the DLQ record key onto the redriven record.** The restore writes back to
+  the DLQ already kept the key; the redrive write to the work topic did not, so a replayed Kafka
+  record lost its key (and per-key partition affinity). It now goes through
+  `publish_keeping_route(..., key=message.key)`, byte-for-byte (null stays null, empty stays empty),
+  per contract §6.5 "Record key on republish". Non-keyed transports behave exactly as before; the
+  `bypass=True` header path still carries no key. No wire change.
+
 ## [1.17.0] - 2026-10-11
 
 ### Added

{babelqueue-1.17.0 → babelqueue-1.19.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: babelqueue
-Version: 1.17.0
+Version: 1.19.0
 Summary: Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers.
 Project-URL: Homepage, https://babelqueue.com
 Project-URL: Source, https://github.com/BabelQueue/babelqueue-python

{babelqueue-1.17.0 → babelqueue-1.19.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.17.0"
+version = "1.19.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"

{babelqueue-1.17.0 → babelqueue-1.19.0}/src/babelqueue/__init__.py

@@ -30,6 +30,7 @@ from .exceptions import BabelQueueError, DecryptError, UnknownUrnError
 from .replay import HEADER_REPLAY_BYPASS, bypass_external_effects, is_replay
 from .routing import UnknownUrnStrategy
 from .transport import (
+    BatchPopper,
     HeaderPublisher,
     InMemoryTransport,
     ReceivedMessage,
@@ -40,7 +41,7 @@ from .transport import (
     Transport,
 )
 
-__version__ = "1.17.0"
+__version__ = "1.19.0"
 
 __all__ = [
     "BabelQueue",
@@ -53,6 +54,7 @@ __all__ = [
     "Transport",
     "InMemoryTransport",
     "ReceivedMessage",
+    "BatchPopper",
     "HeaderPublisher",
     "Redeliverer",
     "Rewinder",

babelqueue-1.19.0/src/babelqueue/_envelope_shape.py

@@ -0,0 +1,153 @@
+"""The poison trigger set of the Kafka transport (§6.5).
+
+A body is undecodable when a strict, typed decode into ``Envelope`` / ``Meta`` / ``DeadLetter`` (the
+Go reference's ``babelqueue.Decode``, except that member names are case-sensitive here) rejects it:
+
+* the body must be exactly one JSON value (no BOM, trailing tokens, ``NaN`` / ``Infinity``, malformed
+  numbers); a bare ``null`` decodes to an empty envelope, any other non-object root is rejected;
+* ``job``, ``trace_id``, ``meta.id|queue|lang`` and ``dead_letter.reason|error|exception|original_queue|lang``
+  are string-or-null;
+* ``attempts``, ``meta.schema_version|created_at`` and ``dead_letter.failed_at|attempts`` are
+  integer-or-null: an integer literal (no fraction or exponent) that fits ``int64``;
+* ``data``, ``meta`` and ``dead_letter`` are object-or-null, and a number inside ``data`` must be a
+  finite double;
+* member names are matched exactly as spelled (case-sensitive, as the contract schema and the core
+  decoder read them) and every occurrence of a duplicated key is checked; any other member (a
+  differently-cased name or a non-string ``urn`` alias included) is only checked for syntax.
+
+A well-typed but non-conformant envelope is not poison: Go's consume path never calls ``Accepts``.
+"""
+
+import json
+import math
+from typing import Any, Dict, List, Optional, Tuple
+
+_INT64_MIN = -(2**63)
+_INT64_MAX = 2**63 - 1
+#: Beyond this many digits an integer literal cannot fit an int64 or a double; it is kept as a
+#: sentinel so ``json`` never trips the interpreter's integer-string-length limit.
+_MAX_INT_DIGITS = 4000
+
+
+class _Pairs(List[Tuple[str, Any]]):
+    """A decoded JSON object as its ``(key, value)`` list, so duplicate keys survive."""
+
+
+class _HugeInt:
+    """An integer literal too large for any int64 or double."""
+
+
+_HUGE = _HugeInt()
+
+_STRING = "string"
+_INTEGER = "integer"
+_OBJECT = "object"
+_DATA = "data"
+
+_META = {
+    "id": _STRING,
+    "queue": _STRING,
+    "lang": _STRING,
+    "schema_version": _INTEGER,
+    "created_at": _INTEGER,
+}
+_DEAD_LETTER = {
+    "reason": _STRING,
+    "error": _STRING,
+    "exception": _STRING,
+    "original_queue": _STRING,
+    "lang": _STRING,
+    "failed_at": _INTEGER,
+    "attempts": _INTEGER,
+}
+_TOP = {
+    "job": _STRING,
+    "trace_id": _STRING,
+    "attempts": _INTEGER,
+    "data": _DATA,
+    "meta": _OBJECT,
+    "dead_letter": _OBJECT,
+}
+
+
+def _reject_constant(name: str) -> Any:
+    raise ValueError("invalid JSON constant " + name)
+
+
+def _parse_int(literal: str) -> Any:
+    return int(literal) if len(literal) <= _MAX_INT_DIGITS else _HUGE
+
+
+def _is_finite_number(value: Any) -> bool:
+    if value is _HUGE:
+        return False
+    if isinstance(value, float):
+        return math.isfinite(value)
+    if isinstance(value, int) and not isinstance(value, bool):
+        try:
+            return math.isfinite(float(value))
+        except OverflowError:
+            return False
+    return True
+
+
+def _all_numbers_finite(root: Any) -> bool:
+    stack = [root]
+    while stack:
+        value = stack.pop()
+        if isinstance(value, _Pairs):
+            stack.extend(item for _, item in value)
+        elif isinstance(value, list):
+            stack.extend(value)
+        elif not _is_finite_number(value):
+            return False
+    return True
+
+
+def _check_member(kind: str, name: str, value: Any) -> Optional[str]:
+    if value is None:
+        return None
+    if kind == _STRING:
+        return None if isinstance(value, str) else "member %r is not a string" % name
+    if kind == _INTEGER:
+        if type(value) is int and _INT64_MIN <= value <= _INT64_MAX:
+            return None
+        return "member %r is not an int64 integer" % name
+    if not isinstance(value, _Pairs):
+        return "member %r is not an object" % name
+    if kind == _DATA and not _all_numbers_finite(value):
+        return "member %r holds a number that is not a finite double" % name
+    return None
+
+
+def _check_object(pairs: _Pairs, kinds: Dict[str, str], prefix: str) -> Optional[str]:
+    for key, value in pairs:
+        kind = kinds.get(key)
+        if kind is None:
+            continue
+        failure = _check_member(kind, prefix + key, value)
+        if failure is not None:
+            return failure
+        if kind == _OBJECT and value is not None:
+            nested = _check_object(value, _META if key == "meta" else _DEAD_LETTER, key + ".")
+            if nested is not None:
+                return nested
+    return None
+
+
+def envelope_violation(text: str) -> Optional[str]:
+    """Why ``text`` is not a decodable BabelQueue body, or ``None`` when Go's typed decode accepts it."""
+    try:
+        parsed = json.loads(
+            text,
+            object_pairs_hook=_Pairs,
+            parse_int=_parse_int,
+            parse_constant=_reject_constant,
+        )
+    except (ValueError, RecursionError) as exc:
+        return "invalid JSON: " + str(exc)
+    if parsed is None:
+        return None  # a bare ``null`` is an empty envelope, not poison
+    if not isinstance(parsed, _Pairs):
+        return "the JSON root is not an object"
+    return _check_object(parsed, _TOP, "")

{babelqueue-1.17.0 → babelqueue-1.19.0}/src/babelqueue/app.py

@@ -41,6 +41,7 @@ from .replay import HEADER_REPLAY_BYPASS, _replay_scope
 from .routing import UnknownUrnStrategy
 from .transport import (
     HeaderPublisher,
+    PoisonRoute,
     ReceivedMessage,
     Redeliverer,
     Rewinder,
@@ -476,6 +477,14 @@ class BabelQueue:
 
     def _route_unknown(self, urn: str, received: ReceivedMessage, envelope: Mapping[str, Any]) -> None:
         strategy = self.on_unknown_urn
+        # A Kafka poison record (§6.5: not valid UTF-8 / not a decodable body) is not an unknown URN: it
+        # never decoded into an envelope, so ``on_unknown_urn`` has no say. It is checked before the
+        # DELETE strategy and settled like the Go reference does -- dead-lettered (wrapped, reason
+        # ``poison``) and committed, or dropped only when no DLQ is configured. Other transports'
+        # version-gated deliveries keep their existing order below.
+        if isinstance(received.route_urn, PoisonRoute):
+            self._settle_version_gated(received)
+            return
         if strategy == UnknownUrnStrategy.DELETE:
             self._ack_settled(received)
             return