babelqueue 1.16.1__tar.gz → 1.18.0__tar.gz

{babelqueue-1.16.1 → babelqueue-1.18.0}/CHANGELOG.md

@@ -9,6 +9,70 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.18.0] - 2026-10-11
+
+### Added
+- `BatchPopper` (optional transport capability, exported from `babelqueue`) with
+  `pop_batch(queue, max=0, timeout=1.0)`: take several deliveries at once and hold them un-settled.
+  `KafkaTransport` implements it; `redrive` uses it when a transport offers it.
+
+### Fixed
+- **`redrive` no longer loops forever (default `max=0`) or repeats records (`max>1`) on Kafka.**
+  `redrive` drains the DLQ before it processes it, but `KafkaTransport.pop` first seeks back to the
+  previous un-acked record, so every pop returned the first record again. The drain now takes the
+  batch through `pop_batch` (one rewind up front, none between the polls), and `KafkaTransport`
+  tracks several in-flight records: acks are committed in offset order, so the cumulative commit
+  never passes a record that is still unsettled, and an abandoned batch is rewound to its lowest
+  offset by the next `pop`. The restore writes are not re-read in the same call. Other transports
+  and the single-record `pop`/`ack`/`rewind` behaviour are unchanged. No wire change.
+  A record whose envelope carries an `attempts` the runtime cannot read as an integer (a foreign
+  producer: `null`, `""`, a list/object, a bool, `"three"`, `NaN`/`Infinity`; an absent key is just 0) no longer raises out
+  of `pop` or the restore publish. `KafkaTransport.pop` hands it over under a reserved poison route
+  (like a non-UTF-8 record), so the runtime dead-letters it raw without running a handler (a failing
+  handler can no longer wedge the partition) and `redrive` restores it to the DLQ instead of
+  replaying it. The redundant `bq-attempts` header is left out of such a write.
+- **`redrive` now carries the DLQ record key onto the redriven record.** The restore writes back to
+  the DLQ already kept the key; the redrive write to the work topic did not, so a replayed Kafka
+  record lost its key (and per-key partition affinity). It now goes through
+  `publish_keeping_route(..., key=message.key)`, byte-for-byte (null stays null, empty stays empty),
+  per contract §6.5 "Record key on republish". Non-keyed transports behave exactly as before; the
+  `bypass=True` header path still carries no key. No wire change.
+
+## [1.17.0] - 2026-10-11
+
+### Added
+- `SettlePublisher` (optional transport capability, exported from `babelqueue`) with
+  `publish_settle(queue, body, route_urn="")`: the runtime's settle writes (retry republish,
+  envelope / poison / gate-rejection DLQ, `redrive` restore) go through it when a transport
+  implements it. `KafkaTransport` implements it.
+- `KafkaDeliveryError` (a `BabelQueueError`) and the optional `KafkaTransport(delivery_timeout=30.0)`
+  constructor argument.
+- `ReceivedMessage.key` (`Optional[bytes]`, `None` = null key) and an optional `key=` argument on
+  `SettlePublisher.publish_settle()` / `publish_keeping_route()`. `KafkaTransport.pop()` fills the
+  key from the consumed record.
+
+### Fixed
+- **Kafka: the original record key now survives the retry republish, the envelope DLQ and the
+  raw (poison / gate-rejection) DLQ.** The key was dropped on every settle write, so a keyed record
+  lost its key (and with it per-key partition affinity) after the first retry. A null key stays
+  null, an empty key stays empty, and a key is never synthesised. `redrive` restores to `<queue>.dlq`
+  also keep it. Partition equality is not guaranteed. The envelope and the `dead_letter` block are
+  unchanged.
+- **Kafka: retry / dead-letter writes are now broker-confirmed before the offset is committed.**
+  `KafkaTransport.publish_settle()` waits for this write's own delivery report
+  (`flush(delivery_timeout)`). A delivery error or timeout raises `KafkaDeliveryError`, nothing is
+  committed, and the record is re-read through the existing rewind/seek path — so such a write can
+  no longer be lost to a commit that raced ahead of it. `publish()` (and so `app.publish()`) is
+  unchanged: still fire-and-forget and strict (a body that is not valid UTF-8 still raises
+  `UnicodeEncodeError`), also from inside a handler.
+- **Kafka: a record whose value is not valid UTF-8 is dead-lettered raw instead of raising.** The
+  original bytes are written to `<queue>.dlq` unchanged, with no projected headers (as
+  Java/Go/Node do for poison records), and the offset is committed only once that write is
+  confirmed; with the DLQ off, or `UnknownUrnStrategy.DELETE`, the record is dropped and
+  committed. It reuses the version-gate settle path, so a failed write logs as a "version-gated"
+  message — the logged route (`...:non-utf8-payload`) tells them apart. A non-UTF-8 `bq-attempts`
+  header no longer raises. No wire change.
+
 ## [1.16.1] - 2026-10-11
 
 ### Changed

{babelqueue-1.16.1 → babelqueue-1.18.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: babelqueue
-Version: 1.16.1
+Version: 1.18.0
 Summary: Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers.
 Project-URL: Homepage, https://babelqueue.com
 Project-URL: Source, https://github.com/BabelQueue/babelqueue-python

{babelqueue-1.16.1 → babelqueue-1.18.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.16.1"
+version = "1.18.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"

{babelqueue-1.16.1 → babelqueue-1.18.0}/src/babelqueue/__init__.py

@@ -30,16 +30,18 @@ from .exceptions import BabelQueueError, DecryptError, UnknownUrnError
 from .replay import HEADER_REPLAY_BYPASS, bypass_external_effects, is_replay
 from .routing import UnknownUrnStrategy
 from .transport import (
+    BatchPopper,
     HeaderPublisher,
     InMemoryTransport,
     ReceivedMessage,
     Redeliverer,
     Rewinder,
     RoutedPublisher,
+    SettlePublisher,
     Transport,
 )
 
-__version__ = "1.16.1"
+__version__ = "1.18.0"
 
 __all__ = [
     "BabelQueue",
@@ -52,10 +54,12 @@ __all__ = [
     "Transport",
     "InMemoryTransport",
     "ReceivedMessage",
+    "BatchPopper",
     "HeaderPublisher",
     "Redeliverer",
     "Rewinder",
     "RoutedPublisher",
+    "SettlePublisher",
     "BabelQueueError",
     "UnknownUrnError",
     "DecryptError",

{babelqueue-1.16.1 → babelqueue-1.18.0}/src/babelqueue/app.py

@@ -308,8 +308,9 @@ class BabelQueue:
                     # It is left unacked; the broker (or a rewind, for cumulative-offset brokers
                     # such as Kafka) brings it back and the gate rejects it again.
                     logger.error(
-                        "Dead-lettering a version-gated message on queue %r failed; it was NOT "
+                        "Dead-lettering a version-gated message (route %r) on queue %r failed; it was NOT "
                         "retried or acknowledged; it will come back and the dead-letter write will be retried.",
+                        received.route_urn,
                         received.queue,
                         exc_info=exc,
                     )
@@ -414,7 +415,7 @@ class BabelQueue:
         or its dead-letter record). Keeps the routing URN the broker property carried on this
         delivery when the transport can pin it (:class:`RoutedPublisher`), so a property/body
         mismatch does not flip to the body URN on the next attempt or in the DLQ."""
-        publish_keeping_route(self.transport, queue, body, received.route_urn)
+        publish_keeping_route(self.transport, queue, body, received.route_urn, received.key)
 
     def _ack_settled(self, received: ReceivedMessage) -> None:
         """Acknowledge a message whose outcome is already final (handled, dead-lettered,
@@ -447,7 +448,8 @@ class BabelQueue:
             self.transport.rewind(received)
         except Exception:  # noqa: BLE001 - reported, never kills the loop
             logger.error(
-                "Failed to rewind an unsettled version-gated message on queue %r.",
+                "Failed to rewind an unsettled version-gated message (route %r) on queue %r.",
+                received.route_urn,
                 received.queue,
                 exc_info=True,
             )