babelqueue 1.11.0__tar.gz → 1.12.0__tar.gz

{babelqueue-1.11.0 → babelqueue-1.12.0}/CHANGELOG.md

@@ -9,6 +9,31 @@ The envelope wire format is versioned separately by `meta.schema_version`
 
 ## [Unreleased]
 
+## [1.12.0] - 2026-06-21
+
+### Added
+- **Transactional outbox helper** (ADR-0029) — the optional `babelqueue.outbox` module ports the
+  PHP `BabelQueue\Outbox` helper to Python, removing the producer **dual write**: the message is
+  persisted **into your database, in the same transaction** as the business data (so it commits or
+  rolls back atomically with it), and a separate **relay** publishes the durable rows afterwards.
+  No distributed transaction; exactly-once *handoff* into the broker, then at-least-once on the wire
+  (the consumer dedupes on `meta.id` via the idempotency helper, the consumer-side mirror, ADR-0022).
+  The core stays **stdlib-only** (GR-7): `OutboxStore` is an abstract `typing.Protocol` the caller
+  binds to their own DB — the module ships only the in-memory `InMemoryOutboxStore` reference and
+  pulls in **no** DB driver. `Outbox.write(envelope)` encodes via the frozen `EnvelopeCodec` and
+  delegates to `OutboxStore.save` **inside the transaction the caller already opened** — it does not
+  begin/commit anything (the caller owns the transaction boundary). `OutboxRelay.flush()` publishes
+  one batch through the existing publish-only `Transport`, marking each row published **only after**
+  the transport accepts it, or failed (caught → `mark_failed`, row left pending, with a bounded
+  linear backoff via an injectable sleeper) so one poison row never blocks the batch;
+  `OutboxRelay.drain()` loops while a pass makes progress, with a safety ceiling. The relay
+  publishes the **stored bytes verbatim** — it never decodes, rebuilds or re-encodes the envelope —
+  so `trace_id` is preserved end-to-end and the body is byte-compatible across SDKs (GR-1/GR-4/GR-5).
+  Unit-tested without a broker (write stores the encoded envelope byte-identical; relay publishes via
+  a fake `Transport` + marks published; a raising publish → `mark_failed`, row still pending, batch
+  continues; `drain` loops to empty and stops on no-progress; backoff grows linearly and caps via the
+  injected sleeper). The envelope is unchanged (`schema_version: 1`); this is purely additive.
+
 ## [1.11.0] - 2026-06-21
 
 ### Added

{babelqueue-1.11.0 → babelqueue-1.12.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: babelqueue
-Version: 1.11.0
+Version: 1.12.0
 Summary: Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers.
 Project-URL: Homepage, https://babelqueue.com
 Project-URL: Source, https://github.com/BabelQueue/babelqueue-python
@@ -234,6 +234,47 @@ def on_created(data, meta): ...
 python manage.py babelqueue_worker --queue orders          # run the consumer
 ```
 
+## Transactional outbox (optional)
+
+The `babelqueue.outbox` helper (ADR-0029) removes the producer **dual write**: "commit the
+business row" and "publish to the broker" are two systems that can disagree on a crash. Instead the
+message is persisted **into your database, in the same transaction** as the business data — so it
+commits or rolls back atomically with it — and a separate **relay** publishes the durable rows
+afterwards. No distributed transaction; exactly-once *handoff* into the broker, then at-least-once
+on the wire (the consumer dedupes on `meta.id` — see the idempotency helper, the mirror of this).
+
+The core stays **stdlib-only**: `OutboxStore` is an abstract `Protocol` you bind to **your own DB**
+(the core ships no driver). The stored value is the `EnvelopeCodec`-encoded envelope **byte-for-byte
+unchanged** (frozen, `schema_version: 1`); the relay publishes those exact bytes — it never decodes,
+rebuilds or re-encodes — so `trace_id` is preserved end-to-end.
+
+```python
+from babelqueue import BabelQueue, EnvelopeCodec
+from babelqueue.outbox import Outbox, OutboxRelay, InMemoryOutboxStore
+
+store = InMemoryOutboxStore()          # production: your own OutboxStore adapter, DB-backed
+outbox = Outbox(store)
+
+# write side — YOU own the transaction boundary (this is the whole point):
+with db.transaction():                 # your own open transaction
+    db.insert_order(order)             # the business write
+    envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042}, queue="orders")
+    outbox.write(envelope)             # same connection, same tx — both, or neither
+
+# read/publish side — run on a short interval, after the business tx commits:
+app = BabelQueue("redis://localhost:6379/0", queue="orders")
+relay = OutboxRelay(app.transport, store)
+relay.drain()                          # publish all pending rows; flush() does one batch
+```
+
+`Outbox.write` only encodes and calls `OutboxStore.save` — it does **not** begin or commit anything.
+A `save` runs inside the transaction you already opened; you commit both together. `OutboxRelay`
+marks a row published only **after** the transport accepts it; a publish that raises is recorded via
+`mark_failed` (with a bounded, injectable-sleeper backoff) and left pending for a later pass, so one
+poison row never blocks the batch. Implement `OutboxStore` over your DB (claim rows oldest-first,
+ideally with `SELECT … FOR UPDATE SKIP LOCKED` so two relays don't double-publish); `InMemoryOutboxStore`
+is the reference for tests and single-process demos (no real transaction).
+
 ## OpenTelemetry tracing (optional)
 
 `pip install "babelqueue[otel]"` adds the optional `babelqueue.otel` module — the core never

{babelqueue-1.11.0 → babelqueue-1.12.0}/README.md

@@ -182,6 +182,47 @@ def on_created(data, meta): ...
 python manage.py babelqueue_worker --queue orders          # run the consumer
 ```
 
+## Transactional outbox (optional)
+
+The `babelqueue.outbox` helper (ADR-0029) removes the producer **dual write**: "commit the
+business row" and "publish to the broker" are two systems that can disagree on a crash. Instead the
+message is persisted **into your database, in the same transaction** as the business data — so it
+commits or rolls back atomically with it — and a separate **relay** publishes the durable rows
+afterwards. No distributed transaction; exactly-once *handoff* into the broker, then at-least-once
+on the wire (the consumer dedupes on `meta.id` — see the idempotency helper, the mirror of this).
+
+The core stays **stdlib-only**: `OutboxStore` is an abstract `Protocol` you bind to **your own DB**
+(the core ships no driver). The stored value is the `EnvelopeCodec`-encoded envelope **byte-for-byte
+unchanged** (frozen, `schema_version: 1`); the relay publishes those exact bytes — it never decodes,
+rebuilds or re-encodes — so `trace_id` is preserved end-to-end.
+
+```python
+from babelqueue import BabelQueue, EnvelopeCodec
+from babelqueue.outbox import Outbox, OutboxRelay, InMemoryOutboxStore
+
+store = InMemoryOutboxStore()          # production: your own OutboxStore adapter, DB-backed
+outbox = Outbox(store)
+
+# write side — YOU own the transaction boundary (this is the whole point):
+with db.transaction():                 # your own open transaction
+    db.insert_order(order)             # the business write
+    envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042}, queue="orders")
+    outbox.write(envelope)             # same connection, same tx — both, or neither
+
+# read/publish side — run on a short interval, after the business tx commits:
+app = BabelQueue("redis://localhost:6379/0", queue="orders")
+relay = OutboxRelay(app.transport, store)
+relay.drain()                          # publish all pending rows; flush() does one batch
+```
+
+`Outbox.write` only encodes and calls `OutboxStore.save` — it does **not** begin or commit anything.
+A `save` runs inside the transaction you already opened; you commit both together. `OutboxRelay`
+marks a row published only **after** the transport accepts it; a publish that raises is recorded via
+`mark_failed` (with a bounded, injectable-sleeper backoff) and left pending for a later pass, so one
+poison row never blocks the batch. Implement `OutboxStore` over your DB (claim rows oldest-first,
+ideally with `SELECT … FOR UPDATE SKIP LOCKED` so two relays don't double-publish); `InMemoryOutboxStore`
+is the reference for tests and single-process demos (no real transaction).
+
 ## OpenTelemetry tracing (optional)
 
 `pip install "babelqueue[otel]"` adds the optional `babelqueue.otel` module — the core never

{babelqueue-1.11.0 → babelqueue-1.12.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "babelqueue"
-version = "1.11.0"
+version = "1.12.0"
 description = "Polyglot Queues, Simplified — the Python core: the canonical BabelQueue wire-envelope codec, contracts and dead-letter helpers."
 readme = "README.md"
 requires-python = ">=3.9"

{babelqueue-1.11.0 → babelqueue-1.12.0}/src/babelqueue/__init__.py

@@ -11,18 +11,26 @@ and dead-letter helpers. Framework adapters (Celery, Django, ...) build on this.
 
 from __future__ import annotations
 
-from . import dead_letter, headers, idempotency, redrive, replay
+from . import dead_letter, headers, idempotency, outbox, redrive, replay
 from .app import BabelQueue
 from .codec import SCHEMA_VERSION, SOURCE_LANG, EnvelopeCodec
 from .contracts import HasTraceId, PolyglotMessage
 from .headers import headers_from_context
 from .idempotency import IdempotencyStore, InMemoryStore
+from .outbox import (
+    InMemoryOutboxStore,
+    Outbox,
+    OutboxRecord,
+    OutboxRelay,
+    OutboxRelayResult,
+    OutboxStore,
+)
 from .exceptions import BabelQueueError, UnknownUrnError
 from .replay import HEADER_REPLAY_BYPASS, bypass_external_effects, is_replay
 from .routing import UnknownUrnStrategy
 from .transport import HeaderPublisher, InMemoryTransport, ReceivedMessage, Transport
 
-__version__ = "1.11.0"
+__version__ = "1.12.0"
 
 __all__ = [
     "BabelQueue",
@@ -41,6 +49,13 @@ __all__ = [
     "dead_letter",
     "headers",
     "idempotency",
+    "outbox",
+    "Outbox",
+    "OutboxStore",
+    "OutboxRecord",
+    "OutboxRelay",
+    "OutboxRelayResult",
+    "InMemoryOutboxStore",
     "redrive",
     "replay",
     "is_replay",

babelqueue-1.12.0/src/babelqueue/outbox.py

@@ -0,0 +1,360 @@
+"""Optional transactional-outbox helper (ADR-0029): atomic write + relayed publish.
+
+The Python mirror of the PHP ``BabelQueue\\Outbox`` helper. It removes the producer
+**dual write** — "commit the business row" *and* "publish to the broker" are two systems
+that can disagree on a crash. Instead the message is written **into the same database, in
+the same transaction** as the business data (so it commits or rolls back atomically with
+it), and a separate :class:`OutboxRelay` publishes the durable rows afterwards. No
+distributed transaction; exactly-once *handoff* into the broker, then at-least-once on the
+wire as always (the consumer dedupes on ``meta.id`` — :mod:`babelqueue.idempotency`,
+ADR-0022, is the consumer-side mirror of this producer-side helper).
+
+    from babelqueue import EnvelopeCodec
+    from babelqueue.outbox import Outbox, OutboxRelay, InMemoryOutboxStore
+
+    store = InMemoryOutboxStore()          # production: a DB-backed OutboxStore adapter
+    outbox = Outbox(store)
+
+    # write side — the CALLER owns the transaction boundary (this is the whole point):
+    with db.transaction():                 # the caller's own open transaction
+        db.insert_order(order)             # the business write
+        envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042})
+        outbox.write(envelope)             # same connection, same tx — commits or rolls back together
+
+    # read/publish side — run on a short interval, after the business tx commits:
+    relay = OutboxRelay(app.transport, store)
+    relay.drain()                          # publish all pending rows through the Transport
+
+The helper is intentionally tiny and **stdlib-only** (GR-7): :class:`OutboxStore` is an
+abstract persistence contract the caller binds to their own DB — the core ships only the
+in-memory :class:`InMemoryOutboxStore` reference and pulls in **no** DB driver. The stored
+value is the :class:`~babelqueue.codec.EnvelopeCodec`-encoded envelope, **byte-for-byte
+unchanged** (GR-1): the relay publishes exactly those bytes — it never decodes, rebuilds or
+re-encodes — so ``trace_id`` is preserved end-to-end (GR-4) and the body is byte-compatible
+across SDKs (GR-5). The outbox's own columns (id, queue, attempts) are bookkeeping *around*
+the envelope, never a field *on* the wire.
+"""
+
+from __future__ import annotations
+
+import time
+from dataclasses import dataclass
+from typing import Any, Callable, Dict, List, Mapping, Protocol, runtime_checkable
+
+from .codec import EnvelopeCodec
+from .transport import Transport
+
+#: Sleep for the given number of seconds (the relay's backoff seam; default :func:`time.sleep`).
+Sleeper = Callable[[float], None]
+
+
+@dataclass
+class OutboxRecord:
+    """One pending row read back from an :class:`OutboxStore` for the :class:`OutboxRelay`.
+
+    It pairs the store's own bookkeeping (``id``, ``attempts``) with the verbatim, frozen
+    wire envelope (``body``) and the queue it should go to. ``body`` is the exact
+    :meth:`~babelqueue.codec.EnvelopeCodec.encode` output handed to :meth:`OutboxStore.save`;
+    the relay publishes these bytes unchanged (GR-1/GR-5), so ``trace_id`` is preserved
+    end-to-end (GR-4) without the relay ever decoding or rebuilding the envelope.
+    """
+
+    id: str  #: The outbox row id (the store's primary key, not ``meta.id``).
+    body: str  #: The frozen, encoded envelope JSON, byte-for-byte as stored.
+    queue: str  #: The logical queue the relay should publish to.
+    attempts: int = 0  #: How many times the relay has already tried to publish this row.
+
+
+@dataclass
+class OutboxRelayResult:
+    """Summary of one :meth:`OutboxRelay.flush` pass (or a whole :meth:`OutboxRelay.drain`):
+    how many pending rows were published and how many failed (and were left pending for a
+    later retry)."""
+
+    published: int = 0
+    failed: int = 0
+
+    @property
+    def attempted(self) -> int:
+        """Total rows the relay attempted in this pass."""
+        return self.published + self.failed
+
+
+@runtime_checkable
+class OutboxStore(Protocol):
+    """The persistence seam for the transactional outbox (ADR-0029) — the durable "outbox"
+    table that an :class:`Outbox` writer fills and an :class:`OutboxRelay` drains.
+
+    **The transaction boundary is the CALLER'S.** The core never opens, commits or rolls
+    back anything: :meth:`save` is invoked from *inside* a transaction the caller already
+    began (around its own ``INSERT INTO orders …``), and the caller commits both together.
+    This keeps the core free of any DB driver (GR-7): the core defines this contract; a
+    concrete adapter binds it to a real connection. The reference :class:`InMemoryOutboxStore`
+    is for tests and single-process demos.
+
+    The stored value is the **frozen wire envelope, byte-for-byte unchanged** (GR-1): an
+    :meth:`~babelqueue.codec.EnvelopeCodec.encode` JSON string. The outbox adds its own
+    bookkeeping columns (id, queue, attempts, status) *around* the envelope; it never adds a
+    field *to* it. What the relay publishes is the same bytes that were stored.
+    """
+
+    def save(self, encoded: str, queue: str) -> str:
+        """Persist one encoded envelope into the outbox, **within the transaction the caller
+        has already opened** around its business write. Return the new row's outbox id (the
+        store's own primary key, NOT ``meta.id``), which the caller may keep for correlation.
+        The body is stored verbatim; do not re-encode or mutate it."""
+        ...
+
+    def fetch_unpublished(self, limit: int) -> List[OutboxRecord]:
+        """Reserve up to ``limit`` rows that are pending publish, **oldest first**, so a relay
+        can forward them. Implementations SHOULD lock/claim the rows they return (e.g.
+        ``SELECT … FOR UPDATE SKIP LOCKED``, or a ``picked_at`` claim) so two concurrent relays
+        do not both publish the same row; at-least-once still tolerates a rare double send.
+        Return an empty list when the outbox is drained."""
+        ...
+
+    def mark_published(self, ids: List[str]) -> None:
+        """Mark the given outbox rows as successfully published (so they are never relayed
+        again). Called by the relay only **after** the transport accepted the message."""
+        ...
+
+    def mark_failed(self, id: str, error: str) -> None:
+        """Record a failed publish attempt for one row: increment its attempt counter and
+        store the last error, leaving it pending so a later relay pass retries it
+        (at-least-once). The store MAY move a row past a max-attempts threshold to a
+        terminal/parked state, but that policy is the adapter's, not the core's. ``error`` is
+        a short, human-readable failure reason (never secrets)."""
+        ...
+
+
+class Outbox:
+    """The **write side** of the transactional outbox (ADR-0029): turn a BabelQueue envelope
+    into a stored outbox row, so the message is persisted *atomically with the business data*
+    and a separate :class:`OutboxRelay` publishes it later.
+
+    Usage — the caller owns the transaction boundary (this is the whole point)::
+
+        with db.transaction():             # the caller's own open transaction
+            db.insert_order(order)         # the business write
+            envelope = EnvelopeCodec.make("urn:babel:orders:created", {"order_id": 1042})
+            outbox.write(envelope)         # same connection, same tx — both, or neither
+
+    Because both writes share one transaction, a crash can never leave the business row
+    committed without its message (the classic dual-write bug) — they commit or roll back
+    together. The handoff to the broker becomes a *local* problem the relay solves.
+
+    This helper only encodes via the frozen :class:`~babelqueue.codec.EnvelopeCodec` (GR-1 —
+    the envelope bytes are stored unchanged; the outbox never adds an envelope field) and
+    delegates persistence to the injected :class:`OutboxStore`, which the caller binds to
+    their own DB (GR-7). It does **not** begin/commit anything.
+    """
+
+    def __init__(self, store: OutboxStore) -> None:
+        self._store = store
+
+    def write(self, envelope: Mapping[str, Any]) -> str:
+        """Encode the envelope (frozen codec, bytes unchanged) and persist it via the store,
+        inside the transaction the caller has already opened. Return the new outbox row id
+        (for the caller's own correlation, if wanted).
+
+        ``envelope`` is a canonical envelope from :meth:`~babelqueue.codec.EnvelopeCodec.make`
+        / :meth:`~babelqueue.codec.EnvelopeCodec.from_message`.
+        """
+        queue = self._queue_of(envelope)
+        return self._store.save(EnvelopeCodec.encode(envelope), queue)
+
+    @staticmethod
+    def _queue_of(envelope: Mapping[str, Any]) -> str:
+        """The logical queue the message targets: its ``meta.queue``, falling back to
+        ``"default"``. Captured at write time so the relay can publish to the right queue
+        without decoding the body."""
+        meta = envelope.get("meta")
+        if isinstance(meta, Mapping):
+            queue = meta.get("queue")
+            if isinstance(queue, str) and queue != "":
+                return queue
+        return "default"
+
+
+class OutboxRelay:
+    """The **read/publish side** of the transactional outbox (ADR-0029): drain pending rows the
+    :class:`Outbox` writer committed and forward each onto the broker through the frozen
+    :class:`~babelqueue.transport.Transport` contract, marking every row published or failed.
+
+    Run it on a short interval (a worker loop, a scheduled command) *after* the business
+    transaction commits. Because the message was committed atomically with the business data,
+    the relay is the only thing standing between "row exists" and "broker has it" — and it only
+    ever reads already-durable rows, so it never invents work.
+
+    **Semantics — at-least-once handoff:**
+
+    - A row is marked **published only after** :meth:`~babelqueue.transport.Transport.publish`
+      returns; if the process dies between publish and :meth:`OutboxStore.mark_published`, the
+      row stays pending and is published **again** on the next pass. That is at-least-once: a
+      downstream consumer must dedupe on the canonical ``meta.id`` (:func:`babelqueue.idempotency.wrap`
+      is exactly that guard, the consumer-side mirror — ADR-0022).
+    - A publish that **raises** is caught, :meth:`OutboxStore.mark_failed` records the error and
+      bumps the attempt count, and the row stays pending for a later retry. One poison row never
+      blocks the rest of the batch.
+    - **``trace_id`` is preserved end-to-end** (GR-4): the relay publishes the stored bytes
+      *verbatim* — it never decodes, rebuilds or re-encodes the envelope — so the body that
+      reaches the broker is byte-identical to what was stored (GR-1/GR-5).
+
+    **Backoff:** between a failed publish and the next attempt within the same pass the relay
+    sleeps for a bounded, linearly-growing delay (capped), to avoid hammering a broker that is
+    briefly down. The sleeper is injectable so tests stay instant.
+    """
+
+    #: Hard safety ceiling on :meth:`drain` passes when the caller passes ``0``.
+    DEFAULT_DRAIN_CEILING = 10000
+
+    def __init__(
+        self,
+        transport: Transport,
+        store: OutboxStore,
+        *,
+        batch_size: int = 100,
+        backoff_step: float = 0.05,
+        backoff_cap: float = 5.0,
+        sleeper: Sleeper = time.sleep,
+    ) -> None:
+        """``transport`` is where published rows go (the same publish-only seam every producer
+        uses); ``store`` is the outbox to drain. ``batch_size`` is how many rows to reserve and
+        publish per :meth:`flush`. ``backoff_step`` (seconds) is the base delay added per prior
+        attempt and ``backoff_cap`` (seconds) is the upper bound on a single backoff sleep.
+        ``sleeper`` sleeps the given number of seconds (default :func:`time.sleep`); inject a
+        no-op or a recorder in tests."""
+        self._transport = transport
+        self._store = store
+        self._batch_size = batch_size
+        self._backoff_step = backoff_step
+        self._backoff_cap = backoff_cap
+        self._sleeper = sleeper
+
+    def flush(self) -> OutboxRelayResult:
+        """Publish one batch of pending rows. Each row the transport accepts is marked
+        published; each that raises is marked failed (with a backoff before continuing) and left
+        pending. Return a per-pass tally. Call it repeatedly (a loop / cron) to drain the outbox;
+        :meth:`drain` loops until it is empty."""
+        records = self._store.fetch_unpublished(self._batch_size)
+
+        published_ids: List[str] = []
+        failed = 0
+
+        for record in records:
+            try:
+                # Publish the stored bytes verbatim — never decode/rebuild/re-encode (GR-1).
+                self._transport.publish(record.queue, record.body)
+                published_ids.append(record.id)
+            except Exception as exc:  # noqa: BLE001 - one poison row must not abort the batch
+                self._store.mark_failed(record.id, self._reason(exc))
+                failed += 1
+                self._sleep(self._backoff_for(record.attempts))
+
+        if published_ids:
+            self._store.mark_published(published_ids)
+
+        return OutboxRelayResult(len(published_ids), failed)
+
+    def drain(self, max_passes: int = 0) -> OutboxRelayResult:
+        """Drain the outbox by repeatedly calling :meth:`flush` while each pass keeps making
+        progress (publishes at least one row), then return the cumulative tally. The loop stops
+        as soon as a pass publishes nothing — the outbox is empty, or only currently failing rows
+        remain (those are left pending for a future :meth:`drain` call once the broker recovers).
+        ``max_passes`` is a hard safety ceiling so a degenerate store can never spin forever
+        (``0`` = a generous internal default)."""
+        ceiling = max_passes if max_passes > 0 else self.DEFAULT_DRAIN_CEILING
+        published = 0
+        failed = 0
+
+        for _pass in range(ceiling):
+            result = self.flush()
+            published += result.published
+            failed += result.failed
+
+            # No progress this pass → drained, or only failing rows remain. Stop.
+            if result.published == 0:
+                break
+
+        return OutboxRelayResult(published, failed)
+
+    def _backoff_for(self, prior_attempts: int) -> float:
+        """The backoff (seconds) for a row that has already failed ``prior_attempts`` times: a
+        linear step per attempt, capped. Kept simple and deterministic so the budget is obvious."""
+        delay = self._backoff_step * max(1, prior_attempts + 1)
+        return min(delay, self._backoff_cap)
+
+    def _sleep(self, seconds: float) -> None:
+        if seconds > 0:
+            self._sleeper(seconds)
+
+    @staticmethod
+    def _reason(exc: BaseException) -> str:
+        """A short, safe failure reason from a raised error (type + message, no traceback)."""
+        return f"{type(exc).__name__}: {exc}"
+
+
+class InMemoryOutboxStore:
+    """Process-local reference :class:`OutboxStore` backed by a dict — for tests and
+    single-process demos. It has **no real transaction**: :meth:`save` just appends, so it
+    cannot deliver the atomic-with-the-business-write guarantee a production store gives. Use a
+    database-backed adapter in production.
+
+    It still faithfully models the relay contract: rows are pending until :meth:`mark_published`,
+    :meth:`fetch_unpublished` returns them oldest-first, and :meth:`mark_failed` bumps the attempt
+    count and stores the last error while leaving the row pending for retry.
+    """
+
+    def __init__(self) -> None:
+        # Insertion order is preserved by dict, so iteration is naturally oldest-first.
+        self._rows: Dict[str, Dict[str, Any]] = {}
+        self._sequence = 0
+
+    def save(self, encoded: str, queue: str) -> str:
+        self._sequence += 1
+        # A non-numeric id keeps the key a genuine string and mirrors the PHP reference.
+        row_id = f"ob-{self._sequence}"
+        self._rows[row_id] = {
+            "body": encoded,
+            "queue": queue,
+            "attempts": 0,
+            "published": False,
+            "error": "",
+        }
+        return row_id
+
+    def fetch_unpublished(self, limit: int) -> List[OutboxRecord]:
+        records: List[OutboxRecord] = []
+        for row_id, row in self._rows.items():
+            if row["published"]:
+                continue
+            records.append(OutboxRecord(row_id, row["body"], row["queue"], row["attempts"]))
+            if len(records) >= limit:
+                break
+        return records
+
+    def mark_published(self, ids: List[str]) -> None:
+        for row_id in ids:
+            row = self._rows.get(row_id)
+            if row is not None:
+                row["published"] = True
+
+    def mark_failed(self, id: str, error: str) -> None:
+        row = self._rows.get(id)
+        if row is not None:
+            row["attempts"] += 1
+            row["error"] = error
+
+    def pending_count(self) -> int:
+        """Test/inspection helper: the number of rows still pending publish."""
+        return sum(1 for row in self._rows.values() if not row["published"])
+
+    def attempts_of(self, id: str) -> int:
+        """Test/inspection helper: the recorded attempt count for one row (0 if unknown)."""
+        row = self._rows.get(id)
+        return row["attempts"] if row is not None else 0
+
+    def last_error_of(self, id: str) -> str:
+        """Test/inspection helper: the last recorded error for one row ('' if none)."""
+        row = self._rows.get(id)
+        return row["error"] if row is not None else ""

babelqueue-1.12.0/tests/test_outbox.py

@@ -0,0 +1,263 @@
+from __future__ import annotations
+
+import unittest
+from typing import List, Tuple
+
+from babelqueue.codec import EnvelopeCodec
+from babelqueue.outbox import (
+    InMemoryOutboxStore,
+    Outbox,
+    OutboxRecord,
+    OutboxRelay,
+    OutboxRelayResult,
+)
+from babelqueue.transport import Transport
+
+
+class FakeTransport(Transport):
+    """A publish-only fake that records what the relay forwarded, optionally raising for a
+    configured set of bodies so a "poison row" can be simulated without a broker."""
+
+    def __init__(self, fail_bodies: Tuple[str, ...] = ()) -> None:
+        self.published: List[Tuple[str, str]] = []  # (queue, body) in publish order
+        self._fail_bodies = set(fail_bodies)
+
+    def publish(self, queue: str, body: str) -> None:
+        if body in self._fail_bodies:
+            raise RuntimeError("broker down")
+        self.published.append((queue, body))
+
+    def pop(self, queue: str, timeout: float = 1.0):  # pragma: no cover - unused by the relay
+        return None
+
+    def ack(self, message) -> None:  # pragma: no cover - unused by the relay
+        return None
+
+
+class RecordingSleeper:
+    """An injected sleeper that records each requested delay instead of sleeping, so backoff
+    growth/capping is asserted without real time passing."""
+
+    def __init__(self) -> None:
+        self.delays: List[float] = []
+
+    def __call__(self, seconds: float) -> None:
+        self.delays.append(seconds)
+
+
+def _envelope(order_id: int, trace_id: str = "trace-abc") -> dict:
+    return EnvelopeCodec.make(
+        "urn:babel:orders:created", {"order_id": order_id}, trace_id=trace_id
+    )
+
+
+class OutboxWriteTest(unittest.TestCase):
+    def test_write_stores_encoded_envelope_verbatim(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        envelope = _envelope(1042)
+
+        row_id = outbox.write(envelope)
+
+        [record] = store.fetch_unpublished(10)
+        self.assertEqual(record.id, row_id)
+        # The stored bytes are exactly the codec output — never decoded/rebuilt/re-encoded.
+        self.assertEqual(record.body, EnvelopeCodec.encode(envelope))
+
+    def test_write_captures_meta_queue(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        envelope = EnvelopeCodec.make("urn:babel:orders:created", {}, queue="orders")
+
+        outbox.write(envelope)
+
+        [record] = store.fetch_unpublished(10)
+        self.assertEqual(record.queue, "orders")
+
+    def test_write_falls_back_to_default_queue(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        # An envelope with no usable meta.queue → "default".
+        outbox.write({"job": "urn:x", "trace_id": "t", "data": {}, "meta": {}, "attempts": 0})
+
+        [record] = store.fetch_unpublished(10)
+        self.assertEqual(record.queue, "default")
+
+
+class OutboxRelayTest(unittest.TestCase):
+    def test_flush_publishes_and_marks_published(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        transport = FakeTransport()
+        outbox.write(EnvelopeCodec.make("urn:babel:orders:created", {}, queue="orders"))
+
+        result = OutboxRelay(transport, store).flush()
+
+        self.assertEqual(result, OutboxRelayResult(published=1, failed=0))
+        self.assertEqual(result.attempted, 1)
+        self.assertEqual(len(transport.published), 1)
+        self.assertEqual(transport.published[0][0], "orders")  # (queue, body) order
+        self.assertEqual(store.pending_count(), 0)  # marked published → no longer pending
+
+    def test_relay_publishes_bytes_verbatim_preserving_trace_id(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        transport = FakeTransport()
+        envelope = _envelope(7, trace_id="keep-me-1234")
+        encoded = EnvelopeCodec.encode(envelope)
+        outbox.write(envelope)
+
+        OutboxRelay(transport, store).flush()
+
+        # The body that reached the transport is byte-identical to what was stored (GR-1/GR-5)…
+        _queue, published_body = transport.published[0]
+        self.assertEqual(published_body, encoded)
+        # …and trace_id survives end-to-end (GR-4) without the relay ever decoding.
+        self.assertEqual(EnvelopeCodec.decode(published_body)["trace_id"], "keep-me-1234")
+
+    def test_failed_publish_marks_failed_leaves_pending_and_continues_batch(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        # Build each envelope once (make() mints a fresh meta.id/created_at per call), so the
+        # bytes the fake transport fails on are exactly the bytes that were stored.
+        e1, e2, e3 = _envelope(1), _envelope(2), _envelope(3)
+        good1 = EnvelopeCodec.encode(e1)
+        poison = EnvelopeCodec.encode(e2)
+        good2 = EnvelopeCodec.encode(e3)
+        # Write three rows; the middle one is the poison row that always fails to publish.
+        id1 = outbox.write(e1)
+        poison_id = outbox.write(e2)
+        id3 = outbox.write(e3)
+
+        transport = FakeTransport(fail_bodies=(poison,))
+        sleeper = RecordingSleeper()
+        result = OutboxRelay(transport, store, sleeper=sleeper).flush()
+
+        # The two good rows published; the poison row failed but did not abort the batch.
+        self.assertEqual(result, OutboxRelayResult(published=2, failed=1))
+        self.assertEqual([b for _q, b in transport.published], [good1, good2])
+        # Poison row stays pending for a later retry, with its attempt counted + error stored.
+        self.assertEqual(store.pending_count(), 1)
+        self.assertEqual(store.attempts_of(poison_id), 1)
+        self.assertIn("RuntimeError: broker down", store.last_error_of(poison_id))
+        # The good rows are gone (published), the poison id remains.
+        remaining = [r.id for r in store.fetch_unpublished(10)]
+        self.assertEqual(remaining, [poison_id])
+        self.assertNotIn(id1, remaining)
+        self.assertNotIn(id3, remaining)
+        # A backoff was slept once for the single failure.
+        self.assertEqual(len(sleeper.delays), 1)
+
+    def test_drain_loops_until_empty(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        # More rows than one batch → drain must loop across multiple flush passes.
+        for i in range(5):
+            outbox.write(_envelope(i))
+        transport = FakeTransport()
+
+        result = OutboxRelay(transport, store, batch_size=2).drain()
+
+        self.assertEqual(result.published, 5)
+        self.assertEqual(result.failed, 0)
+        self.assertEqual(store.pending_count(), 0)
+        self.assertEqual(len(transport.published), 5)
+
+    def test_drain_stops_when_only_failing_rows_remain(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        e = _envelope(99)
+        poison = EnvelopeCodec.encode(e)
+        outbox.write(e)
+        # A transport that always fails → no progress → drain must not spin forever.
+        transport = FakeTransport(fail_bodies=(poison,))
+        sleeper = RecordingSleeper()
+
+        result = OutboxRelay(transport, store, sleeper=sleeper).drain()
+
+        self.assertEqual(result.published, 0)
+        self.assertEqual(result.failed, 1)  # exactly one pass: it published nothing, so it stopped
+        self.assertEqual(store.pending_count(), 1)
+
+    def test_backoff_grows_with_attempts_and_caps(self) -> None:
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        e = _envelope(1)
+        poison = EnvelopeCodec.encode(e)
+        poison_id = outbox.write(e)
+        transport = FakeTransport(fail_bodies=(poison,))
+        sleeper = RecordingSleeper()
+        relay = OutboxRelay(
+            transport, store, backoff_step=0.05, backoff_cap=0.2, sleeper=sleeper
+        )
+
+        # Each flush retries the same poison row, whose attempts climb 0,1,2,3,4…
+        # backoff_for(attempts) = 0.05 * (attempts + 1), capped at 0.2:
+        #   attempts=0 → 0.05, 1 → 0.10, 2 → 0.15, 3 → 0.20, 4 → 0.20 (capped).
+        for _ in range(5):
+            relay.flush()
+
+        self.assertEqual(store.attempts_of(poison_id), 5)
+        self.assertEqual(
+            [round(d, 2) for d in sleeper.delays],
+            [0.05, 0.10, 0.15, 0.20, 0.20],
+        )
+        # Growth then a hard cap.
+        self.assertEqual(max(sleeper.delays), 0.20)
+
+    def test_default_sleeper_is_skipped_for_zero_delay(self) -> None:
+        # A non-positive backoff never calls the sleeper (e.g. backoff_step=0).
+        store = InMemoryOutboxStore()
+        outbox = Outbox(store)
+        e = _envelope(1)
+        poison = EnvelopeCodec.encode(e)
+        outbox.write(e)
+        transport = FakeTransport(fail_bodies=(poison,))
+        sleeper = RecordingSleeper()
+
+        OutboxRelay(transport, store, backoff_step=0.0, sleeper=sleeper).flush()
+
+        self.assertEqual(sleeper.delays, [])
+
+
+class InMemoryOutboxStoreTest(unittest.TestCase):
+    def test_fetch_unpublished_is_oldest_first_and_limited(self) -> None:
+        store = InMemoryOutboxStore()
+        ids = [store.save(f"body-{i}", "q") for i in range(3)]
+
+        records = store.fetch_unpublished(2)
+
+        self.assertEqual([r.id for r in records], ids[:2])  # oldest-first, capped at limit
+        self.assertTrue(all(isinstance(r, OutboxRecord) for r in records))
+
+    def test_mark_published_removes_from_pending(self) -> None:
+        store = InMemoryOutboxStore()
+        a = store.save("a", "q")
+        b = store.save("b", "q")
+
+        store.mark_published([a])
+
+        self.assertEqual(store.pending_count(), 1)
+        self.assertEqual([r.id for r in store.fetch_unpublished(10)], [b])
+
+    def test_mark_failed_bumps_attempts_and_records_error(self) -> None:
+        store = InMemoryOutboxStore()
+        a = store.save("a", "q")
+
+        store.mark_failed(a, "boom")
+        store.mark_failed(a, "boom-again")
+
+        self.assertEqual(store.attempts_of(a), 2)
+        self.assertEqual(store.last_error_of(a), "boom-again")
+        self.assertEqual(store.pending_count(), 1)  # still pending after failures
+
+    def test_unknown_ids_are_ignored(self) -> None:
+        store = InMemoryOutboxStore()
+        store.mark_published(["nope"])  # no row → no error
+        store.mark_failed("nope", "x")  # no row → no error
+        self.assertEqual(store.attempts_of("nope"), 0)
+        self.assertEqual(store.last_error_of("nope"), "")
+
+
+if __name__ == "__main__":
+    unittest.main()