faststream-outbox 0.1.0__tar.gz

faststream_outbox-0.1.0/PKG-INFO

@@ -0,0 +1,132 @@
+Metadata-Version: 2.4
+Name: faststream-outbox
+Version: 0.1.0
+Summary: FastStream broker integration for the transactional outbox pattern: a Postgres table is the queue
+Author: Artur Shiriev
+Author-email: Artur Shiriev <me@shiriev.ru>
+License-Expression: MIT
+Requires-Dist: faststream~=0.6
+Requires-Dist: sqlalchemy[asyncio]>=2.0
+Requires-Dist: asyncpg>=0.29 ; extra == 'asyncpg'
+Requires-Python: >=3.13, <4
+Provides-Extra: asyncpg
+Description-Content-Type: text/markdown
+
+faststream-outbox
+==
+
+[![Supported versions](https://img.shields.io/pypi/pyversions/faststream-outbox.svg)](https://pypi.python.org/pypi/faststream-outbox)
+[![downloads](https://img.shields.io/pypi/dm/faststream-outbox.svg)](https://pypistats.org/packages/faststream-outbox)
+
+`faststream-outbox` is a [FastStream](https://faststream.airt.ai) broker integration for the **transactional outbox pattern** — a Postgres table is the message queue.
+
+A producer writes a domain entity and an outbox row in the *same* SQLAlchemy transaction by calling `broker.publish(body, queue=..., session=session)`. A subscriber polls the table directly with `FOR UPDATE SKIP LOCKED`, runs the handler, and deletes the row on success. No downstream broker, no separate relay process — the table *is* the queue.
+
+```python
+from sqlalchemy import MetaData
+from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
+from faststream import FastStream
+from faststream_outbox import OutboxBroker, make_outbox_table
+
+metadata = MetaData()
+outbox_table = make_outbox_table(metadata, table_name="outbox")
+
+engine = create_async_engine("postgresql+asyncpg://localhost/app")
+broker = OutboxBroker(engine, outbox_table=outbox_table)
+app = FastStream(broker)
+
+@broker.subscriber("orders", max_workers=4)
+async def handle(order_id: int) -> None:
+    print(f"order {order_id}")
+
+# Producer side — share the caller's open transaction:
+session_factory = async_sessionmaker(engine, expire_on_commit=False)
+async with session_factory() as session, session.begin():
+    session.add(Order(id=1))
+    await broker.publish(1, queue="orders", session=session)
+```
+
+## How it works
+
+`make_outbox_table(metadata, table_name="outbox")` returns a `sqlalchemy.Table` that you attach to your own `MetaData` and migrate via Alembic. The package does **not** own your schema; it only describes the columns it needs.
+
+`broker.publish(body, *, queue, session, headers=None, correlation_id=None)` inserts one outbox row through the caller's `AsyncSession`. It does not flush, commit, or open its own transaction — the whole point is that the row commits atomically with the caller's domain writes. Use it inside an `async with session.begin():` block.
+
+`broker.publish_batch(*bodies, queue, session, headers=None)` inserts many rows in a single round-trip with the same transactional contract.
+
+A subscriber owns three async loops:
+
+1. **fetch** — claims due rows via `SELECT … FOR UPDATE SKIP LOCKED → UPDATE state='processing', acquired_token=:uuid RETURNING *` in a single CTE.
+2. **workers** (× `max_workers`) — dispatch to the handler. On success, `DELETE WHERE id=:id AND acquired_token=:token`. On failure, the retry strategy decides: schedule another attempt, or terminal `DELETE`.
+3. **release-stuck** — periodically flips `processing` rows back to `pending` if their lease is older than `release_stuck_timeout`. Wrapped in a Postgres advisory lock so multiple processes don't compete.
+
+The `acquired_token` is critical: a slow handler whose lease expired and was re-claimed by another worker will find its terminal `DELETE`/`UPDATE` to be a no-op (the token no longer matches), preventing it from clobbering the new lease holder's row.
+
+## Recommended index
+
+Add this to your Alembic migration alongside the table:
+
+```sql
+CREATE INDEX outbox_pending_idx ON outbox (queue, next_attempt_at)
+  WHERE state = 'pending';
+```
+
+## Schema validation
+
+Schema validation is opt-in:
+
+```python
+await broker.validate_schema()  # raises if user's table drifts from expected columns
+```
+
+Call it from a `/health` endpoint or startup hook — not at `broker.start()`, so Alembic can run migrations against the same DB without a startup loop.
+
+## Retry strategies
+
+```python
+from faststream_outbox import ExponentialRetry, ConstantRetry, LinearRetry, NoRetry
+
+@broker.subscriber(
+    "orders",
+    retry_strategy=ExponentialRetry(
+        initial_delay_seconds=1.0,
+        max_delay_seconds=300.0,
+        max_attempts=5,
+        jitter_factor=0.5,
+    ),
+)
+async def handle(order_id: int) -> None: ...
+```
+
+Strategies receive the raised `exception` so users may subclass for "retry only on transient errors":
+
+```python
+class TransientOnly(ExponentialRetry):
+    def get_next_attempt_at(self, *, exception=None, **kw):
+        if exception and not isinstance(exception, TransientError):
+            return None
+        return super().get_next_attempt_at(exception=exception, **kw)
+```
+
+## Failure modes
+
+- **Handlers must be idempotent.** Crash between commit-of-handler-side-effects and the broker's `DELETE` re-delivers the message.
+- **Best-effort ordering only.** `FOR UPDATE SKIP LOCKED` does not preserve strict order under concurrent workers. If you need strict per-aggregate ordering, route to a single subscriber and run a single worker.
+- **No DLQ / archive.** Terminal failures `DELETE` the row. Hook `on_terminal_failure(row)` to capture them in your own table or alerting.
+
+## Connection ownership
+
+`OutboxBroker` does **not** close the `AsyncEngine` you pass in — the caller owns its lifecycle.
+
+## Tuning
+
+Per-subscriber knobs (passed to `@broker.subscriber("…", …)`):
+
+- `max_workers` (default `1`) — concurrent handlers per subscriber.
+- `fetch_batch_size` (default `10`) — rows claimed per fetch cycle.
+- `min_fetch_interval` / `max_fetch_interval` (default `1.0` / `10.0` s) — base + ceiling for the adaptive idle backoff with jitter.
+- `release_stuck_timeout` (default `300.0` s) — how long a `processing` row may live before being released back to `pending`.
+- `release_stuck_interval` (default `release_stuck_timeout / 2`).
+- `max_deliveries` (default `None` — unbounded) — total claims (including stuck-recovery re-claims) after which the row is dropped without invoking the handler. Defends against handlers that consistently wedge.
+
+## 📝 [License](LICENSE)

faststream_outbox-0.1.0/README.md

@@ -0,0 +1,118 @@
+faststream-outbox
+==
+
+[![Supported versions](https://img.shields.io/pypi/pyversions/faststream-outbox.svg)](https://pypi.python.org/pypi/faststream-outbox)
+[![downloads](https://img.shields.io/pypi/dm/faststream-outbox.svg)](https://pypistats.org/packages/faststream-outbox)
+
+`faststream-outbox` is a [FastStream](https://faststream.airt.ai) broker integration for the **transactional outbox pattern** — a Postgres table is the message queue.
+
+A producer writes a domain entity and an outbox row in the *same* SQLAlchemy transaction by calling `broker.publish(body, queue=..., session=session)`. A subscriber polls the table directly with `FOR UPDATE SKIP LOCKED`, runs the handler, and deletes the row on success. No downstream broker, no separate relay process — the table *is* the queue.
+
+```python
+from sqlalchemy import MetaData
+from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
+from faststream import FastStream
+from faststream_outbox import OutboxBroker, make_outbox_table
+
+metadata = MetaData()
+outbox_table = make_outbox_table(metadata, table_name="outbox")
+
+engine = create_async_engine("postgresql+asyncpg://localhost/app")
+broker = OutboxBroker(engine, outbox_table=outbox_table)
+app = FastStream(broker)
+
+@broker.subscriber("orders", max_workers=4)
+async def handle(order_id: int) -> None:
+    print(f"order {order_id}")
+
+# Producer side — share the caller's open transaction:
+session_factory = async_sessionmaker(engine, expire_on_commit=False)
+async with session_factory() as session, session.begin():
+    session.add(Order(id=1))
+    await broker.publish(1, queue="orders", session=session)
+```
+
+## How it works
+
+`make_outbox_table(metadata, table_name="outbox")` returns a `sqlalchemy.Table` that you attach to your own `MetaData` and migrate via Alembic. The package does **not** own your schema; it only describes the columns it needs.
+
+`broker.publish(body, *, queue, session, headers=None, correlation_id=None)` inserts one outbox row through the caller's `AsyncSession`. It does not flush, commit, or open its own transaction — the whole point is that the row commits atomically with the caller's domain writes. Use it inside an `async with session.begin():` block.
+
+`broker.publish_batch(*bodies, queue, session, headers=None)` inserts many rows in a single round-trip with the same transactional contract.
+
+A subscriber owns three async loops:
+
+1. **fetch** — claims due rows via `SELECT … FOR UPDATE SKIP LOCKED → UPDATE state='processing', acquired_token=:uuid RETURNING *` in a single CTE.
+2. **workers** (× `max_workers`) — dispatch to the handler. On success, `DELETE WHERE id=:id AND acquired_token=:token`. On failure, the retry strategy decides: schedule another attempt, or terminal `DELETE`.
+3. **release-stuck** — periodically flips `processing` rows back to `pending` if their lease is older than `release_stuck_timeout`. Wrapped in a Postgres advisory lock so multiple processes don't compete.
+
+The `acquired_token` is critical: a slow handler whose lease expired and was re-claimed by another worker will find its terminal `DELETE`/`UPDATE` to be a no-op (the token no longer matches), preventing it from clobbering the new lease holder's row.
+
+## Recommended index
+
+Add this to your Alembic migration alongside the table:
+
+```sql
+CREATE INDEX outbox_pending_idx ON outbox (queue, next_attempt_at)
+  WHERE state = 'pending';
+```
+
+## Schema validation
+
+Schema validation is opt-in:
+
+```python
+await broker.validate_schema()  # raises if user's table drifts from expected columns
+```
+
+Call it from a `/health` endpoint or startup hook — not at `broker.start()`, so Alembic can run migrations against the same DB without a startup loop.
+
+## Retry strategies
+
+```python
+from faststream_outbox import ExponentialRetry, ConstantRetry, LinearRetry, NoRetry
+
+@broker.subscriber(
+    "orders",
+    retry_strategy=ExponentialRetry(
+        initial_delay_seconds=1.0,
+        max_delay_seconds=300.0,
+        max_attempts=5,
+        jitter_factor=0.5,
+    ),
+)
+async def handle(order_id: int) -> None: ...
+```
+
+Strategies receive the raised `exception` so users may subclass for "retry only on transient errors":
+
+```python
+class TransientOnly(ExponentialRetry):
+    def get_next_attempt_at(self, *, exception=None, **kw):
+        if exception and not isinstance(exception, TransientError):
+            return None
+        return super().get_next_attempt_at(exception=exception, **kw)
+```
+
+## Failure modes
+
+- **Handlers must be idempotent.** Crash between commit-of-handler-side-effects and the broker's `DELETE` re-delivers the message.
+- **Best-effort ordering only.** `FOR UPDATE SKIP LOCKED` does not preserve strict order under concurrent workers. If you need strict per-aggregate ordering, route to a single subscriber and run a single worker.
+- **No DLQ / archive.** Terminal failures `DELETE` the row. Hook `on_terminal_failure(row)` to capture them in your own table or alerting.
+
+## Connection ownership
+
+`OutboxBroker` does **not** close the `AsyncEngine` you pass in — the caller owns its lifecycle.
+
+## Tuning
+
+Per-subscriber knobs (passed to `@broker.subscriber("…", …)`):
+
+- `max_workers` (default `1`) — concurrent handlers per subscriber.
+- `fetch_batch_size` (default `10`) — rows claimed per fetch cycle.
+- `min_fetch_interval` / `max_fetch_interval` (default `1.0` / `10.0` s) — base + ceiling for the adaptive idle backoff with jitter.
+- `release_stuck_timeout` (default `300.0` s) — how long a `processing` row may live before being released back to `pending`.
+- `release_stuck_interval` (default `release_stuck_timeout / 2`).
+- `max_deliveries` (default `None` — unbounded) — total claims (including stuck-recovery re-claims) after which the row is dropped without invoking the handler. Defends against handlers that consistently wedge.
+
+## 📝 [License](LICENSE)

faststream_outbox-0.1.0/faststream_outbox/__init__.py

@@ -0,0 +1,25 @@
+from faststream_outbox.broker import OutboxBroker
+from faststream_outbox.retry import (
+    ConstantRetry,
+    ExponentialRetry,
+    LinearRetry,
+    NoRetry,
+    RetryStrategyProto,
+)
+from faststream_outbox.router import OutboxRouter
+from faststream_outbox.schema import OutboxState, make_outbox_table
+from faststream_outbox.testing import TestOutboxBroker
+
+
+__all__ = [
+    "ConstantRetry",
+    "ExponentialRetry",
+    "LinearRetry",
+    "NoRetry",
+    "OutboxBroker",
+    "OutboxRouter",
+    "OutboxState",
+    "RetryStrategyProto",
+    "TestOutboxBroker",
+    "make_outbox_table",
+]

faststream_outbox-0.1.0/faststream_outbox/broker.py

@@ -0,0 +1,237 @@
+"""
+OutboxBroker — a FastStream broker whose queue is a Postgres table.
+
+Producers call ``broker.publish(body, queue=..., session=session)`` inside their
+own SQLAlchemy transaction; the row commits with their domain writes. The broker
+owns subscribers on the consumer side.
+"""
+
+import logging
+import typing
+from collections.abc import Iterable, Sequence
+
+from faststream import BaseMiddleware
+from faststream._internal.basic_types import LoggerProto
+from faststream._internal.broker import BrokerUsecase
+from faststream._internal.broker.registrator import Registrator
+from faststream._internal.constants import EMPTY
+from faststream._internal.di import FastDependsConfig
+from faststream._internal.logger import DefaultLoggerStorage, make_logger_state
+from faststream._internal.logger.logging import get_broker_logger
+from faststream._internal.types import BrokerMiddleware, CustomCallable
+from faststream.specification.schema import BrokerSpec
+from faststream.specification.schema.extra import Tag, TagDict
+from sqlalchemy import insert
+from sqlalchemy.ext.asyncio import AsyncSession
+
+from faststream_outbox.client import OutboxClient
+from faststream_outbox.configs import EngineState, OutboxBrokerConfig
+from faststream_outbox.envelope import _encode_payload
+from faststream_outbox.message import OutboxInnerMessage
+from faststream_outbox.registrator import OutboxRegistrator
+
+
+if typing.TYPE_CHECKING:
+    from fast_depends.dependencies import Dependant
+    from faststream._internal.context.repository import ContextRepo
+    from sqlalchemy import Table
+    from sqlalchemy.ext.asyncio import AsyncEngine
+
+    from faststream_outbox.subscriber.usecase import OutboxSubscriber
+
+
+class OutboxParamsStorage(DefaultLoggerStorage):
+    _max_msg_id_ln = -1
+    _max_queue_name = 7
+
+    def get_logger(self, *, context: "ContextRepo") -> LoggerProto:
+        if logger := self._get_logger_ref():
+            return logger
+        logger = get_broker_logger(
+            name="outbox",
+            default_context={"queue": "", "message_id": ""},
+            message_id_ln=self._max_msg_id_ln,
+            fmt=(
+                "%(asctime)s %(levelname)-8s - "
+                f"%(queue)-{self._max_queue_name}s | "
+                f"%(message_id)-{self._max_msg_id_ln}s "
+                "- %(message)s"
+            ),
+            context=context,
+            log_level=self.logger_log_level,
+        )
+        self._logger_ref.add(logger)
+        return logger
+
+
+class OutboxBroker(
+    OutboxRegistrator,
+    BrokerUsecase[OutboxInnerMessage, "AsyncEngine", OutboxBrokerConfig],
+):
+    """FastStream broker backed by a Postgres outbox table."""
+
+    _subscribers: list["OutboxSubscriber"]
+
+    def __init__(  # noqa: PLR0913
+        self,
+        engine: "AsyncEngine | None" = None,
+        *,
+        outbox_table: "Table",
+        decoder: CustomCallable | None = None,
+        parser: CustomCallable | None = None,
+        dependencies: Iterable["Dependant"] = (),
+        middlewares: Sequence[type[BaseMiddleware] | BrokerMiddleware[OutboxInnerMessage]] = (),
+        graceful_timeout: float | None = 15.0,
+        routers: Sequence[Registrator[OutboxInnerMessage]] = (),
+        # Logging
+        logger: LoggerProto | None = EMPTY,
+        log_level: int = logging.INFO,
+        # FastDepends
+        apply_types: bool = True,
+        # AsyncAPI
+        description: str | None = None,
+        tags: Iterable[Tag | TagDict] = (),
+    ) -> None:
+        self._outbox_table = outbox_table
+        engine_state = EngineState(engine)
+        client = OutboxClient(engine, outbox_table) if engine is not None else None
+        fd_config = FastDependsConfig(use_fastdepends=apply_types)
+        broker_config = OutboxBrokerConfig(
+            engine_state=engine_state,
+            outbox_table=outbox_table,
+            client=client,
+            broker_middlewares=middlewares,
+            broker_parser=parser,
+            broker_decoder=decoder,
+            logger=make_logger_state(
+                logger=logger,
+                log_level=log_level,
+                default_storage_cls=OutboxParamsStorage,
+            ),
+            fd_config=fd_config,
+            broker_dependencies=dependencies,
+            graceful_timeout=graceful_timeout,
+            extra_context={"broker": self},
+            producer=_NoProducer(),  # ty: ignore[invalid-argument-type]
+        )
+        specification = BrokerSpec(
+            url=[],
+            protocol="postgresql",
+            protocol_version=None,
+            description=description,
+            tags=tags,
+            security=None,
+        )
+        super().__init__(config=broker_config, specification=specification, routers=routers)  # ty: ignore[unknown-argument]
+
+    @property
+    def client(self) -> OutboxClient:
+        client = self.config.broker_config.client
+        if client is None:
+            msg = "OutboxBroker is not connected; pass an AsyncEngine to the constructor."
+            raise RuntimeError(msg)
+        return client
+
+    @typing.override
+    async def _connect(self) -> "AsyncEngine":
+        return self.config.broker_config.engine_state.engine
+
+    @typing.override
+    async def __aenter__(self) -> typing.Self:
+        await self.start()
+        return self
+
+    @typing.override
+    async def start(self) -> None:
+        await self.connect()
+        await super().start()
+
+    @typing.override
+    async def ping(self, timeout: float | None = None) -> bool:
+        client = self.config.broker_config.client
+        if client is None:
+            return False
+        if not await client.ping():
+            return False
+        for subscriber in self._subscribers:
+            for task in subscriber.tasks:
+                if task.done():
+                    return False
+        return True
+
+    async def validate_schema(self) -> None:
+        """Validate the user's table matches what the package expects. Opt-in."""
+        await self.client.validate_schema()
+
+    async def publish(  # ty: ignore[invalid-method-override]
+        self,
+        body: typing.Any,
+        *,
+        queue: str,
+        session: AsyncSession,
+        headers: dict[str, str] | None = None,
+        correlation_id: str | None = None,
+    ) -> None:
+        """
+        Insert one outbox row using *session*'s open transaction.
+
+        Must be called inside a transaction the caller owns (typically inside an
+        ``async with session.begin():`` block). ``publish`` does not flush, commit,
+        or open its own transaction — that is the whole point of the transactional
+        outbox pattern: the row commits atomically with the caller's domain writes.
+        """
+        if not isinstance(session, AsyncSession):
+            msg = "broker.publish requires an sqlalchemy.ext.asyncio.AsyncSession"
+            raise TypeError(msg)
+        payload, hdrs = _encode_payload(body, headers=headers, correlation_id=correlation_id)
+        await session.execute(insert(self._outbox_table).values(queue=queue, payload=payload, headers=hdrs))
+
+    async def publish_batch(  # ty: ignore[invalid-method-override]
+        self,
+        *bodies: typing.Any,
+        queue: str,
+        session: AsyncSession,
+        headers: dict[str, str] | None = None,
+    ) -> None:
+        """
+        Insert multiple outbox rows via *session*. Same transactional contract as ``publish``.
+
+        Each row gets its own auto-generated ``correlation_id``; pass *headers* to
+        share static headers across all rows.
+        """
+        if not isinstance(session, AsyncSession):
+            msg = "broker.publish_batch requires an sqlalchemy.ext.asyncio.AsyncSession"
+            raise TypeError(msg)
+        if not bodies:
+            return
+        rows = []
+        for body in bodies:
+            payload, hdrs = _encode_payload(body, headers=headers)
+            rows.append({"queue": queue, "payload": payload, "headers": hdrs})
+        await session.execute(insert(self._outbox_table), rows)
+
+    async def request(self, *args: typing.Any, **kwargs: typing.Any) -> typing.NoReturn:
+        msg = "OutboxBroker does not support request-reply"
+        raise NotImplementedError(msg)
+
+
+class _NoProducer:
+    """Stub satisfying FastStream's broker producer slot — the outbox has no real producer."""
+
+    async def publish(self, *_args: typing.Any, **_kwargs: typing.Any) -> typing.NoReturn:
+        msg = "OutboxBroker has no producer"
+        raise NotImplementedError(msg)
+
+    async def request(self, *_args: typing.Any, **_kwargs: typing.Any) -> typing.NoReturn:
+        msg = "OutboxBroker has no producer"
+        raise NotImplementedError(msg)
+
+    async def publish_batch(self, *_args: typing.Any, **_kwargs: typing.Any) -> typing.NoReturn:
+        msg = "OutboxBroker has no producer"
+        raise NotImplementedError(msg)
+
+    def connect(self, *_args: typing.Any, **_kwargs: typing.Any) -> None:
+        pass
+
+    def disconnect(self) -> None:
+        pass