xtr-orm-messenger 6.0.0__py3-none-any.whl

xtr_orm_messenger/.agents/skills/xtr-orm-messenger/SKILL.md

@@ -0,0 +1,108 @@
+---
+name: xtr-orm-messenger
+description: How to queue xtr-messenger messages in a database table with xtr-orm-messenger — an orm DSN naming an xtr-orm connection, OrmTransportFactory registered by the orm bundle, queues, delays, redelivery of messages a dead worker held, PostgreSQL notifications, sending in the unit of work's transaction, the messages table in migrations, and counting, listing or finding queued messages. Use when an application wants a message queue without a broker, installs xtr-messenger[orm], configures an orm transport, routes messages to the database, runs a worker on a table, migrates the messenger_messages table, or tests code that sends through the orm transport.
+---
+
+# xtr-orm-messenger
+
+A transport for xtr-messenger whose messages live in a table on a connection xtr-orm
+configures. It ships no bundle: with the messenger and orm bundles active, the orm bundle
+registers it once it is installed. For buses, routing and workers load the `xtr-messenger`
+skill; for connections and migrations, `xtr-orm`.
+
+## Quick reference
+
+- DSN — `orm://<connection>`, `orm://` for the default connection; queue with
+  `TransportConfig(dsn, queue="jobs")` or `?queue=jobs`, `default` otherwise.
+- Options (DSN query or `options`) — `table_name` (`messenger_messages`), `redeliver_timeout`
+  (`3600` s), `auto_setup` (`true`), `poll_interval` (`1.0` s), `transactional` (`false`),
+  `use_notify` (`true`), `check_delayed_interval` / `get_notify_timeout` (`60000` ms).
+- `OrmTransportFactory(registry, serializer, clock=None)` — not discovered by scheme: the orm
+  bundle registers it with the messenger's envelope serializer, or you pass it to
+  `MessageBusFactory` / `WorkerFactory`.
+- Migrations — each connection's migrator adds the table of every `orm://` transport on it, so
+  `orm:migrations:diff` writes it; `messages_tables(bus_config, connection, default)` names them.
+- Settling deletes the row; a row handed out longer than `redeliver_timeout` is claimed again.
+- A `DelayStamp` makes the message due later, rounded up to a whole second.
+
+## Send and consume without a container
+
+```python
+from xtr_messenger import (
+    JsonSerializer,
+    MessageBusConfig,
+    MessageBusFactory,
+    TransportConfig,
+    WorkerFactory,
+)
+from xtr_orm import ConnectionRegistry
+from xtr_orm_messenger import OrmTransportFactory
+from xtr_serializer import JsonEncoder, PydanticNormalizer, Serializer
+
+registry = ConnectionRegistry()
+registry.register("default", engine=engine, migrator=migrator, database=database)
+wire = JsonSerializer(Serializer([PydanticNormalizer()], [JsonEncoder()]))
+factory = OrmTransportFactory(registry, wire)
+config = MessageBusConfig(
+    transports={"jobs": TransportConfig("orm://default", queue="jobs")},
+    routing={IngestDocument: "jobs"},
+)
+
+await MessageBusFactory(config, [factory]).bus().dispatch(IngestDocument(document_id=1))
+await WorkerFactory(config, [factory]).worker(["jobs"]).run()
+```
+
+Pass the same factory to the bus and the worker: it keeps one transport per name.
+
+## Send with the handler's changes
+
+`TransportConfig("orm://default?transactional=true")` writes the message in the session of the
+unit of work under way, so it is sent only when that unit commits. List `orm_transaction` in the
+bus middleware, or commit the session yourself; outside a unit the message commits at once.
+
+## Look into a queue
+
+```python
+transport = factory.transport("jobs", config.transports["jobs"])
+await transport.count()
+messages = [envelope async for envelope in transport.all(limit=10)]
+await transport.find(message_id)
+await transport.keepalive(envelope, seconds=600)  # a handler running past redeliver_timeout
+```
+
+## Testing
+
+Run against a SQLite file per test (`sqlite+aiosqlite:///{tmp_path / 'app.sqlite'}`) and freeze
+a `MockClock`: `OrmTransportFactory(registry, wire, clock=clock)` makes delays and redelivery happen
+when the test moves the clock. A worker's `get()` never runs dry — stop the worker from a
+handler, or run it in a task and call `stop()`. `in-memory://` stays the transport for tests that
+do not care where messages are kept.
+
+## Use in an application
+
+1. With `MessengerBundle` and `OrmBundle` already active, `uv add "xtr-messenger[orm]"`. Nothing
+   to activate: the orm bundle registers the transport. No driver to add either: the transport
+   uses the connection's, notifications included when that driver is asyncpg.
+2. In `<app>/config/packages/messenger.py`, name a transport `orm://default` and route messages
+   to it.
+3. Run `<script> orm:migrations:diff` — the next revision creates the messages table. Add
+   `auto_setup=false` to the DSN to leave the table to the migrations alone.
+4. Run a worker with `<script> messenger:consume <transport>`.
+5. Check with `<script> debug:config messenger` — the `orm://` transports are listed.
+
+## Errors
+
+- `OrmMessengerError` — the base, itself a `MessageBusError`.
+- `TransportError` — the database refused a send, claim, settle or listing, or an envelope was
+  settled on a transport it was not received from; `transport`, `operation`, `reason`.
+- `InvalidOrmDsnError` — the DSN names a connection the registry does not hold; a `LookupError`.
+- `UnknownTransportOptionError` / `InvalidTransportOptionError` — from xtr-messenger, for a
+  setting the transport does not take or a value it cannot read.
+
+## Do not
+
+- Do not put a database URL in the DSN: it names a connection, configured in the orm.
+- Do not rely on `transactional=true` without something committing the unit's session.
+- Do not set `redeliver_timeout` below the time a handler takes, unless it calls `keepalive`.
+- Do not expect notifications through a PostgreSQL driver other than asyncpg: it polls.
+- Do not look for an `OrmMessengerBundle`: there is none; list `OrmBundle` and `MessengerBundle`.

xtr_orm_messenger/__init__.py

@@ -0,0 +1,60 @@
+"""A database transport for xtr-messenger, on the connections xtr-orm configures.
+
+Messages are kept in a table on one of the application's connections, named by
+the DSN — ``orm://default``. A worker claims the oldest one of its queue that
+is due, a delayed one falls due when its delay has passed, a message whose
+worker died is handed out again, and on PostgreSQL a waiting worker is woken
+the moment a message is sent.
+"""
+
+from __future__ import annotations
+
+from importlib.metadata import PackageNotFoundError, version
+
+from .exception import InvalidOrmDsnError, OrmMessengerError, TransportError
+from .transport import (
+    DEFAULT_QUEUE,
+    DEFAULT_TABLE_NAME,
+    ORM_OPTIONS,
+    ORM_SCHEME,
+    Connection,
+    MessagesTableDriver,
+    OrmOptions,
+    OrmReceivedStamp,
+    OrmReceiver,
+    OrmSender,
+    OrmTransport,
+    OrmTransportFactory,
+    StoredMessage,
+    messages_table,
+    messages_tables,
+)
+
+try:
+    __version__ = version("xtr-orm-messenger")
+except PackageNotFoundError:  # pragma: no cover
+    # Running from a source tree with no installed metadata to read; having no
+    # version is better than refusing to import.
+    __version__ = "0+unknown"
+
+__all__ = [
+    "DEFAULT_QUEUE",
+    "DEFAULT_TABLE_NAME",
+    "ORM_OPTIONS",
+    "ORM_SCHEME",
+    "Connection",
+    "InvalidOrmDsnError",
+    "MessagesTableDriver",
+    "OrmMessengerError",
+    "OrmOptions",
+    "OrmReceivedStamp",
+    "OrmReceiver",
+    "OrmSender",
+    "OrmTransport",
+    "OrmTransportFactory",
+    "StoredMessage",
+    "TransportError",
+    "__version__",
+    "messages_table",
+    "messages_tables",
+]

xtr_orm_messenger/exception/__init__.py

@@ -0,0 +1,9 @@
+"""Everything this library raises, each one an :class:`OrmMessengerError`."""
+
+from __future__ import annotations
+
+from .invalid_orm_dsn_error import InvalidOrmDsnError
+from .orm_messenger_error import OrmMessengerError
+from .transport_error import TransportError
+
+__all__ = ["InvalidOrmDsnError", "OrmMessengerError", "TransportError"]

xtr_orm_messenger/exception/invalid_orm_dsn_error.py

@@ -0,0 +1,34 @@
+"""An ``orm://`` DSN names a connection the application does not configure."""
+
+from __future__ import annotations
+
+from .orm_messenger_error import OrmMessengerError
+
+__all__ = ["InvalidOrmDsnError"]
+
+
+class InvalidOrmDsnError(OrmMessengerError, LookupError):
+    """An ``orm://`` DSN names a connection the application does not configure.
+
+    Also a :class:`LookupError`, as a missing key in any other mapping is.
+
+    Attributes:
+        transport: The transport whose DSN names it.
+        connection: The connection named.
+        known: Every connection that is configured.
+    """
+
+    transport: str
+    connection: str
+    known: tuple[str, ...]
+
+    def __init__(self, transport: str, connection: str, known: tuple[str, ...]) -> None:
+        """Record the transport, the connection it named and the ones that exist."""
+        self.transport = transport
+        self.connection = connection
+        self.known = known
+        names = ", ".join(f'"{each}"' for each in known) or "none"
+        super().__init__(
+            f'The "{transport}" transport names the connection "{connection}", which is not '
+            f"configured; the connections are: {names}."
+        )

xtr_orm_messenger/exception/orm_messenger_error.py

@@ -0,0 +1,15 @@
+"""The root every error in this library derives from."""
+
+from __future__ import annotations
+
+from xtr_messenger import MessageBusError
+
+__all__ = ["OrmMessengerError"]
+
+
+class OrmMessengerError(MessageBusError):
+    """Base class for every error raised by this library.
+
+    Also a :class:`~xtr_messenger.MessageBusError`, so code catching what the
+    bus can go wrong with catches this transport's failures too.
+    """

xtr_orm_messenger/exception/transport_error.py

@@ -0,0 +1,32 @@
+"""The database refused what a transport asked of it."""
+
+from __future__ import annotations
+
+from .orm_messenger_error import OrmMessengerError
+
+__all__ = ["TransportError"]
+
+
+class TransportError(OrmMessengerError):
+    """The database refused what a transport asked of it.
+
+    Raised from the driver's own error, which stays the cause, so nothing it
+    said is lost. The message names the transport and the operation, never the
+    connection's URL, whose credentials a traceback would otherwise carry.
+
+    Attributes:
+        transport: The transport the operation was for.
+        operation: What was asked — ``send``, ``get``, ``ack``, ...
+        reason: What went wrong.
+    """
+
+    transport: str
+    operation: str
+    reason: str
+
+    def __init__(self, transport: str, operation: str, reason: str) -> None:
+        """Record which transport failed, at what, and why."""
+        self.transport = transport
+        self.operation = operation
+        self.reason = reason
+        super().__init__(f'The "{transport}" transport could not {operation}: {reason}.')

xtr_orm_messenger/recipe/__init__.py

@@ -0,0 +1,15 @@
+"""This directory holds the package's recipe: its manifest.
+
+The recipe is what *Use in an application* in the README lists, written as
+data: the steps only a person can take — naming an ``orm://`` transport in the
+message bus configuration and routing messages to it. An application applies it
+with ``xtr-recipes recipes:sync``, which finds this module through the
+``xtr_recipes`` entry point.
+
+Nothing is importable from here. The manifest is read as a file, so a recipe
+is the same from a wheel as from a working tree.
+"""
+
+from __future__ import annotations
+
+__all__: list[str] = []

xtr_orm_messenger/recipe/manifest.toml

@@ -0,0 +1,21 @@
+# The recipe for xtr-orm-messenger: what *Use in an application* in the README
+# takes, as data. Keep the two in step — the README is the source.
+#
+# No [bundles]: the package ships none — the orm bundle registers the transport
+# factory when the messenger bundle is active too. No [files] and no [env]: a
+# transport is named in the message bus configuration, which is the messenger's,
+# and its DSN names a connection the orm already configures.
+
+[gitignore]
+# Nothing: the messages live in the database.
+lines = []
+
+[notes]
+steps = [
+  "Name a transport orm://default in config/packages/messenger.py and route your messages to it: TransportConfig(\"orm://default\", queue=\"jobs\").",
+  "Run orm:migrations:diff: it writes the messages table of every orm:// transport on that connection. Add auto_setup=false to the DSN to leave the table to the migrations alone.",
+]
+check = [
+  "<script> debug:bundles — messenger and orm are listed and active",
+]
+run = ["<script> messenger:consume <transport>"]

xtr_orm_messenger/transport/__init__.py

@@ -0,0 +1,38 @@
+"""The ``orm://`` transport: messages kept in a table on an xtr-orm connection."""
+
+from __future__ import annotations
+
+from .connection import LISTING_PAGE_SIZE, Connection
+from .messages_table import messages_table
+from .messages_table_driver import MessagesTableDriver
+from .messages_tables import messages_tables
+from .orm_options import DEFAULT_QUEUE, DEFAULT_TABLE_NAME, ORM_OPTIONS, ORM_SCHEME, OrmOptions
+from .orm_received_stamp import OrmReceivedStamp
+from .orm_receiver import MAX_RETRIES, RECONNECT_TIMEOUT, OrmReceiver
+from .orm_sender import OrmSender
+from .orm_transport import OrmTransport
+from .orm_transport_factory import OrmTransportFactory
+from .postgresql_notifier import PostgreSqlNotifier
+from .stored_message import StoredMessage
+
+__all__ = [
+    "DEFAULT_QUEUE",
+    "DEFAULT_TABLE_NAME",
+    "LISTING_PAGE_SIZE",
+    "MAX_RETRIES",
+    "ORM_OPTIONS",
+    "ORM_SCHEME",
+    "RECONNECT_TIMEOUT",
+    "Connection",
+    "MessagesTableDriver",
+    "OrmOptions",
+    "OrmReceivedStamp",
+    "OrmReceiver",
+    "OrmSender",
+    "OrmTransport",
+    "OrmTransportFactory",
+    "PostgreSqlNotifier",
+    "StoredMessage",
+    "messages_table",
+    "messages_tables",
+]

xtr_orm_messenger/transport/_database_errors.py

@@ -0,0 +1,71 @@
+"""Telling apart what a driver's error means, whichever database raised it.
+
+Each database reports a deadlock, a lock it gave up waiting for or a missing
+table its own way — PostgreSQL with an SQLSTATE, MySQL with an error number,
+SQLite only in its message — and the driver's error is all SQLAlchemy hands
+back. Reading it in one place keeps every caller to one question.
+"""
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, Final
+
+if TYPE_CHECKING:
+    from sqlalchemy.exc import DBAPIError
+
+__all__ = ["is_missing_table", "is_retryable"]
+
+_RETRYABLE_SQLSTATES: Final = frozenset({"40001", "40P01", "55P03"})
+"""Serialization failure, deadlock, lock not available."""
+
+_RETRYABLE_MYSQL_CODES: Final = frozenset({1205, 1213})
+"""Lock wait timeout, deadlock."""
+
+_RETRYABLE_MESSAGES: Final = ("database is locked", "database table is locked")
+"""SQLite, whose errors carry no code a driver exposes."""
+
+_MISSING_TABLE_SQLSTATE: Final = "42P01"
+_MISSING_TABLE_MYSQL_CODE: Final = 1146
+_MISSING_TABLE_MESSAGE: Final = "no such table"
+
+
+def is_retryable(error: DBAPIError) -> bool:
+    """Tell whether ``error`` is a conflict with another transaction, worth trying again.
+
+    Workers claiming messages from one table run into each other: a deadlock,
+    a lock they waited too long for. Neither says anything about the message,
+    so the same statement a moment later goes through.
+    """
+    original = error.orig
+    return (
+        _sqlstate(original) in _RETRYABLE_SQLSTATES
+        or _mysql_code(original) in _RETRYABLE_MYSQL_CODES
+        or any(text in str(original) for text in _RETRYABLE_MESSAGES)
+    )
+
+
+def is_missing_table(error: DBAPIError) -> bool:
+    """Tell whether ``error`` says the table the statement names does not exist."""
+    original = error.orig
+    return (
+        _sqlstate(original) == _MISSING_TABLE_SQLSTATE
+        or _mysql_code(original) == _MISSING_TABLE_MYSQL_CODE
+        or _MISSING_TABLE_MESSAGE in str(original)
+    )
+
+
+def _sqlstate(original: BaseException | None) -> str | None:
+    """Return the SQLSTATE a PostgreSQL driver put on its error, if any."""
+    state: object = getattr(original, "sqlstate", None)
+    return state if isinstance(state, str) else None
+
+
+def _mysql_code(original: BaseException | None) -> int | None:
+    """Return the error number a MySQL driver put first in its error's arguments, if any."""
+    if original is None:
+        return None
+    arguments: tuple[object, ...] = original.args
+    if not arguments:
+        return None
+    first = arguments[0]
+    return first if isinstance(first, int) and not isinstance(first, bool) else None