django-aiogram 4.0.0.dev0__py3-none-any.whl

django_aiogram/eventlog/signals.py

@@ -0,0 +1,84 @@
+"""The seam a project gets metrics out of.
+
+Deliberately a ``django.dispatch.Signal`` rather than a setting naming a dotted
+path. A setting would need an entry in ``Settings.md``, a check id, an
+``import_string``, a lazy cache to keep the import off the hot path, and a
+decision about what to do when the path is wrong. A signal needs none of that,
+``send_robust`` contains most of what a receiver can do wrong, and connecting one is
+the thing every Django developer already knows how to do. *Most*: see
+:meth:`~django_aiogram.eventlog.recorder.EventRecorder._publish` for the receiver shape
+Django's own containment misses, which is why this package does not rely on it
+alone.
+
+This module imports ``django.dispatch`` and nothing else — not aiogram, not the
+ORM, not the rest of this package. A metrics module can import it at settings
+time without dragging anything in.
+"""
+
+from django.dispatch import Signal
+
+#: Fired once per batch of recorded events, on whichever thread flushed that batch —
+#: normally the event writer's own, and three other threads can be it.
+#:
+#: Receivers get ``events``: a tuple of :class:`~django_aiogram.eventlog.recorder.Event`,
+#: whose field names are pinned by ``tests/test_public_surface.py`` and are
+#: therefore public API. ``sender`` is the recorder instance.
+#:
+#: Four things about it are load-bearing, and three of them are surprising:
+#:
+#: * **It fires whether or not the event log is on.** The table and the metrics
+#:   are separate decisions: connect a receiver and the events flow, with
+#:   ``EVENT_LOG`` left off and no migration in sight.
+#: * **Payload summaries are the only part of ``detail`` the log gates.** With the
+#:   log off, ``detail`` still carries what the recording seam measured itself —
+#:   a send's ``duration_ms``, a retry's ``retry_after``, a queueing failure's
+#:   ``stage``, a gap's ``dropped`` count. What is missing is the summarized
+#:   arguments, because redacting and bounding a payload is the expensive part of
+#:   recording and no part of counting. A receiver that needs message bodies needs
+#:   the log on as well, and then ``EVENT_LOG_PAYLOAD`` decides what is in there.
+#: * **``EVENT_LOG_KINDS`` filters this too, with one exemption.** It is one answer
+#:   to "which events does this deployment care about", not two — so a receiver
+#:   sees exactly the kinds the table would have kept. ``log.dropped`` is the
+#:   exception, in both directions: it is the record that recording itself fell
+#:   behind, and a deployment that filtered it out would read the hole as quiet
+#:   traffic. The table has always been exempt from the filter for that row, and
+#:   receivers are exempt with it.
+#: * **The write is attempted before receivers see the batch.** So nothing a
+#:   receiver does can change a row that was written, which is why they get real
+#:   ``Event`` objects rather than copies — the ``detail`` dict inside one is an
+#:   ordinary mutable dict shared with the other receivers, so treat it as
+#:   read-only. It is *attempted*, not guaranteed: a write that failed still
+#:   publishes, because a database being down is exactly when someone is watching a
+#:   dashboard. Receiving a batch is therefore not evidence that a row exists for
+#:   it, and with ``EVENT_LOG`` off there is no row by design.
+#:
+#: The rule is **whichever thread flushed the batch publishes it**, and normally that
+#: is the event writer's, once the batch's own write has been attempted — so a slow
+#: receiver delays neither a send nor that write, only later batches and the writer's
+#: shutdown. Never delaying a send is the point of the design; the rest follows from
+#: the rule rather than being promised separately.
+#:
+#: Three other threads can flush a batch, so three other threads can publish one:
+#:
+#: * whatever calls ``EventRecorder.drain_once()``, which exists so a test can drive
+#:   the real flush path on its own thread
+#: * at shutdown, whichever thread called ``stop()``, for whatever the writer had not
+#:   drained. Those events are published rather than dropped because they are the last
+#:   ones before the process goes, and there is by then no writer left to hand them to
+#: * under ``EVENT_LOG_SYNC``, the thread that recorded the event, which is the one
+#:   case with no batch and no writer involved at all
+#:
+#: ``EVENT_LOG_SYNC`` only takes effect with the log on — there is nothing to insert
+#: synchronously otherwise — so the write is always attempted there, and may still
+#: fail. It is a testing setting, and receivers running inside the send path is one
+#: more reason to keep it one.
+#:
+#: A dispatch that fails can leave later receivers without the batch: ``send_robust``
+#: stops its own loop when Django's failure logging raises, which it does for a
+#: callable instance. This package catches that and logs
+#: ``publishing recorded events failed``, but the receivers after the offending one
+#: were never called.
+#:
+#: The write happens before either, and only when ``EVENT_LOG`` is on: with the log
+#: off nothing is written at all, and a receiver is the only thing the batch reaches.
+events_recorded: Signal = Signal()

django_aiogram/eventlog/writer.py

@@ -0,0 +1,231 @@
+"""Turning recorded events into rows, on the one thread allowed to do it.
+
+This is the only module in the package that touches the ORM, and it is imported
+from inside the writer thread rather than at module scope — that is what keeps
+``recorder`` free of ``django.db``, and therefore keeps ``record()`` legal from
+a coroutine and free in a process with the log switched off.
+"""
+
+import datetime
+import logging
+from collections.abc import Sequence
+
+from django.conf import settings as django_settings
+from django.db import (
+    DEFAULT_DB_ALIAS,
+    DatabaseError,
+    InterfaceError,
+    OperationalError,
+    connections,
+    transaction,
+)
+from django.utils import timezone
+
+from django_aiogram.eventlog.dbrouter import event_log_database
+from django_aiogram.eventlog.recorder import Event
+from django_aiogram.exceptions import DjangoRedisAiogramError
+from django_aiogram.models import TelegramEvent
+from django_aiogram.wire.payloads import redact_keys, redact_text, redact_values, secrets
+
+logger = logging.getLogger('django_aiogram')
+
+#: below this a failed batch is retried row by row rather than bisected
+ROW_BY_ROW = 32
+
+
+class EventLogRefusedError(DjangoRedisAiogramError):
+    """Every row of a batch was refused.
+
+    Not a `DatabaseError`: it is this module's verdict on one, and the difference
+    matters because the recorder treats it as a failed flush rather than as
+    something to bisect further. It never escapes that flush.
+    """
+
+    def __init__(self, count: int) -> None:
+        """Name how many rows were lost, which is what the log line reports."""
+        super().__init__(f'the database refused all {count} rows of this batch')
+
+
+def log_alias() -> str:
+    """Return the alias rows are written to and read from."""
+    return event_log_database() or DEFAULT_DB_ALIAS
+
+
+def _moment(stamp: float) -> datetime.datetime:
+    """Return the recorded instant, in whichever flavour of datetime this project stores."""
+    moment = datetime.datetime.fromtimestamp(stamp, tz=datetime.timezone.utc)
+    return moment if django_settings.USE_TZ else timezone.make_naive(moment)
+
+
+def _text(value: object, length: int) -> str:
+    """Cut a value to its column's width, without the NULs PostgreSQL refuses."""
+    if value is None:
+        return ''
+    return str(value).replace('\x00', '')[:length]
+
+
+def to_row(
+    event: Event,
+    keys: frozenset[str] | None = None,
+    configured: tuple[str, ...] | None = None,
+) -> TelegramEvent:
+    """Build the unsaved row for one event, sanitised so it cannot poison a batch.
+
+    Redaction happens here as well as at the producer. This is the boundary rows
+    cross, and the rule is that the token must not reach one: a caller that
+    builds an Event by hand, or a new seam that forgets, would otherwise put an
+    aiogram error message — which carries the API URL, which carries the token —
+    straight into a column.
+    """
+    if keys is None:
+        keys = redact_keys()
+    if configured is None:
+        configured = secrets()
+    return TelegramEvent(
+        created_at=_moment(event.created_at),
+        correlation_id=event.correlation_id,
+        kind=_text(event.kind, 48),
+        function=_text(event.function, 64),
+        chat_id=event.chat_id,
+        user_id=event.user_id,
+        message_id=event.message_id,
+        update_id=event.update_id,
+        worker=_text(event.worker, 128),
+        attempt=max(0, event.attempt),
+        duration_ms=event.duration_ms,
+        error_code=_text(event.error_code, 64),
+        error=_text(redact_text(str(event.error or ''), configured), 20000),
+        detail=redact_values(event.detail or {}, keys, configured),
+    )
+
+
+def write_batch(events: Sequence[Event]) -> int:
+    """Insert one batch, recycling a connection the database has since dropped.
+
+    Returns how many rows did **not** land, so the caller can count them. A total
+    refusal raises; a partial one used to return here indistinguishable from a clean
+    write, so the rows the database refused one at a time disappeared with no counter,
+    no ``log.dropped`` row and nothing but a per-row line in the log — while the feed
+    read as complete coverage of the period that lost them.
+    """
+    alias = log_alias()
+    _recycle(alias)
+    # resolved once for the batch: both walk the settings, and a batch is 200 rows
+    keys, configured = redact_keys(), secrets()
+    rows = [to_row(event, keys, configured) for event in events]
+    manager = TelegramEvent.objects.using(alias)
+    try:
+        # the savepoint is what keeps a failed log write from taking the
+        # caller's data with it: under EVENT_LOG_SYNC this runs on the caller's
+        # thread, inside whatever atomic() block the caller opened
+        with transaction.atomic(using=alias):
+            manager.bulk_create(rows)
+    except (OperationalError, InterfaceError) as error:
+        # the connection died between the check above and the insert; one retry
+        # on a fresh one is the difference between losing a batch and not
+        _recycle(alias, force=True)
+        # the retry needs the same net as the first attempt: a fresh connection
+        # rejecting one poison row must not cost the whole batch
+        written = _write_half(rows, alias)
+        _refused(rows, written, error)
+        return len(rows) - written
+    except DatabaseError as error:
+        written = _write_one_by_one(rows, alias)
+        _refused(rows, written, error)
+        return len(rows) - written
+    return 0
+
+
+def _recycle(alias: str, *, force: bool = False) -> None:
+    """Discard a connection the database has since dropped, unless we are inside a transaction.
+
+    Before the work, not after: this is what discards a connection whose
+    CONN_MAX_AGE expired, that a restart killed, or that a previous error marked
+    unusable. Closing afterward would leave a broken one in place.
+
+    Never while the caller holds a transaction open, though. Under EVENT_LOG_SYNC
+    this runs on the caller's thread inside their ``atomic()`` block — and on
+    PostgreSQL and MySQL, closing a connection there marks the whole transaction
+    for rollback, so recording an event would destroy the writes the caller made
+    alongside it. A connection Django is already using is not stale anyway.
+
+    ``in_atomic_block`` is only half of what *open transaction* means. With autocommit
+    off — ``transaction.set_autocommit(False)``, or ``AUTOCOMMIT: False`` on the alias —
+    the server holds one from the first statement with no block anywhere in sight, and
+    then it is worse than the block case: ``close_if_unusable_or_obsolete`` sees
+    ``get_autocommit()`` disagreeing with the configured value and closes on that basis,
+    while ``close()`` skips setting ``needs_rollback`` precisely *because*
+    ``in_atomic_block`` is False. So the caller's writes are rolled back by the server
+    and nothing raises. Measured on PostgreSQL 16: the caller's row was gone after a
+    successful ``commit()``, with the event row written.
+
+    One alias, not all of them. ``close_old_connections()`` walks every initialized
+    connection, so with ``EVENT_LOG_DATABASE`` pointing somewhere of its own it
+    reaches past the log's connection — which is not in a transaction — and closes
+    the caller's ``default`` one, which is. The guard above would then be reading
+    the wrong connection's state. The log has no business touching one it never
+    writes to.
+    """
+    connection = connections[alias]
+    if connection.in_atomic_block or not connection.get_autocommit():
+        return
+    if force:
+        connection.close()
+        return
+    connection.close_if_unusable_or_obsolete()
+
+
+def _refused(rows: list[TelegramEvent], written: int, error: Exception) -> None:
+    """Raise when a batch reached the database and none of it landed.
+
+    The bisecting ladder below catches ``DatabaseError`` at every rung, so a
+    database that refuses everything — no table, no permission, no disk — used to
+    end in a normal return. The recorder read that as success: its failure counter
+    never moved, so the backoff never engaged and no ``log.dropped`` row was ever
+    written. It hammered the database once per flush interval, for ever.
+    """
+    if rows and not written:
+        raise EventLogRefusedError(len(rows)) from error
+
+
+def _write_half(rows: list[TelegramEvent], alias: str) -> int:
+    """Insert one half of a bisected batch, splitting it again if it still fails."""
+    try:
+        with transaction.atomic(using=alias):
+            TelegramEvent.objects.using(alias).bulk_create(rows)
+    except (DatabaseError, InterfaceError):
+        # both, because Django defines them as siblings under `Error`: a connection the
+        # server dropped raises `InterfaceError`, which `DatabaseError` alone lets escape
+        # `write_batch` — with no refused count and no `EventLogRefusedError`
+        return _write_one_by_one(rows, alias)
+    return len(rows)
+
+
+def _write_row(row: TelegramEvent, alias: str) -> bool:
+    """Insert one row, dropping it if the database refuses it."""
+    try:
+        # the savepoint is not optional: on PostgreSQL a failed statement aborts
+        # the transaction, so one bad row would take every later one with it
+        with transaction.atomic(using=alias):
+            row.save(force_insert=True, using=alias)
+    except (DatabaseError, InterfaceError):
+        logger.exception('dropping an event the database refused', extra={'tg_kind': row.kind})
+        return False
+    return True
+
+
+def _write_one_by_one(rows: list[TelegramEvent], alias: str) -> int:
+    """Save rows individually, dropping only the ones the database refuses."""
+    if len(rows) > ROW_BY_ROW:
+        # bisect first, so a 200-row batch does not become 200 statements
+        middle = len(rows) // 2
+        return _write_half(rows[:middle], alias) + _write_half(rows[middle:], alias)
+    return sum(_write_row(row, alias) for row in rows)
+
+
+def close_connections() -> None:
+    """Release the calling thread's connections; nothing else ever will."""
+    try:
+        connections.close_all()
+    except Exception:
+        logger.exception('could not close the event writer connection')

django_aiogram/exceptions.py

@@ -0,0 +1,60 @@
+"""The exceptions this package raises.
+
+They live together, and build their own messages, so that call sites raise a
+named domain error instead of a bare builtin with a formatted string.
+"""
+
+
+class DjangoRedisAiogramError(Exception):
+    """Base class for every error this package raises."""
+
+
+class LoopUnavailableError(DjangoRedisAiogramError, RuntimeError):
+    """No event loop in this process can take an update right now.
+
+    A refusal, not a handler failure: nothing has run, so the update is still
+    Telegram's to redeliver. The webhook view answers a non-2xx for this and 200
+    for everything else, which is the difference between a retry and a loop.
+
+    Raised as one of the two below, so a caller can tell a shutdown from a fault.
+    """
+
+
+class ShuttingDownError(LoopUnavailableError):
+    """The update arrived while this process was tearing the bot down."""
+
+    def __init__(self) -> None:
+        """Say which window this is, since it closes on its own."""
+        super().__init__(
+            'the bot is shutting down, so this update was not handled here. '
+            'Telegram will redeliver it; another process, or this one after it '
+            'restarts, will take it.',
+        )
+
+
+class LoopThreadNotStartedError(LoopUnavailableError):
+    """The thread this process gives the loop did not start in time."""
+
+    def __init__(self, timeout: float) -> None:
+        """Name the deadline, which is the only number worth acting on."""
+        super().__init__(
+            f'the event loop thread did not start within {timeout}s, and driving '
+            'the loop from this thread instead would put two threads on one loop. '
+            'The update was not handled here.',
+        )
+
+
+class SerializationError(DjangoRedisAiogramError):
+    """A payload could not be encoded or decoded."""
+
+
+class UnknownApiMethodError(DjangoRedisAiogramError, ValueError):
+    """A queued payload named something that is not a Telegram API method."""
+
+    def __init__(self, function: str, method_count: int) -> None:
+        """Name the rejected method and how many the Bot API actually has."""
+        super().__init__(
+            f'{function!r} is not a Telegram API method. Queued payloads may only '
+            f'name one of the {method_count} methods aiogram exposes for the '
+            f'Bot API; see the Serialization page.',
+        )