iddqueue 0.13.0rc2__tar.gz → 0.13.0rc4__tar.gz

{iddqueue-0.13.0rc2 → iddqueue-0.13.0rc4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: iddqueue
-Version: 0.13.0rc2
+Version: 0.13.0rc4
 Summary: Postgres Broker for Dramatiq Task Queue
 Keywords: postgres,task queue,dramatiq
 Author: Étienne BERSAC
@@ -12,11 +12,13 @@ Requires-Dist: psycopg-pool>=3.3.3,<4
 Requires-Dist: tenacity>=9,<10
 Requires-Dist: psycopg[binary]>=3.3.6,<4 ; extra == 'binary'
 Requires-Dist: dramatiq[prometheus]>=2.2.1,<3 ; extra == 'monitoring'
+Requires-Dist: sqlalchemy>=2.0,<3 ; extra == 'sqlalchemy'
 Requires-Python: >=3.10, <4
 Project-URL: Repository, https://github.com/wa-pis/iddqueue
 Project-URL: Issues, https://github.com/wa-pis/iddqueue/issues
 Provides-Extra: binary
 Provides-Extra: monitoring
+Provides-Extra: sqlalchemy
 Description-Content-Type: text/markdown
 
 # IDDQueue
@@ -27,8 +29,8 @@ A PostgreSQL broker and Results backend for [Dramatiq](https://dramatiq.io/),
 built on synchronous Psycopg 3. Tasks, results and coordination live in PostgreSQL;
 no Redis or ORM is required by IDDQueue.
 
-**Current release: [0.13.0rc2](https://pypi.org/project/iddqueue/0.13.0rc2/)** —
-a release candidate for evaluation. [GitHub assets](https://github.com/wa-pis/iddqueue/releases/tag/v0.13.0rc2)
+**Current release: [0.13.0rc4](https://pypi.org/project/iddqueue/0.13.0rc4/)** —
+a release candidate for evaluation. [GitHub assets](https://github.com/wa-pis/iddqueue/releases/tag/v0.13.0rc4)
 include the wheel, sdist and SHA256 checksums.
 
 IDDQueue is a fork of [DALIBO's dramatiq-pg](https://gitlab.com/dalibo/dramatiq-pg)
@@ -56,7 +58,7 @@ See [compatibility and API stability](SUPPORT.md).
 
 ```sh
 uv venv
-uv pip install "iddqueue[binary]==0.13.0rc2"
+uv pip install "iddqueue[binary]==0.13.0rc4"
 ```
 
 The `binary` extra supplies libpq. Base installs require system libpq;
@@ -119,11 +121,13 @@ Close broker-owned pools on shutdown; callers close pools they supply.
 
 ## Guides and examples
 
+[Domain actors and one worker bootstrap](docs/domains.md) are included since RC3.
+
 - [Documentation index](docs/index.md), [user guide](docs/user-guide.md), [API](docs/api.md).
 - [Detailed recipes](docs/recipes.md): transactions, middleware, controls, metrics and schedules.
 - [FastAPI](docs/fastapi.md): lifespan, async endpoint thread offload and separate worker.
 - [Deployment and retention](docs/deployment-guide.md), [why PostgreSQL](docs/why.md).
-- [RC notes](docs/rc-0.13.0rc2.md), [changelog](CHANGELOG.md), [release checks](docs/release.md).
+- [RC notes](docs/rc-0.13.0rc4.md), [changelog](CHANGELOG.md), [release checks](docs/release.md).
 
 The broker and result APIs are synchronous. Async actors use Dramatiq's AsyncIO
 middleware; FastAPI async endpoints offload publication to a thread.
@@ -138,27 +142,28 @@ is unverified. No tested Django integration is currently provided.
 ## Compared with dramatiq-pg
 
 Baseline: [dramatiq-pg 0.12.0](https://pypi.org/project/dramatiq-pg/0.12.0/),
-compared with the published IDDQueue 0.13.0rc2 prerelease. This describes the published version, not every
+compared with the published IDDQueue 0.13.0rc4 prerelease. This describes the published version, not every
 future upstream revision. Both projects provide a PostgreSQL Dramatiq broker.
 
-| Area | dramatiq-pg 0.12.0 | IDDQueue 0.13.0rc2 / practical benefit |
+| Area | dramatiq-pg 0.12.0 | IDDQueue 0.13.0rc4 / practical benefit |
 | --- | --- | --- |
 | Core storage and delivery | JSONB tasks/results, delayed tasks, LISTEN/NOTIFY, advisory locks, recovery and maintenance CLI | Preserved; at-least-once delivery still requires idempotent actors |
+| Actor initialization | Broker configured before actor imports | Opt-in Domain declarations import before DSN; explicit startup registration and domain queues |
 | Runtime | Python >=3.6,<4; Dramatiq >=1.5,<2; Psycopg 2 | Python >=3.10,<4; Dramatiq >=2.2.1,<3; synchronous Psycopg 3 and psycopg-pool |
 | Pool lifecycle | Psycopg 2 pools | Lazy owned pools, explicit close, caller-owned pool support |
-| Transactional publication | Regular enqueue | Caller-owned transaction API: commit business data and tasks together |
+| Transactional publication | Regular enqueue | Caller-owned Psycopg 3 or optional sync SQLAlchemy transaction: commit business data and tasks together |
 | Coordination | No PostgreSQL rate-limit/barrier backend | PostgreSQL rate limits, durable barriers and standard GroupCallbacks without Redis |
 | Task operations | stats, purge, recover, flush | Failed-task inspection/targeted retry, pause/resume, cooperative cancellation and opt-in attempt history |
 | Publication controls | Single-message enqueue | Deduplication keys/TTL and atomic batches of up to 1000; deduplication is not exactly-once execution |
 | Scheduling | Per-message delay | Also a multi-process fixed-interval scheduler; no cron/calendar expressions |
-| Observability | Basic CLI statistics | Ready/scheduled backlog, task age and optional Prometheus collector |
+| Observability | Basic CLI statistics | Queue/domain backlog and age, native error/retry counters, optional Prometheus collector and Grafana kit |
 | Storage namespaces | Configurable schema/table prefix; shared channel/lock domains | Instance-specific queries and schema/prefix-specific channels/locks; storage separation is not access control |
 | Dramatiq integration | Broker/results implementation | Verified pipelines, groups, AsyncIO and standard middleware; these are Dramatiq features, not a replacement orchestration engine |
 | Compatibility evidence | Historical upstream tests | CI on Python 3.10/3.13/3.14 × PostgreSQL 14/18, functional tests and installed-wheel checks |
 
 See [migration from dramatiq-pg](docs/migration.md) for dependency, pool,
 import, CLI and database changes, including backup and rollback. No throughput
-advantage is claimed here. The 0.13.0rc2 prerelease is available on PyPI; the legacy Django
+advantage is claimed here. The 0.13.0rc4 prerelease is available on PyPI; the legacy Django
 integration is archived and has not been verified with IDDQueue.
 
 ## Development and support

{iddqueue-0.13.0rc2 → iddqueue-0.13.0rc4}/README.md

@@ -6,8 +6,8 @@ A PostgreSQL broker and Results backend for [Dramatiq](https://dramatiq.io/),
 built on synchronous Psycopg 3. Tasks, results and coordination live in PostgreSQL;
 no Redis or ORM is required by IDDQueue.
 
-**Current release: [0.13.0rc2](https://pypi.org/project/iddqueue/0.13.0rc2/)** —
-a release candidate for evaluation. [GitHub assets](https://github.com/wa-pis/iddqueue/releases/tag/v0.13.0rc2)
+**Current release: [0.13.0rc4](https://pypi.org/project/iddqueue/0.13.0rc4/)** —
+a release candidate for evaluation. [GitHub assets](https://github.com/wa-pis/iddqueue/releases/tag/v0.13.0rc4)
 include the wheel, sdist and SHA256 checksums.
 
 IDDQueue is a fork of [DALIBO's dramatiq-pg](https://gitlab.com/dalibo/dramatiq-pg)
@@ -35,7 +35,7 @@ See [compatibility and API stability](SUPPORT.md).
 
 ```sh
 uv venv
-uv pip install "iddqueue[binary]==0.13.0rc2"
+uv pip install "iddqueue[binary]==0.13.0rc4"
 ```
 
 The `binary` extra supplies libpq. Base installs require system libpq;
@@ -98,11 +98,13 @@ Close broker-owned pools on shutdown; callers close pools they supply.
 
 ## Guides and examples
 
+[Domain actors and one worker bootstrap](docs/domains.md) are included since RC3.
+
 - [Documentation index](docs/index.md), [user guide](docs/user-guide.md), [API](docs/api.md).
 - [Detailed recipes](docs/recipes.md): transactions, middleware, controls, metrics and schedules.
 - [FastAPI](docs/fastapi.md): lifespan, async endpoint thread offload and separate worker.
 - [Deployment and retention](docs/deployment-guide.md), [why PostgreSQL](docs/why.md).
-- [RC notes](docs/rc-0.13.0rc2.md), [changelog](CHANGELOG.md), [release checks](docs/release.md).
+- [RC notes](docs/rc-0.13.0rc4.md), [changelog](CHANGELOG.md), [release checks](docs/release.md).
 
 The broker and result APIs are synchronous. Async actors use Dramatiq's AsyncIO
 middleware; FastAPI async endpoints offload publication to a thread.
@@ -117,27 +119,28 @@ is unverified. No tested Django integration is currently provided.
 ## Compared with dramatiq-pg
 
 Baseline: [dramatiq-pg 0.12.0](https://pypi.org/project/dramatiq-pg/0.12.0/),
-compared with the published IDDQueue 0.13.0rc2 prerelease. This describes the published version, not every
+compared with the published IDDQueue 0.13.0rc4 prerelease. This describes the published version, not every
 future upstream revision. Both projects provide a PostgreSQL Dramatiq broker.
 
-| Area | dramatiq-pg 0.12.0 | IDDQueue 0.13.0rc2 / practical benefit |
+| Area | dramatiq-pg 0.12.0 | IDDQueue 0.13.0rc4 / practical benefit |
 | --- | --- | --- |
 | Core storage and delivery | JSONB tasks/results, delayed tasks, LISTEN/NOTIFY, advisory locks, recovery and maintenance CLI | Preserved; at-least-once delivery still requires idempotent actors |
+| Actor initialization | Broker configured before actor imports | Opt-in Domain declarations import before DSN; explicit startup registration and domain queues |
 | Runtime | Python >=3.6,<4; Dramatiq >=1.5,<2; Psycopg 2 | Python >=3.10,<4; Dramatiq >=2.2.1,<3; synchronous Psycopg 3 and psycopg-pool |
 | Pool lifecycle | Psycopg 2 pools | Lazy owned pools, explicit close, caller-owned pool support |
-| Transactional publication | Regular enqueue | Caller-owned transaction API: commit business data and tasks together |
+| Transactional publication | Regular enqueue | Caller-owned Psycopg 3 or optional sync SQLAlchemy transaction: commit business data and tasks together |
 | Coordination | No PostgreSQL rate-limit/barrier backend | PostgreSQL rate limits, durable barriers and standard GroupCallbacks without Redis |
 | Task operations | stats, purge, recover, flush | Failed-task inspection/targeted retry, pause/resume, cooperative cancellation and opt-in attempt history |
 | Publication controls | Single-message enqueue | Deduplication keys/TTL and atomic batches of up to 1000; deduplication is not exactly-once execution |
 | Scheduling | Per-message delay | Also a multi-process fixed-interval scheduler; no cron/calendar expressions |
-| Observability | Basic CLI statistics | Ready/scheduled backlog, task age and optional Prometheus collector |
+| Observability | Basic CLI statistics | Queue/domain backlog and age, native error/retry counters, optional Prometheus collector and Grafana kit |
 | Storage namespaces | Configurable schema/table prefix; shared channel/lock domains | Instance-specific queries and schema/prefix-specific channels/locks; storage separation is not access control |
 | Dramatiq integration | Broker/results implementation | Verified pipelines, groups, AsyncIO and standard middleware; these are Dramatiq features, not a replacement orchestration engine |
 | Compatibility evidence | Historical upstream tests | CI on Python 3.10/3.13/3.14 × PostgreSQL 14/18, functional tests and installed-wheel checks |
 
 See [migration from dramatiq-pg](docs/migration.md) for dependency, pool,
 import, CLI and database changes, including backup and rollback. No throughput
-advantage is claimed here. The 0.13.0rc2 prerelease is available on PyPI; the legacy Django
+advantage is claimed here. The 0.13.0rc4 prerelease is available on PyPI; the legacy Django
 integration is archived and has not been verified with IDDQueue.
 
 ## Development and support

{iddqueue-0.13.0rc2 → iddqueue-0.13.0rc4}/iddqueue/__init__.py

@@ -1,10 +1,12 @@
 from .broker import PostgresBroker
 from .cancellation import ResultCancelled
+from .domains import Domain
 from .rate_limits import PostgresRateLimiterBackend
 from .results import PostgresBackend
 from .schema import generate_coordination_sql, generate_init_sql, generate_upgrade_sql
 
 __all__ = [
+    "Domain",
     "ResultCancelled",
     "PostgresBackend",
     "PostgresBroker",

{iddqueue-0.13.0rc2 → iddqueue-0.13.0rc4}/iddqueue/broker.py

@@ -6,6 +6,7 @@ from itertools import islice
 from queue import Empty, Queue
 from random import randint
 from textwrap import dedent
+from uuid import UUID
 
 from dramatiq.broker import Broker, Consumer, MessageProxy
 from dramatiq.common import compute_backoff, current_millis, dq_name, q_name
@@ -326,15 +327,25 @@ class PostgresConsumer(Consumer):
         # If we have some notifies, loop to find one todo.
         while self.notifies:
             notify = self.notifies.pop(0)
-            payload = json.loads(notify.payload)
-            if payload.get("scan"):
+            try:
+                payload = json.loads(notify.payload)
+                if not isinstance(payload, dict):
+                    continue
+                if payload.get("scan") is not True:
+                    message_id = payload.get("message_id")
+                    if not isinstance(message_id, str):
+                        continue
+                    message_id = str(UUID(message_id))
+            except (ValueError, RecursionError):
+                continue
+            if payload.get("scan") is True:
                 self.notifies += self.fetch_pending_notifies()
                 continue
             # Legacy full payloads are hints too; claim returns durable data.
-            message = Message(self.queue_name, "", (), {}, {}, message_id=payload["message_id"])
+            message = Message(self.queue_name, "", (), {}, {}, message_id=message_id)
             claimed = self.consume_one(message)
             if claimed:
-                self.in_processing.add(claimed.message_id)
+                self.in_processing.add(str(UUID(str(claimed.message_id))))
                 return MessageProxy(claimed)
             else:
                 logger.debug(
@@ -353,13 +364,13 @@ class PostgresConsumer(Consumer):
         # This function is executed in worker thread!
         if getattr(message, "_pg_cancelled", False):
             self.unlock_q.put_nowait(message)
-            self.in_processing.remove(message.message_id)
+            self.in_processing.remove(str(UUID(str(message.message_id))))
             return
         if getattr(message, "_pg_paused", False):
             with transaction(self.pool) as curs:
                 curs.execute(self.queries.DEFER_PAUSED, (message.message_id, message.queue_name))
             self.unlock_q.put_nowait(message)
-            self.in_processing.remove(message.message_id)
+            self.in_processing.remove(str(UUID(str(message.message_id))))
             return
 
         with transaction(self.pool) as curs:
@@ -380,7 +391,7 @@ class PostgresConsumer(Consumer):
                 ),
             )
         self.unlock_q.put_nowait(message)
-        self.in_processing.remove(message.message_id)
+        self.in_processing.remove(str(UUID(str(message.message_id))))
 
     @raise_connection_error
     def auto_purge(self):
@@ -442,7 +453,7 @@ class PostgresConsumer(Consumer):
 
     @raise_connection_error
     def consume_one(self, message):
-        if message.message_id in self.in_processing:
+        if str(UUID(str(message.message_id))) in self.in_processing:
             logger.debug("%s already consumed by self.", message.message_id)
             return
 
@@ -487,7 +498,7 @@ class PostgresConsumer(Consumer):
                 ),
             )
         self.unlock_q.put_nowait(message)
-        self.in_processing.remove(message.message_id)
+        self.in_processing.remove(str(UUID(str(message.message_id))))
 
     @raise_connection_error
     def fetch_pending_notifies(self):
@@ -552,7 +563,7 @@ _max_positive_int = 2**63
 def message_lock(message, *, schema="dramatiq", prefix=""):
     # create sha256 hash from input and create a 64 bit int from it, using
     # 16 hex char. any 16 char range is ok. it takes the center ones
-    global_id = message.queue_name + str(message.message_id)
+    global_id = message.queue_name + str(UUID(str(message.message_id)))
     if schema != "dramatiq" or prefix:
         global_id = storage_namespace(schema, prefix) + global_id
     hex = sha256(global_id.encode("utf-8")).hexdigest()
@@ -592,7 +603,8 @@ QUERIES = QueryManager(
                 mtime = NOW()
             WHERE message_id = %s AND queue_name = %s
             AND state IN ('queued', 'consumed')
-            AND pg_try_advisory_lock(%s)
+            -- Uncorrelated subquery acquires the session lock once per statement.
+            AND (SELECT pg_try_advisory_lock(%s))
             RETURNING message::text;
         """
         ),

iddqueue-0.13.0rc4/iddqueue/domains.py

@@ -0,0 +1,70 @@
+"""Framework-independent declarations, bound explicitly before worker startup."""
+
+import re
+
+from dramatiq import Actor
+from dramatiq.broker import Broker
+
+
+class _DeclarationBroker(Broker):
+    def __init__(self):
+        super().__init__(middleware=[])
+
+    def declare_queue(self, queue_name):
+        # Declarations create no transport resources.
+        pass
+
+    def enqueue(self, message, *, delay=None):
+        raise RuntimeError("Register the actor's Domain with a broker before sending")
+
+
+class Domain:
+    """One queue and qualified actor names; register once per process."""
+
+    def __init__(self, name):
+        if not isinstance(name, str) or not re.fullmatch(r"[a-zA-Z_][a-zA-Z0-9._-]*", name):
+            raise ValueError("domain name must be a valid Dramatiq queue name")
+        self.name = name
+        self._declarations = _DeclarationBroker()
+        self._broker = None
+        self._failed = False
+
+    def actor(self, fn=None, *, actor_name=None, priority=0, **options):
+        def decorate(fn):
+            if self._broker is not None or self._failed:
+                raise RuntimeError("Declare all domain actors before registration")
+            if "broker" in options or "queue_name" in options:
+                raise ValueError("Domain owns the broker and queue_name")
+            name = actor_name if actor_name is not None else fn.__name__
+            if not isinstance(name, str) or not name:
+                raise ValueError("actor_name must be a nonempty string")
+            return Actor(fn, broker=self._declarations, actor_name=f"{self.name}.{name}",
+                         queue_name=self.name, priority=priority, options=dict(options))
+        return decorate if fn is None else decorate(fn)
+
+    def register(self, broker):
+        if self._failed:
+            raise RuntimeError("Registration failed; create a new Domain and broker")
+        if not isinstance(broker, Broker):
+            raise TypeError("register requires a Dramatiq Broker")
+        if broker is self._broker:
+            return
+        if self._broker is not None:
+            raise RuntimeError("Domain is already registered with another broker")
+        actors = list(self._declarations.actors.values())
+        for actor in actors:
+            invalid = set(actor.options) - broker.actor_options
+            if invalid:
+                raise ValueError(f"Undefined actor options for {actor.actor_name}: {sorted(invalid)}")
+            if actor.actor_name in broker.actors:
+                raise ValueError(f"Actor {actor.actor_name!r} is already registered")
+        try:
+            for actor in actors:
+                actor.broker = broker
+                broker.declare_actor(actor)
+        except Exception:
+            for actor in actors:
+                actor.broker = self._declarations
+            self._failed = True
+            raise
+        self._broker = broker

iddqueue-0.13.0rc4/iddqueue/metrics.py

@@ -0,0 +1,115 @@
+"""Queue snapshots; Prometheus is imported only when collecting metrics."""
+
+from psycopg import sql
+
+from .utils import transaction
+
+STATES = ("queued", "consumed", "done", "rejected", "cancelled")
+
+
+def statistics_query(schema="dramatiq", prefix=""):
+    return sql.SQL("""
+        SELECT queue_name,
+            count(*) FILTER (WHERE state = 'queued'),
+            count(*) FILTER (WHERE state = 'consumed'),
+            count(*) FILTER (WHERE state = 'done'),
+            count(*) FILTER (WHERE state = 'rejected'),
+            count(*) FILTER (WHERE state = 'cancelled'),
+            count(*) FILTER (WHERE state = 'queued' AND ready_at <= now()),
+            count(*) FILTER (WHERE state IN ('queued', 'consumed') AND ready_at > now()),
+            coalesce(greatest(0, extract(epoch FROM now() - min(ready_at)
+                FILTER (WHERE state = 'queued' AND ready_at <= now()))), 0)
+        FROM (
+            SELECT *, greatest(mtime,
+                to_timestamp(coalesce((message->'options'->>'eta')::double precision / 1000, 0))) AS ready_at
+            FROM {} WHERE (%s::text IS NULL OR queue_name = %s)
+                AND (%s::text[] IS NULL OR queue_name = ANY(%s::text[]))
+        ) AS messages GROUP BY queue_name ORDER BY queue_name
+    """).format(sql.Identifier(schema, prefix + "queue"))
+
+
+def _queue_statistics(pool, *, schema="dramatiq", prefix="", queue=None, queues=None):
+    with transaction(pool) as cursor:
+        cursor.execute(statistics_query(schema, prefix), (queue, queue, queues, queues))
+        rows = cursor.fetchall()
+    snapshots = [dict(queue=row[0], counts=dict(zip(STATES, row[1:6])),
+                      ready=row[6], scheduled=row[7], oldest_ready_seconds=float(row[8]))
+                 for row in rows]
+    if not snapshots and queue is not None:
+        snapshots.append(dict(queue=queue, counts=dict.fromkeys(STATES, 0),
+                              ready=0, scheduled=0, oldest_ready_seconds=0.0))
+    return snapshots
+
+
+def queue_statistics(pool, *, schema="dramatiq", prefix="", queue=None):
+    return _queue_statistics(pool, schema=schema, prefix=prefix, queue=queue)
+
+
+def domain_statistics(pool, *, domains=None, schema="dramatiq", prefix=""):
+    if domains is not None:
+        if isinstance(domains, str):
+            raise TypeError("domains must be an iterable of domain names, not a string")
+        domains = sorted(set(domains))
+        if any(not isinstance(name, str) or not name or name.endswith(".DQ") for name in domains):
+            raise ValueError("domains must contain nonempty names without the reserved .DQ suffix")
+    queues = None if domains is None else [queue for name in domains for queue in (name, name + ".DQ")]
+    snapshots = {}
+    for row in _queue_statistics(pool, schema=schema, prefix=prefix, queues=queues):
+        domain = row["queue"][:-3] if row["queue"].endswith(".DQ") else row["queue"]
+        result = snapshots.setdefault(domain, dict(domain=domain, counts=dict.fromkeys(STATES, 0),
+                                                   ready=0, scheduled=0, oldest_ready_seconds=0.0))
+        for state in STATES:
+            result["counts"][state] += row["counts"][state]
+        result["ready"] += row["ready"]
+        result["scheduled"] += row["scheduled"]
+        result["oldest_ready_seconds"] = max(result["oldest_ready_seconds"], row["oldest_ready_seconds"])
+    for domain in domains or ():
+        snapshots.setdefault(domain, dict(domain=domain, counts=dict.fromkeys(STATES, 0),
+                                          ready=0, scheduled=0, oldest_ready_seconds=0.0))
+    return [snapshots[name] for name in sorted(snapshots)]
+
+
+class PostgresQueueCollector:
+    """Register in an exporter process; the caller owns the pool."""
+
+    metric_prefix = "iddqueue_queue"
+    label = "queue"
+    snapshot = staticmethod(queue_statistics)
+
+    def __init__(self, pool, *, schema="dramatiq", prefix="", queue=None):
+        self.pool = pool
+        self.options = dict(schema=schema, prefix=prefix, queue=queue)
+
+    def describe(self):
+        # Avoid database access during registry registration.
+        return iter(())
+
+    def collect(self):
+        from prometheus_client.core import GaugeMetricFamily
+
+        counts = GaugeMetricFamily(f"{self.metric_prefix}_messages", "Stored messages by state", labels=[self.label, "state"])
+        ready = GaugeMetricFamily(f"{self.metric_prefix}_ready", "Ready queued messages", labels=[self.label])
+        scheduled = GaugeMetricFamily(f"{self.metric_prefix}_scheduled", "Future delayed messages, including prefetched", labels=[self.label])
+        age = GaugeMetricFamily(f"{self.metric_prefix}_oldest_ready_seconds", "Age of oldest ready queued message", labels=[self.label])
+        for snapshot in self.snapshot(self.pool, **self.options):
+            queue = snapshot[self.label]
+            for state, count in snapshot["counts"].items():
+                counts.add_metric([queue, state], count)
+            ready.add_metric([queue], snapshot["ready"])
+            scheduled.add_metric([queue], snapshot["scheduled"])
+            age.add_metric([queue], snapshot["oldest_ready_seconds"])
+        yield from (counts, ready, scheduled, age)
+
+
+class PostgresDomainCollector(PostgresQueueCollector):
+    """Domain gauges; caller owns the pool, actor imports are not required."""
+
+    metric_prefix = "iddqueue_domain"
+    label = "domain"
+    snapshot = staticmethod(domain_statistics)
+
+    def __init__(self, pool, *, schema="dramatiq", prefix="", domains=None):
+        self.pool = pool
+        self.options = dict(schema=schema, prefix=prefix, domains=None if domains is None else tuple(domains))
+        if isinstance(domains, str):
+            raise TypeError("domains must be an iterable of domain names, not a string")

iddqueue-0.13.0rc4/iddqueue/sqlalchemy.py

@@ -0,0 +1,36 @@
+"""Optional synchronous SQLAlchemy input for caller-owned transactions."""
+
+import psycopg
+from psycopg.pq import TransactionStatus
+from sqlalchemy.engine import Connection
+from sqlalchemy.orm import Session
+
+
+def enqueue_sqlalchemy(broker, message, *, connection, delay=None,
+                       deduplication_key=None, deduplication_ttl=None):
+    """Publish in an already active SQLAlchemy/Psycopg 3 transaction.
+
+    Does not flush, commit, rollback, close or retry the caller's transaction.
+    Multi-bind sessions should pass the intended Connection explicitly.
+    """
+    if isinstance(connection, Session):
+        if not connection.is_active or not connection.in_transaction():
+            raise ValueError("Session requires an active transaction")
+        if connection.bind is None:
+            raise ValueError("Pass an explicit Connection for a multi-bind Session")
+        connection = connection.connection()
+    if not isinstance(connection, Connection):
+        raise TypeError("Expected a synchronous SQLAlchemy Connection or Session")
+    if connection.closed or connection.invalidated or not connection.in_transaction():
+        raise ValueError("Connection requires an active transaction")
+    dialect = connection.engine.dialect
+    if dialect.name != "postgresql" or dialect.driver != "psycopg" or dialect.is_async:
+        raise ValueError("Only synchronous postgresql+psycopg is supported")
+    driver = connection.connection.driver_connection
+    if not isinstance(driver, psycopg.Connection):
+        raise TypeError("Expected a synchronous Psycopg 3 driver connection")
+    if driver.info.transaction_status != TransactionStatus.INTRANS:
+        raise ValueError("Execute SQL or flush explicitly to start the database transaction")
+    return broker.enqueue_in_transaction(message, connection=driver, delay=delay,
+                                         deduplication_key=deduplication_key,
+                                         deduplication_ttl=deduplication_ttl)

{iddqueue-0.13.0rc2 → iddqueue-0.13.0rc4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "iddqueue"
-version = "0.13.0rc2"
+version = "0.13.0rc4"
 description = "Postgres Broker for Dramatiq Task Queue"
 authors = [{name = "Étienne BERSAC"}]
 license = "PostgreSQL"
@@ -21,6 +21,7 @@ Issues = "https://github.com/wa-pis/iddqueue/issues"
 
 [project.optional-dependencies]
 binary = ["psycopg[binary]>=3.3.6,<4"]
+sqlalchemy = ["sqlalchemy>=2.0,<3"]
 monitoring = ["dramatiq[prometheus]>=2.2.1,<3"]
 
 [project.scripts]
@@ -30,7 +31,7 @@ iddqueue = "iddqueue.cli:entrypoint"
 dev = [
     "pytest>=8", "pytest-timeout>=2", "pytest-mock>=3",
     "psycopg[binary]>=3.3.6,<4",
-    "docutils>=0.21", "pygments>=2.20", "ruff>=0.9",
+    "sqlalchemy[asyncio]>=2.0,<3", "docutils>=0.21", "pygments>=2.20", "ruff>=0.9",
 ]
 
 fastapi-example = ["fastapi>=0.115,<1", "httpx>=0.28,<1", "uvicorn>=0.34,<1"]

iddqueue-0.13.0rc2/iddqueue/metrics.py

@@ -1,68 +0,0 @@
-"""Queue snapshots; Prometheus is imported only when collecting metrics."""
-
-from psycopg import sql
-
-from .utils import transaction
-
-STATES = ("queued", "consumed", "done", "rejected", "cancelled")
-
-
-def statistics_query(schema="dramatiq", prefix=""):
-    return sql.SQL("""
-        SELECT queue_name,
-            count(*) FILTER (WHERE state = 'queued'),
-            count(*) FILTER (WHERE state = 'consumed'),
-            count(*) FILTER (WHERE state = 'done'),
-            count(*) FILTER (WHERE state = 'rejected'),
-            count(*) FILTER (WHERE state = 'cancelled'),
-            count(*) FILTER (WHERE state = 'queued' AND ready_at <= now()),
-            count(*) FILTER (WHERE state IN ('queued', 'consumed') AND ready_at > now()),
-            coalesce(greatest(0, extract(epoch FROM now() - min(ready_at)
-                FILTER (WHERE state = 'queued' AND ready_at <= now()))), 0)
-        FROM (
-            SELECT *, greatest(mtime,
-                to_timestamp(coalesce((message->'options'->>'eta')::double precision / 1000, 0))) AS ready_at
-            FROM {} WHERE (%s::text IS NULL OR queue_name = %s)
-        ) AS messages GROUP BY queue_name ORDER BY queue_name
-    """).format(sql.Identifier(schema, prefix + "queue"))
-
-
-def queue_statistics(pool, *, schema="dramatiq", prefix="", queue=None):
-    with transaction(pool) as cursor:
-        cursor.execute(statistics_query(schema, prefix), (queue, queue))
-        rows = cursor.fetchall()
-    snapshots = [dict(queue=row[0], counts=dict(zip(STATES, row[1:6])),
-                      ready=row[6], scheduled=row[7], oldest_ready_seconds=float(row[8]))
-                 for row in rows]
-    if not snapshots and queue is not None:
-        snapshots.append(dict(queue=queue, counts=dict.fromkeys(STATES, 0),
-                              ready=0, scheduled=0, oldest_ready_seconds=0.0))
-    return snapshots
-
-
-class PostgresQueueCollector:
-    """Register in an exporter process; the caller owns the pool."""
-
-    def __init__(self, pool, *, schema="dramatiq", prefix="", queue=None):
-        self.pool = pool
-        self.options = dict(schema=schema, prefix=prefix, queue=queue)
-
-    def describe(self):
-        # Avoid database access during registry registration.
-        return iter(())
-
-    def collect(self):
-        from prometheus_client.core import GaugeMetricFamily
-
-        counts = GaugeMetricFamily("iddqueue_queue_messages", "Stored messages by state", labels=["queue", "state"])
-        ready = GaugeMetricFamily("iddqueue_queue_ready", "Ready queued messages", labels=["queue"])
-        scheduled = GaugeMetricFamily("iddqueue_queue_scheduled", "Future delayed messages, including prefetched", labels=["queue"])
-        age = GaugeMetricFamily("iddqueue_queue_oldest_ready_seconds", "Age of oldest ready queued message", labels=["queue"])
-        for snapshot in queue_statistics(self.pool, **self.options):
-            queue = snapshot["queue"]
-            for state, count in snapshot["counts"].items():
-                counts.add_metric([queue, state], count)
-            ready.add_metric([queue], snapshot["ready"])
-            scheduled.add_metric([queue], snapshot["scheduled"])
-            age.add_metric([queue], snapshot["oldest_ready_seconds"])
-        yield from (counts, ready, scheduled, age)