dbworker 0.0.1__py3-none-any.whl

dbworker-0.0.1.dist-info/METADATA

@@ -0,0 +1,223 @@
+Metadata-Version: 2.4
+Name: dbworker
+Version: 0.0.1
+Summary: Database-backed coordination for process workers.
+License: MIT
+License-File: LICENSE
+Author: Ziyang Song
+Requires-Python: >=3.12,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: sqlalchemy (>=2.0,<3.0)
+Project-URL: Repository, https://github.com/zysilm/dbworker
+Description-Content-Type: text/markdown
+
+# dbworker
+
+Portable database-backed work coordination with SQLAlchemy and process workers. A single-script Redis + Celery alternative that uses your existing database.
+
+Requires Python 3.12+. Install into your application:
+
+```sh
+poetry add /path/to/dbworker
+```
+
+## Quick start
+
+Use your existing SQLAlchemy `session_factory` and database URL:
+
+```python
+from dbworker import Coordinator
+
+coordinator = Coordinator(session_factory, database_url=database_url)
+```
+
+You write a **handler**: a Python function containing the work you want to run. DBWorker runs handlers in separate processes.
+
+Before calling your handler, DBWorker chooses the next task and marks it as “being worked on” in the database. This step is called a **claim**. It lets several processes share the tasks without choosing the same task at the same time.
+
+DBWorker keeps the claim active until your handler finishes. If the process crashes, the claim expires so another process can take over the task.
+
+`source` is the database table that supplies input to your handler, specified as a SQLAlchemy model. It can store user requests, or any entries whose creation should automatically start a procedure. Each new entry gives DBWorker new work to run, and the selected entry is passed to your handler. Its primary key identifies that work so DBWorker can track its completion; the model must have a single primary-key column.
+
+`eligible` controls which inputs can be claimed next. Without it, any new entry can be picked up. Add it when work should wait for a condition or run in a particular order. In the example below, `YourModel` stands for your model. The `enabled` filter and ordering by `id` illustrate a selection rule; replace them with your own conditions and ordering.
+
+Decorate your handler like this. The comments describe where your application logic goes:
+
+**Complete the work in one invocation:**
+
+```python
+from sqlalchemy import select
+from sqlalchemy.orm import Session
+from dbworker import Finished
+
+@coordinator.transactional_worker(
+    name="process",
+    source=YourModel,
+    eligible=lambda: (
+        select(YourModel)
+        .where(YourModel.enabled.is_(True))
+        .order_by(YourModel.id)
+    ),
+    concurrency=4,
+)
+def process(source: YourModel, session: Session) -> Finished:
+    # Your procedure goes here.
+    # Return Finished() when the procedure is complete.
+    return Finished()
+```
+
+- `name` identifies the worker for status queries.
+- `concurrency=4` allows four handlers to run in child processes.
+- `source` is the selected instance of your source model.
+- `session` is a normal SQLAlchemy session supplied by DBWorker. The handler can leave either argument unused.
+
+Before each claim, DBWorker calls `eligible` and uses its query to select the next item. If nothing can be claimed, it waits and checks again. If your application later enables an item, a subsequent check can select it. Finished, failed, and currently claimed items are excluded automatically; you do not need to put those checks in your query. Omitting `eligible` lets DBWorker select from all entries in `source`.
+
+**Process part of the work, then continue in another invocation:**
+
+```python
+from dbworker import Finished, Outcome, Unfinished
+
+@coordinator.transactional_worker(
+    name="process_in_steps",
+    source=YourModel,
+    eligible=lambda: (
+        select(YourModel)
+        .where(YourModel.enabled.is_(True))
+        .order_by(YourModel.id)
+    ),
+    concurrency=4,
+)
+def process_in_steps(source: YourModel, session: Session) -> Outcome:
+    # Your procedure goes here.
+    # Return Unfinished() if it needs another invocation to continue.
+    # Return Finished() instead when it is complete.
+    return Unfinished()
+```
+
+`Finished()` means the work item is complete. `Unfinished()` means this invocation is done, but more work remains. Both save the execution status and commit any writes made through the supplied session; the handler does not have to make any database writes. After `Unfinished()`, DBWorker can invoke the handler again when eligible. Each invocation starts the function from the beginning. Your procedure decides how to continue; DBWorker does not save its position in the function. Use `Outcome` as the return annotation when a handler can return either result.
+
+If the handler raises an exception, DBWorker rolls back its transaction and marks the work failed. If your procedure has prerequisites that can be checked in the database, `eligible` can delay claiming until they are met.
+
+DBWorker owns the handler's commit and session cleanup. Do not commit or close this session yourself. For long computations, you can release a read transaction with `session.rollback()` first. Copy needed values before rollback, and do this before making writes you want to keep.
+
+### Start and stop
+
+Register handlers at module scope in an importable module. In your worker service, initialize DBWorker's tables and start the coordinator:
+
+```python
+if __name__ == "__main__":
+    coordinator.create_worker_tables()
+    coordinator.start()
+```
+
+`start()` returns while workers keep running. When a source entry matches `eligible`, DBWorker can claim its work and call the handler—there is no enqueue call.
+
+On service shutdown, call:
+
+```python
+coordinator.stop()
+```
+
+`stop()` stops new claims and waits for active handlers to finish. Your API can run independently, using the same database. See the [complete example](examples/imagededup_system_dbwork/README.md) for runnable API and worker commands with shutdown handling.
+
+## Track progress and coordinate dependent work
+
+Your application may need to show whether work is running or complete. Another worker may also depend on that information: one procedure prepares something, and a second can begin only after preparation finishes. If preparation fails, the application may need to report that failure instead of continuing.
+
+DBWorker tracks execution separately for each worker and source entry. Access it through the coordinator using the worker's registered name and the source entry's primary key; you do not need to manage or query DBWorker's internal tables yourself.
+
+Use `execution_status()` to get the current status in your API or handler:
+
+```python
+from dbworker import ExecutionStatus
+
+with session_factory() as session:
+    status = coordinator.execution_status(
+        session, worker="process", source_id=source_id,
+    )
+```
+
+This returns `None` before the first claim, or an `ExecutionStatus` enum: `WORKING`, `UNFINISHED`, `FINISHED`, or `FAILED`.
+
+Sometimes you need to select inputs based on the status of their work—for example, list only inputs whose processing has finished. Calling `execution_status()` for each input would mean checking them individually.
+
+`has_execution_status()` lets you include that check in a database query. It returns a SQLAlchemy condition meaning: **“Does this worker have one of these execution statuses for this input?”** It does not run a query or return a Python `True` or `False` when called.
+
+```python
+finished = coordinator.has_execution_status(
+    worker="process",
+    source_id=YourModel.id,
+    statuses=(ExecutionStatus.FINISHED,),
+)
+```
+
+- `worker` names the registered worker whose status you want to check.
+- `source_id` identifies its input. Using `YourModel.id` checks the corresponding input for each entry considered by the query.
+- `statuses` contains the acceptable statuses. The condition matches if any one applies; an input with no execution record does not match.
+
+Use the condition in an ordinary SQLAlchemy query:
+
+```python
+query = select(YourModel).where(finished)
+
+with session_factory() as session:
+    inputs = session.scalars(query).all()
+```
+
+This returns inputs whose work under `process` is finished. The database checks their statuses as part of this query.
+
+The same condition can be combined with other query filters, including in an `eligible` query. Register the named worker before calling `has_execution_status()`.
+
+## Understand failures and retry work
+
+When a handler raises an exception, DBWorker records the error and marks the work as `FAILED`. It will not automatically run that work again. Your application may need to show what went wrong and let someone retry after correcting the cause.
+
+`state()` reads the execution details for one input, giving you more information than its status alone:
+
+```python
+with session_factory() as session:
+    state = coordinator.workers["process"].state(session, source_id)
+    error = state["error"] if state is not None else None
+```
+
+`"process"` is the worker's registered name, and `source_id` is the input's primary key. The result is a mapping containing `execution_status`, the recorded `error`, and `lease_expires_at` (the claim's expiration time). It returns `None` if that worker has never claimed this input. You can use the error to explain the failure in your API or logs.
+
+After correcting the cause, use `reset_failed()` to allow another attempt:
+
+```python
+with session_factory.begin() as session:
+    reset = coordinator.workers["process"].reset_failed(session, source_id)
+```
+
+`reset_failed()` changes a failed execution to `UNFINISHED` and clears its recorded error and claim. It returns `True` if it reset a failed execution, or `False` if there was no failed execution to reset. The `begin()` block commits this change.
+
+Resetting does not call the handler immediately. The running coordinator can claim the work again when it matches `eligible`. The handler starts from the beginning; resetting does not delete application results or progress. If the handler has effects outside the database transaction, make them safe to repeat.
+
+## Configuration
+
+| `Coordinator` argument | Purpose | Default |
+|---|---|---|
+| `session_factory` | SQLAlchemy session factory used for coordination. | Required |
+| `database_url` | Database URL used by child processes; use the same database. | Required |
+| `engine_options` | SQLAlchemy `create_engine()` options for child processes. | `None` |
+| `lease_seconds` | Claim lifetime without renewal. Active claims are renewed automatically. | `30` |
+| `poll_seconds` | Initial wait after finding no claimable work. | `0.25` |
+| `max_poll_seconds` | Maximum wait after exponential backoff. Successful claims continue without waiting. | `10` |
+
+For SQLite, use a file-backed database. Worker names must start with a lowercase letter and contain only lowercase letters, digits and underscores.
+
+## Examples
+
+- [Image deduplication](examples/imagededup_system_dbwork/README.md): independent FastAPI and worker services, artifact building, paged comparisons, and worker dependencies.
+- [Redis + Celery equivalent](examples/imagededup_system_redis_celery/README.md).
+- [Benchmarks](benchmarks/imagededup_benckmark/README.md) with structured JSON results.
+
+## License
+
+[MIT](LICENSE) © 2026 Ziyang Song.
+

dbworker-0.0.1.dist-info/RECORD

@@ -0,0 +1,5 @@
+dbworker.py,sha256=ebmKFpTs6NACY-fP9YrlRgClNhY74L1nP4pysxWxSBA,20878
+dbworker-0.0.1.dist-info/METADATA,sha256=SvXkUZgRrP7xNOwEJNtBTc3OCc4_moAbG4G6QPByqno,11460
+dbworker-0.0.1.dist-info/WHEEL,sha256=EGEvSphFYqXKs23-kQBeyNoJP1nrT8ZJKQoi5p5DYL8,88
+dbworker-0.0.1.dist-info/licenses/LICENSE,sha256=ReFQTfmGpZE0yiEVGrv-i8eSDDemttgydXblWgoB4UM,1068
+dbworker-0.0.1.dist-info/RECORD,,

dbworker-0.0.1.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: poetry-core 2.4.0
+Root-Is-Purelib: true
+Tag: py3-none-any

dbworker-0.0.1.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ziyang Song
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dbworker.py

@@ -0,0 +1,467 @@
+"""Database-owned work, independent of application models and result storage."""
+
+from __future__ import annotations
+
+import logging
+import multiprocessing
+import pickle
+import re
+import threading
+import time
+import uuid
+from collections.abc import Callable, Iterable
+from concurrent.futures import Future, ProcessPoolExecutor
+from dataclasses import dataclass
+from functools import cached_property
+from enum import StrEnum
+from datetime import datetime, timedelta, timezone
+from typing import Any, TypeAlias, TypeVar, cast
+from multiprocessing.util import Finalize
+
+from sqlalchemy import Column, DateTime, Enum, ForeignKey, String, Table, Text, and_, create_engine, event, exists, inspect, insert, or_, select, update
+from sqlalchemy.exc import IntegrityError
+from sqlalchemy.engine import CursorResult, RowMapping, URL, make_url
+from sqlalchemy.orm import DeclarativeBase, Mapper, Session, sessionmaker
+from sqlalchemy.sql import ColumnElement, Select
+from sqlalchemy.sql.selectable import Exists
+
+logger = logging.getLogger(__name__)
+
+
+def now() -> datetime:
+    # Database columns use naive UTC on every supported backend.
+    return datetime.now(timezone.utc).replace(tzinfo=None)
+
+
+class ExecutionStatus(StrEnum):
+    WORKING = "working"
+    UNFINISHED = "unfinished"
+    FINISHED = "finished"
+    FAILED = "failed"
+
+
+@dataclass(frozen=True)
+class Finished:
+    pass
+
+
+@dataclass(frozen=True)
+class Unfinished:
+    pass
+
+
+Outcome: TypeAlias = Finished | Unfinished
+Handler: TypeAlias = Callable[[Any, Session], Outcome]
+_Function = TypeVar("_Function", bound=Callable[..., Any])
+
+
+@dataclass(frozen=True)
+class Claim:
+    source_id: object
+    token: str
+
+
+class LostClaim(Exception):
+    """The result belongs to an execution whose ownership has been replaced."""
+
+
+class _Worker:
+    def __init__(
+        self,
+        *,
+        name: str,
+        source: type[DeclarativeBase],
+        handler: Handler,
+        eligible: Callable[[], Select[tuple[Any]]] | None = None,
+        concurrency: int = 1,
+    ) -> None:
+        if not re.fullmatch(r"[a-z][a-z0-9_]*", name):
+            raise ValueError("Worker name must contain lowercase letters, digits, and underscores")
+        if concurrency < 1:
+            raise ValueError("Worker concurrency must be positive")
+        self.name = name
+        self.source = source
+        self.handler = handler
+        self.eligible = eligible or (lambda: select(source))
+        self.concurrency = concurrency
+        mapper = cast(Mapper[Any], inspect(source))
+        if len(mapper.primary_key) != 1:
+            raise ValueError("Worker sources must have one primary-key column")
+        self.source_key = cast(Column[Any], mapper.primary_key[0])
+        self.source_table = cast(Table, mapper.local_table)
+        metadata = self.source_table.metadata
+        table_name = f"{name}_work"
+        if table_name in metadata.tables:
+            self.table = metadata.tables[table_name]
+            references = list(self.table.c.source_id.foreign_keys)
+            if len(references) != 1 or references[0].column is not self.source_key:
+                raise ValueError(f"Worker table {table_name} references another source")
+        else:
+            self.table = Table(
+                table_name, metadata,
+                Column("source_id", self.source_key.type.copy(), ForeignKey(self.source_key), primary_key=True),
+                Column("execution_status", Enum(
+                    ExecutionStatus, native_enum=False,
+                    values_callable=lambda enum: [member.value for member in enum],
+                    validate_strings=True,
+                ), nullable=False),
+                Column("claim_token", String(36)),
+                Column("lease_expires_at", DateTime, index=True),
+                Column("error", Text),
+            )
+
+    def state(self, session: Session, source_id: object) -> RowMapping | None:
+        return session.execute(select(self.table).where(self.table.c.source_id == source_id)).mappings().first()
+
+    def reset_failed(self, session: Session, source_id: object) -> bool:
+        result = cast(CursorResult[Any], session.execute(
+            update(self.table)
+            .where(self.table.c.source_id == source_id, self.table.c.execution_status == ExecutionStatus.FAILED)
+            .values(execution_status=ExecutionStatus.UNFINISHED, error=None, claim_token=None, lease_expires_at=None)
+        ))
+        return result.rowcount == 1
+
+
+@dataclass(frozen=True)
+class _WorkerDefinition:
+    """Importable callables and model classes, never live database resources."""
+
+    name: str
+    source: type[DeclarativeBase]
+    handler: Handler
+
+
+_process_worker: _Worker | None = None
+_process_session_factory: sessionmaker[Session] | None = None
+
+
+def _initialize_process(
+    database_url: str | URL,
+    engine_options: dict[str, Any],
+    definitions: tuple[_WorkerDefinition, ...],
+    worker_name: str,
+) -> None:
+    global _process_worker, _process_session_factory
+    engine = create_engine(database_url, **engine_options)
+    _process_session_factory = sessionmaker(engine, expire_on_commit=False)
+    Finalize(None, engine.dispose, exitpriority=10)
+    # Rebuild registered work-table metadata locally, including dependencies used
+    # by application queries. Eligibility callbacks stay in the parent process.
+    for definition in definitions:
+        worker = _Worker(name=definition.name, source=definition.source, handler=definition.handler)
+        if definition.name == worker_name:
+            _process_worker = worker
+
+
+def _execute_in_process(claim: Claim) -> Outcome:
+    if _process_worker is None or _process_session_factory is None:
+        raise RuntimeError("Handler process has not been initialized")
+    return _execute_claim(_process_worker, claim, _process_session_factory)
+
+
+def _execute_claim(worker: _Worker, claim: Claim, session_factory: sessionmaker[Session]) -> Outcome:
+    """Run a whole handler and atomically persist its writes and outcome."""
+    with session_factory() as session:
+        def prevent_handler_commit(session: Session) -> None:
+            raise RuntimeError("The worker commits the handler session; return Finished() or Unfinished() instead")
+
+        event.listen(session, "before_commit", prevent_handler_commit)
+        try:
+            source = session.get(worker.source, claim.source_id)
+            if source is None:
+                raise ValueError("Worker source no longer exists")
+            outcome = worker.handler(source, session)
+            if not isinstance(outcome, (Finished, Unfinished)):
+                raise TypeError("A worker handler must return Finished() or Unfinished()")
+            # All handler writes, including already-flushed SQL, are still in
+            # this transaction. Rejecting ownership rolls them all back.
+            with session.no_autoflush:
+                result = cast(CursorResult[Any], session.execute(
+                    update(worker.table).where(
+                        worker.table.c.source_id == claim.source_id,
+                        worker.table.c.execution_status == ExecutionStatus.WORKING,
+                        worker.table.c.claim_token == claim.token,
+                    ).values(execution_status=ExecutionStatus.FINISHED if isinstance(outcome, Finished) else ExecutionStatus.UNFINISHED,
+                             claim_token=None, lease_expires_at=None, error=None)
+                ))
+            if result.rowcount != 1:
+                raise LostClaim()
+        finally:
+            event.remove(session, "before_commit", prevent_handler_commit)
+        session.commit()
+        return outcome
+
+
+class Coordinator:
+    """Parent-process claiming, scheduling and lease renewal.
+
+    Entire handlers execute in spawned processes with their own engines/sessions.
+    database_url and engine_options configure those independent child engines.
+    """
+
+    def __init__(
+        self,
+        session_factory: sessionmaker[Session],
+        *,
+        database_url: str | URL,
+        engine_options: dict[str, Any] | None = None,
+        lease_seconds: float = 30,
+        poll_seconds: float = 0.25,
+        max_poll_seconds: float = 10,
+    ) -> None:
+        if lease_seconds <= 0 or poll_seconds <= 0:
+            raise ValueError("Lease and poll intervals must be positive")
+        if max_poll_seconds < poll_seconds:
+            raise ValueError("Maximum poll interval must be at least the initial interval")
+        self.database_url = database_url
+        self.engine_options = dict(engine_options or {})
+        self.session_factory = session_factory
+        self.lease_seconds = lease_seconds
+        self.poll_seconds = poll_seconds
+        self.max_poll_seconds = max_poll_seconds
+        self.workers: dict[str, _Worker] = {}
+        self._running: dict[str, tuple[threading.Thread, ProcessPoolExecutor]] = {}
+        self._stop = threading.Event()
+        self._wakeups: dict[str, threading.Event] = {}
+
+    @cached_property
+    def _supports_skip_locked(self) -> bool:
+        # Resolve on the first claim so importing decorated handlers in spawned
+        # processes does not open a parent-coordinator database connection.
+        with self.session_factory() as session:
+            dialect = session.connection().dialect
+            version = dialect.server_version_info or ()
+            return (
+                dialect.name == "postgresql" and version >= (9, 5)
+                or dialect.name == "mysql" and not getattr(dialect, "is_mariadb", False) and version >= (8, 0, 1)
+                or dialect.name in ("mysql", "mariadb") and getattr(dialect, "is_mariadb", False) and version >= (10, 6)
+            )
+
+    def transactional_worker(
+        self,
+        *,
+        name: str,
+        source: type[DeclarativeBase],
+        eligible: Callable[[], Select[tuple[Any]]] | None = None,
+        concurrency: int = 1,
+    ) -> Callable[[_Function], _Function]:
+        """Register a handler whose session and final commit belong to the runtime.
+
+        Registration creates worker metadata, but starts no processing. The
+        original function is returned unchanged; direct calls are not managed.
+        """
+        def decorate(handler: _Function) -> _Function:
+            if self._running:
+                raise RuntimeError("Register workers before starting the coordinator")
+            if name in self.workers:
+                raise ValueError(f"Duplicate worker name: {name}")
+            self.workers[name] = _Worker(
+                name=name, source=source, handler=handler,
+                eligible=eligible, concurrency=concurrency,
+            )
+            return handler
+
+        return decorate
+
+    def execution_status(
+        self, session: Session, *, worker: str, source_id: object,
+    ) -> ExecutionStatus | None:
+        """Return the recorded execution status, or None if never claimed.
+
+        Lease expiry does not change the recorded status. Unknown worker names
+        raise KeyError, rather than being confused with an unclaimed source.
+        """
+        table = self.workers[worker].table
+        value = session.scalar(select(table.c.execution_status).where(table.c.source_id == source_id))
+        return ExecutionStatus(value) if value is not None else None
+
+    def has_execution_status(
+        self, *, worker: str, source_id: object, statuses: Iterable[ExecutionStatus],
+    ) -> Exists:
+        """Build a SQL predicate for a source ID or SQLAlchemy source-ID expression.
+
+        An unclaimed source matches no status; an empty status set matches none.
+        This only constructs SQL and does not open a database connection.
+        """
+        table = self.workers[worker].table
+        return exists().where(
+            table.c.source_id == source_id,
+            table.c.execution_status.in_(tuple(statuses)),
+        ).correlate_except(table)
+
+    def create_worker_tables(self) -> None:
+        with self.session_factory() as session:
+            for worker in self.workers.values():
+                worker.table.create(session.get_bind(), checkfirst=True)
+
+    def _available(self, table: Table, timestamp: datetime) -> ColumnElement[bool]:
+        return or_(
+            table.c.execution_status == ExecutionStatus.UNFINISHED,
+            and_(table.c.execution_status == ExecutionStatus.WORKING, table.c.lease_expires_at < timestamp),
+        )
+
+    def _candidate(self, worker: _Worker, timestamp: datetime) -> Select[tuple[Any]]:
+        table = worker.table
+        # Keep availability separate from application eligibility. Joining work
+        # state onto the eligibility query can make SQLite evaluate expensive
+        # correlated predicates for every finished source before rejecting it.
+        available_ids = (
+            select(worker.source_key)
+            .outerjoin(table, table.c.source_id == worker.source_key)
+            .where(or_(table.c.source_id.is_(None), self._available(table, timestamp)))
+        )
+        return (
+            worker.eligible()
+            .with_only_columns(worker.source_key, maintain_column_froms=True)
+            .where(worker.source_key.in_(available_ids))
+            .limit(1)
+        )
+
+    def _record_claim(
+        self, session: Session, worker: _Worker, source_id: object | None, timestamp: datetime,
+    ) -> Claim | None:
+        if source_id is None:
+            return None
+        table = worker.table
+        token = str(uuid.uuid4())
+        values: dict[str, object] = dict(execution_status=ExecutionStatus.WORKING, claim_token=token,
+                      lease_expires_at=now() + timedelta(seconds=self.lease_seconds), error=None)
+        existing = session.scalar(select(table.c.source_id).where(table.c.source_id == source_id))
+        if existing is None:
+            # The primary key arbitrates simultaneous first claims. A savepoint
+            # lets us inspect a uniqueness conflict without poisoning the session.
+            try:
+                with session.begin_nested():
+                    session.execute(insert(table).values(source_id=source_id, **values))
+            except IntegrityError as exc:
+                code = getattr(exc.orig, "sqlite_errorcode", None)
+                sqlstate = getattr(exc.orig, "sqlstate", None) or getattr(exc.orig, "pgcode", None)
+                mysql_code = exc.orig.args[0] if exc.orig is not None and exc.orig.args else None
+                if code not in (1555, 2067) and sqlstate != "23505" and mysql_code != 1062:
+                    raise
+                return None
+        else:
+            result = cast(CursorResult[Any], session.execute(update(table).where(
+                table.c.source_id == source_id, self._available(table, timestamp),
+            ).values(**values)))
+            if result.rowcount != 1:
+                return None
+        return Claim(source_id, token)
+
+    def _claim_with_db_lock(self, worker: _Worker) -> Claim | None:
+        timestamp = now()
+        with self.session_factory.begin() as session:
+            # Lock the source row, which exists even before the first work row.
+            source_id = session.scalar(self._candidate(worker, timestamp).with_for_update(
+                skip_locked=True, of=worker.source_table,
+            ))
+            return self._record_claim(session, worker, source_id, timestamp)
+
+    def _claim_with_conditional_update(self, worker: _Worker) -> Claim | None:
+        timestamp = now()
+        with self.session_factory.begin() as session:
+            source_id = session.scalar(self._candidate(worker, timestamp))
+            return self._record_claim(session, worker, source_id, timestamp)
+
+    def claim(self, worker: _Worker) -> Claim | None:
+        if self._supports_skip_locked:
+            return self._claim_with_db_lock(worker)
+        return self._claim_with_conditional_update(worker)
+
+    def renew(self, worker: _Worker, claims: Iterable[Claim]) -> None:
+        table = worker.table
+        with self.session_factory.begin() as session:
+            for claim in claims:
+                session.execute(update(table).where(
+                    table.c.source_id == claim.source_id, table.c.execution_status == ExecutionStatus.WORKING,
+                    table.c.claim_token == claim.token,
+                ).values(lease_expires_at=now() + timedelta(seconds=self.lease_seconds)))
+
+    def _fail(self, worker: _Worker, claim: Claim, error: Exception) -> None:
+        table = worker.table
+        with self.session_factory.begin() as session:
+            session.execute(update(table).where(
+                table.c.source_id == claim.source_id, table.c.execution_status == ExecutionStatus.WORKING,
+                table.c.claim_token == claim.token,
+            ).values(execution_status=ExecutionStatus.FAILED, error=str(error), claim_token=None, lease_expires_at=None))
+
+    def _run(
+        self, worker: _Worker, handlers: ProcessPoolExecutor,
+        wakeup: threading.Event,
+    ) -> None:
+        active: dict[Future[Outcome], Claim] = {}
+        next_renewal = time.monotonic()
+        next_claim_at = 0.0
+        delay = self.poll_seconds
+        while not self._stop.is_set() or active:
+            # Clear before inspecting futures so a completion during this iteration
+            # still wakes the wait below.
+            wakeup.clear()
+            for future in list(active):
+                if future.done():
+                    claim = active.pop(future)
+                    try:
+                        future.result()
+                    except LostClaim:
+                        logger.info("Discarded stale result for %s:%s", worker.name, claim.source_id)
+                    except Exception as exc:
+                        logger.exception("Worker %s failed for %s", worker.name, claim.source_id)
+                        self._fail(worker, claim, exc)
+            if time.monotonic() >= next_renewal:
+                if active:
+                    self.renew(worker, active.values())
+                next_renewal = time.monotonic() + self.lease_seconds / 3
+            if not self._stop.is_set() and time.monotonic() >= next_claim_at:
+                while len(active) < worker.concurrency and not self._stop.is_set():
+                    next_claim = self.claim(worker)
+                    if next_claim is None:
+                        next_claim_at = time.monotonic() + delay
+                        delay = min(delay * 2, self.max_poll_seconds)
+                        break
+                    delay = self.poll_seconds
+                    next_claim_at = 0.0
+                    future = handlers.submit(_execute_in_process, next_claim)
+                    active[future] = next_claim
+                    future.add_done_callback(lambda completed: wakeup.set())
+            deadlines: list[float] = []
+            if active:
+                deadlines.append(next_renewal)
+            if not self._stop.is_set() and len(active) < worker.concurrency:
+                deadlines.append(next_claim_at)
+            if not deadlines:
+                break
+            wakeup.wait(max(0.0, min(deadlines) - time.monotonic()))
+
+    def start(self) -> None:
+        if self._running:
+            raise RuntimeError("Coordinator already started")
+        url = make_url(self.database_url)
+        if url.get_backend_name() == "sqlite" and (url.database in (None, "", ":memory:") or url.query.get("mode") == "memory"):
+            raise ValueError("Handler processes require file-backed SQLite, not an in-memory database")
+        definitions = tuple(_WorkerDefinition(w.name, w.source, w.handler) for w in self.workers.values())
+        try:
+            pickle.dumps((self.database_url, self.engine_options, definitions))
+        except (TypeError, AttributeError, pickle.PicklingError) as exc:
+            raise ValueError("Handlers and models must be importable; bound handler arguments and engine options must be serializable") from exc
+        self._stop.clear()
+        for worker in self.workers.values():
+            handlers = ProcessPoolExecutor(
+                max_workers=worker.concurrency, mp_context=multiprocessing.get_context("spawn"),
+                initializer=_initialize_process,
+                initargs=(self.database_url, self.engine_options, definitions, worker.name),
+            )
+            wakeup = threading.Event()
+            self._wakeups[worker.name] = wakeup
+            thread = threading.Thread(target=self._run, args=(worker, handlers, wakeup), name=worker.name, daemon=True)
+            self._running[worker.name] = (thread, handlers)
+            thread.start()
+
+    def stop(self) -> None:
+        self._stop.set()
+        for wakeup in self._wakeups.values():
+            wakeup.set()
+        for thread, handlers in self._running.values():
+            thread.join()
+            handlers.shutdown(wait=True)
+        self._running.clear()
+        self._wakeups.clear()