cdt-kit 0.1.0__py3-none-any.whl

cdt_kit-0.1.0.dist-info/METADATA

@@ -0,0 +1,180 @@
+Metadata-Version: 2.5
+Name: cdt-kit
+Version: 0.1.0
+Summary: Server-side app kit of the Controlled Delivery Toolkit: config, migrations, jobs and HTTP basics
+Project-URL: Source, https://github.com/fromerobr-ai/cdt-kit
+Project-URL: Changelog, https://github.com/fromerobr-ai/cdt-kit/blob/main/CHANGELOG.md
+Author: fromerobr-ai
+Classifier: Framework :: FastAPI
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Requires-Python: >=3.11
+Requires-Dist: psycopg<4,>=3.2
+Provides-Extra: fastapi
+Requires-Dist: fastapi>=0.115; extra == 'fastapi'
+Provides-Extra: jobs
+Requires-Dist: procrastinate<4,>=3.10; extra == 'jobs'
+Provides-Extra: test
+Requires-Dist: pytest-asyncio>=0.24; extra == 'test'
+Requires-Dist: pytest>=8; extra == 'test'
+Description-Content-Type: text/markdown
+
+# cdt-kit
+
+The server-side app kit of the Controlled Delivery Toolkit (CDT): a Python library every app pins for what the toolkit's
+standard asks of its request path, so projects stop rebuilding it (claude-deploy-toolkit #422). Distribution `cdt-kit`
+on PyPI, import name `cdtkit` (PyPI's `cdt` is another project's module).
+
+0.1.0 holds what the pilot, magic-minutes, needs first:
+
+| Module | What it does |
+|---|---|
+| `cdtkit.config` | `*_FILE` secrets, the database from `DATABASE_*`, `ENV_CLASS`, `CDT_LANGUAGE`, `TOOLKIT_JOB_TYPES` |
+| `cdtkit.migrations` | forward-only SQL files at start (advisory lock, `schema_migrations`), the expand-only lint |
+| `cdtkit.jobs` | procrastinate on the app's Postgres: one queue per job type, backoff with jitter, `NoRow` (`skipped=no-row`), the final-failure hook |
+| `cdtkit.jobs.worker` | the worker process: one worker per queue, heartbeat health check, graceful stop |
+| `cdtkit.jobs.recovery` | dead-worker and stuck-job recovery |
+| `cdtkit.jobs.prune` | finished jobs removed after `failed_retention` (run from a declared schedule) |
+| `cdtkit.http` | logfmt logging on stdout, the `/healthz` checks |
+| `cdtkit.fastapi` | the FastAPI adapter: `health_router` |
+| `cdtkit.testing` | test stand-ins: a Postgres (`cdt_postgres`, `cdt_database` fixtures) and the in-memory queue |
+
+Identity, the account contract, legal, i18n and consent come in 0.2.0; `/metrics`, RFC 9457 problems, rate classes and
+Idempotency-Key later (see #422).
+
+## Install and pin
+
+```
+cdt-kit[jobs,fastapi]==0.1.0      # requirements.txt of an app; [test] for its test environment
+```
+
+The core depends only on `psycopg` 3 (choose its C build in the app: `psycopg[binary]`); `jobs` adds procrastinate
+3.x, `fastapi` the adapter, `test` pytest and the fixtures. Python 3.11 or newer.
+
+**Production pins only stable kit versions.** Every deployed kit candidate is published to PyPI at once, so its pilot
+can test it before promotion; a version is stable once promoted (`RELEASES.md` of this repository, the GitHub Release
+marked latest). An app may pin a candidate on staging; its production deployments pin a stable version. The toolkit's
+rule `kit-version` (planned, claude-deploy-toolkit #423) will check it; until then it is a rule each app keeps.
+
+Versions are semantic: a minor adds, a major breaks; anything deprecated keeps working for two minors with a warning.
+At most one kit release per toolkit minor.
+
+## Use
+
+Configuration, read once at start:
+
+```python
+from cdtkit import config
+
+db = config.database()                      # DATABASE_HOST/_PORT/_NAME/_USER + DATABASE_PASSWORD_FILE
+conninfo = db.conninfo("web")               # libpq string; repr(db) never shows the password
+key = config.read_secret("KEYS_ENCRYPTION_KEY")          # KEYS_ENCRYPTION_KEY_FILE, else the variable (local runs)
+optional = config.read_secret("SMTP_PASSWORD", required=False)   # "" when unset or an empty file
+env_class, language = config.env_class(), config.language()     # "staging"/"production", CDT_LANGUAGE
+job_types = config.job_types(service="worker")            # this worker's TOOLKIT_JOB_TYPES entry: {id: params}
+```
+
+Missing or invalid configuration raises `config.ConfigError` (a `SystemExit`: the process stops with one line).
+
+Migrations at start, in every process (web and workers may start together; one migrates, the others wait):
+
+```python
+from cdtkit import migrations
+migrations.migrate(conninfo, "app/migrations", queue_app=queue_app)   # *.sql in name order, each once
+```
+
+Each file runs in its own transaction and is recorded in `schema_migrations (name, applied_at)`; with a procrastinate
+app, its schema is applied when missing. The lint reports contracting statements (drops, renames, type changes, new
+`NOT NULL`, `UNIQUE` or `CHECK` on existing data):
+
+```
+python -m cdtkit.migrations lint app/migrations [--after 0007_accounts_ui.sql]    # exit 1 on findings
+```
+
+A contract belongs in a cleanup release that declares it in `deploy/package.json` `"contracts"`; mark its line
+`-- cdt: contract <reason>` and the lint accepts it.
+
+Jobs (one queue per job type; parameters from `deploy/workers.json` through `TOOLKIT_JOB_TYPES`):
+
+```python
+from cdtkit import jobs
+from cdtkit.jobs import worker
+
+DEFAULTS = {"convert": {"concurrency": 1, "max_attempts": 3, "backoff": "60s", "timeout_item": "3h"}}
+
+async def convert(meeting_id):
+    meeting = load(meeting_id)
+    if meeting is None:
+        raise jobs.NoRow("meeting gone")    # deleted while queued: the job ends, logged skipped=no-row, no retry
+    ...
+
+app = jobs.make_app(conninfo, job_types, {"convert": convert}, DEFAULTS, pool_max=2, on_final_failure=mark_failed)
+await jobs.defer_async(app, "convert", meeting_id=7)          # web processes defer; payloads carry IDs only
+await worker.run(app, ["convert"], job_types, DEFAULTS, ping=db_ping, on_given_up=mark_failed,
+                 never_give_up=("account-delete",))
+```
+
+- `max_attempts` counts runs in total (at most 25); the wait after the n-th failure is `backoff * 2^(n-1)`, capped at 6
+  hours, shortened by up to half at random (jitter).
+- `on_final_failure(job_id, payload)` runs when the last attempt raises; `on_given_up` when the recovery gives a job up
+  (its worker died with no attempt left, or it ran past `timeout_item` plus 5 minutes). Both mark what the user sees.
+- The worker's container health check: `python -m cdtkit.jobs.worker --health` (exit 0 while the heartbeat in
+  `/tmp/worker-heartbeat` is younger than 80 seconds; the heartbeat runs `ping` first).
+- Finished jobs: successful ones are deleted at once; failed, cancelled and aborted ones after the entry's
+  `failed_retention` by `python -m cdtkit.jobs.prune --service worker`, which the app declares as a schedule
+  (`deploy/schedules.json`), never a loop in its code.
+
+HTTP basics:
+
+```python
+from cdtkit.http import Health
+from cdtkit.http.logging import setup
+from cdtkit.fastapi import health_router
+
+setup()                                                        # logfmt on stdout, one line per event
+app.include_router(health_router(Health(database=db.ping)))    # GET /healthz: 200 "ok" or 503 "unavailable: database"
+```
+
+Tests: with `cdt-kit[test]` installed the fixtures are there without imports:
+
+```python
+def test_store(cdt_database):                 # a fresh empty database (a cdtkit.config.Database)
+    migrations.migrate(cdt_database.conninfo(), "app/migrations")
+```
+
+They use the server named by `CDT_TEST_DATABASE_HOST` (`_PORT`, `_USER`, `_PASSWORD`, `_NAME`), else start a
+throwaway cluster with the local `initdb`/`pg_ctl` (not as root), else skip. `cdtkit.testing.queue.memory_app`,
+`drain` and `statuses` run job handlers on procrastinate's in-memory connector.
+
+## Layout
+
+```
+src/cdtkit/            the library (docs/PROJECT_NOTES.md has the module map)
+tests/                 the test suite (pytest; Postgres tests use the fixtures above)
+selftest/              the self-test stack's run script and the testdb sidecar's script
+compose.yaml           the self-test stack (below); Dockerfile its image
+requirements-selftest.txt   pinned dependencies of the self-test image (arm64 wheels checked)
+.github/workflows/publish.yml   publishing to PyPI
+```
+
+## The self-test stack
+
+The toolkit deploys Compose stacks, and a library has no service, so cdt-kit's stack proves the kit on each server that
+deploys it: `selftest` builds the wheel from the tree, installs it with the pinned dependencies and runs the test suite
+once per build against `testdb`, a throwaway Postgres 18 on an internal network that stops (exit 0) after the run.
+`selftest` then idles, healthy only when the tests passed, so a failing suite fails the deploy. The result is kept in
+`data/selftest/` per build: a restart does not rerun the tests. It is a workaround until the toolkit offers a
+check-only kind (claude-project-common#176); its two conformance exceptions point there.
+
+Run it by hand on the primary: `cd /opt/stacks/cdt-kit && docker compose logs selftest`.
+
+## Publishing
+
+`.github/workflows/publish.yml` publishes the wheel and the sdist to PyPI when the servers create the version's GitHub
+Release, that is when the candidate is deployed on the primary; PyPI Trusted Publishing, no stored token. The one-time
+setup on pypi.org is in `RUNBOOK.md`. The workflow's result shows on the release issue. A version that failed to
+publish is published again with the workflow's "Run workflow" button and its tag.
+
+Running the tests locally: `pip install -e .[jobs,fastapi,test]` then `pytest` (Postgres binaries on PATH or
+`CDT_TEST_DATABASE_HOST`).

cdt_kit-0.1.0.dist-info/RECORD

@@ -0,0 +1,18 @@
+cdtkit/__init__.py,sha256=V42QnqEHQGbySWrPI0Q3UiOg8xTvb_4yEuVyAhWWjPA,445
+cdtkit/config.py,sha256=zlGKnLchDbNISI54rG6XSwBcmeXj7xM1NL_yqJTt1Ic,5017
+cdtkit/fastapi.py,sha256=fxkfpmgqG9rNYbAkP6MAycWH3ft1CblINlwOX5Rvi6E,1261
+cdtkit/migrations.py,sha256=wGusSx8yqB6KjZMOHWljLn7jfDGypRw5HRuRQzxx3Es,8329
+cdtkit/http/__init__.py,sha256=xWC1jepmCg1ghJbPoMojGpRORtfXHUT7xsHc2yOSFz4,1529
+cdtkit/http/logging.py,sha256=-Yb5sTMdU_CAmuxR6EOB5Iz-kABUT5m2_joq00sx9tc,2433
+cdtkit/jobs/__init__.py,sha256=3E2y6TpihQCX3u1hQZlUaeRFzYBFzAhKQ4Hw6WliAoY,6344
+cdtkit/jobs/prune.py,sha256=g8UF32uWVe70IgBIw1uUhjJsvdtRCKPUDp41Qi4jDBU,2382
+cdtkit/jobs/recovery.py,sha256=USjfPRyU6S8BEHEBomZSZNy9Pvwvl-_51f_ULaYJT_I,5902
+cdtkit/jobs/worker.py,sha256=vLeqVGxF-POwysfeb8F0o6qyUNv-BcZpcydS4NSZA8A,4459
+cdtkit/testing/__init__.py,sha256=araX4XyMO3wvDZVSLAEcsekvDTGpyVhuvCU12JQPBeU,481
+cdtkit/testing/plugin.py,sha256=RNSOlMd7lPv7qzZTOgHplA1INek9rdClJNh7nSI-zWU,600
+cdtkit/testing/postgres.py,sha256=JQVLr1hbTTMF87TvgUaz5dTIJe91tCf8UP9iHMFiqBA,4133
+cdtkit/testing/queue.py,sha256=aC6y5lOft0ZSre45NCpEU6NcmY90vEPUUJNewijy3V0,1447
+cdt_kit-0.1.0.dist-info/METADATA,sha256=8g07r29ch162yTh2ybqFAiLPHIr3g-KjzEQ_GsJwdQc,9443
+cdt_kit-0.1.0.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
+cdt_kit-0.1.0.dist-info/entry_points.txt,sha256=kUsYjhFKWHOStFMm5Kd3bHoXrBXu-aWY740Y6F9Xj-8,42
+cdt_kit-0.1.0.dist-info/RECORD,,

cdt_kit-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.33.0
+Root-Is-Purelib: true
+Tag: py3-none-any

cdt_kit-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[pytest11]
+cdtkit = cdtkit.testing.plugin

cdtkit/__init__.py

@@ -0,0 +1,7 @@
+"""cdt-kit: the server-side app kit of the Controlled Delivery Toolkit (CDT).
+
+Modules (0.1.0): `cdtkit.config` (environment and `*_FILE` secrets), `cdtkit.migrations` (forward-only SQL files),
+`cdtkit.jobs` (procrastinate on the app's Postgres), `cdtkit.http` (logfmt logging and the health check) and the
+FastAPI adapter `cdtkit.fastapi`; `cdtkit.testing` holds the test stand-ins. The core imports no web framework.
+"""
+__version__ = "0.1.0"

cdtkit/config.py

@@ -0,0 +1,118 @@
+"""Configuration the toolkit gives every app: read once at start from the environment and the secret files.
+
+Values come from `config/defaults.env`, `.toolkit/vars.env` (rendered by the toolkit) and `.env`, through Compose's
+`env_file`; every secret from the file named by `<NAME>_FILE` (the toolkit mounts `.toolkit/secrets/<NAME>`). The
+database is given in parts (`DATABASE_HOST`, `DATABASE_PORT`, `DATABASE_NAME`, `DATABASE_USER`, the password in
+`DATABASE_PASSWORD_FILE`), never as a URL (standard, chapter "database").
+"""
+import json
+import os
+from dataclasses import dataclass
+
+ENV_CLASSES = ("staging", "production")
+
+
+class ConfigError(SystemExit):
+    """A configuration the app cannot start with: the process stops with this message (no traceback)."""
+
+
+def _env(env):
+    return os.environ if env is None else env
+
+
+def read_secret(name, env=None, required=True):
+    """The value of secret NAME: the file named by NAME_FILE (toolkit secrets), else NAME itself (local runs, tests).
+
+    Surrounding whitespace is removed. An empty value counts as unset: an optional secret with no value on a server
+    arrives as an empty file (toolkit 2.111.0). Returns "" for an unset optional secret; raises ConfigError for an
+    unset required one or an unreadable file.
+    """
+    env = _env(env)
+    path = env.get(name + "_FILE")
+    if path:
+        try:
+            with open(path, encoding="utf-8") as f:
+                value = f.read().strip()
+        except OSError as e:
+            raise ConfigError("%s: cannot read %s (%s)" % (name, path, e.strerror))
+    else:
+        value = (env.get(name) or "").strip()
+    if not value and required:
+        raise ConfigError("%s: missing or empty (written by the toolkit into .toolkit/secrets/)" % name)
+    return value
+
+
+@dataclass(frozen=True)
+class Database:
+    host: str
+    port: int
+    name: str
+    user: str
+    password: str
+
+    def conninfo(self, application_name=""):
+        """A libpq connection string; values are quoted, so a password with spaces or quotes stays one value."""
+        parts = [("host", self.host), ("port", str(self.port)), ("dbname", self.name), ("user", self.user),
+                 ("password", self.password)]
+        if application_name:
+            parts.append(("application_name", application_name))
+        return " ".join("%s=%s" % (k, _quote(v)) for k, v in parts)
+
+    def __repr__(self):   # never the password in a log line or a traceback
+        return "Database(host=%r, port=%d, name=%r, user=%r)" % (self.host, self.port, self.name, self.user)
+
+
+def _quote(value):
+    if value and not any(c in value for c in " '\\"):
+        return value
+    return "'" + value.replace("\\", "\\\\").replace("'", "\\'") + "'"
+
+
+def database(env=None):
+    """The app's database from DATABASE_* and the password file; ConfigError names every missing variable."""
+    env = _env(env)
+    missing = [n for n in ("DATABASE_HOST", "DATABASE_NAME", "DATABASE_USER") if not (env.get(n) or "").strip()]
+    if missing:
+        raise ConfigError("missing configuration: %s" % ", ".join(missing))
+    raw_port = (env.get("DATABASE_PORT") or "").strip() or "5432"
+    if not raw_port.isdigit():
+        raise ConfigError("DATABASE_PORT must be a port number")
+    return Database(env["DATABASE_HOST"].strip(), int(raw_port), env["DATABASE_NAME"].strip(),
+                    env["DATABASE_USER"].strip(), read_secret("DATABASE_PASSWORD", env))
+
+
+def env_class(env=None, default="staging"):
+    """ENV_CLASS: "staging" or "production" (the toolkit writes it); `default` when unset (local runs, tests)."""
+    value = (_env(env).get("ENV_CLASS") or "").strip() or default
+    if value not in ENV_CLASSES and value != default:
+        raise ConfigError("ENV_CLASS must be one of %s" % ", ".join(ENV_CLASSES))
+    return value
+
+
+def is_production(env=None):
+    return env_class(env) == "production"
+
+
+def language(env=None):
+    """CDT_LANGUAGE, the deployment's default language (a BCP 47 tag); "" means the app's own default."""
+    return (_env(env).get("CDT_LANGUAGE") or "").strip()
+
+
+def worker_entry(env=None, service="worker"):
+    """This worker service's entry of TOOLKIT_JOB_TYPES (deploy/workers.json, rendered by the toolkit):
+    {"queue", "failed_retention", "job_types": {id: params}}; {} when the variable is unset or names no such service."""
+    raw = (_env(env).get("TOOLKIT_JOB_TYPES") or "").strip()
+    if not raw:
+        return {}
+    try:
+        entry = json.loads(raw).get(service) or {}
+    except (ValueError, AttributeError):
+        raise ConfigError("TOOLKIT_JOB_TYPES is not valid JSON")
+    if not isinstance(entry, dict) or not isinstance(entry.get("job_types", {}), (dict, type(None))):
+        raise ConfigError("TOOLKIT_JOB_TYPES: the entry of %s is not an object" % service)
+    return entry
+
+
+def job_types(env=None, service="worker"):
+    """The worker service's job types from TOOLKIT_JOB_TYPES: {id: params}; {} when unset."""
+    return worker_entry(env, service).get("job_types", {}) or {}

cdtkit/fastapi.py

@@ -0,0 +1,35 @@
+"""The FastAPI adapter (the starter's framework): the health route of `cdtkit.http`.
+
+    from cdtkit.fastapi import health_router
+    app.include_router(health_router(Health(database=db.ping)))
+
+GET /healthz answers 200 "ok" or 503 "unavailable: <checks>", plain text, never cached, outside the OpenAPI schema.
+Sync checks run in the thread pool, so a slow database never blocks the event loop.
+"""
+from fastapi import APIRouter
+from fastapi.concurrency import run_in_threadpool
+from fastapi.responses import PlainTextResponse
+import inspect
+
+from cdtkit.http import HEALTH_PATH, Health
+
+NO_STORE = {"Cache-Control": "no-store"}
+
+
+def _threaded(check):
+    if inspect.iscoroutinefunction(check):
+        return check
+    return lambda: run_in_threadpool(check)
+
+
+def health_router(health=None, path=HEALTH_PATH):
+    """An APIRouter with the health route; `health` is a `cdtkit.http.Health` (none: always "ok")."""
+    checks = Health(**{name: _threaded(c) for name, c in (health.checks if health else {}).items()})
+    router = APIRouter()
+
+    @router.get(path, include_in_schema=False)
+    async def healthz():
+        ok, body = await checks.run()
+        return PlainTextResponse(body, status_code=200 if ok else 503, headers=NO_STORE)
+
+    return router

cdtkit/http/__init__.py

@@ -0,0 +1,39 @@
+"""The HTTP rules every app shares, framework-agnostic; `cdtkit.fastapi` adapts them to FastAPI.
+
+0.1.0: logfmt logging (`cdtkit.http.logging`) and the health check (`Health`, answered on `/healthz`, the path
+deploy/web.json's `"health"` names). Later releases add `/metrics`, RFC 9457 problem details, rate-limit classes and the
+Idempotency-Key store (claude-deploy-toolkit #422).
+"""
+import inspect
+import logging
+
+log = logging.getLogger("cdtkit.http")
+HEALTH_PATH = "/healthz"
+
+
+class Health:
+    """Named checks (sync or async callables; a check passes when it returns without raising), e.g. the database's
+    `SELECT 1`. With no checks the process answers "ok" as soon as it serves requests."""
+
+    def __init__(self, **checks):
+        self.checks = dict(checks)
+
+    def add(self, name, check):
+        self.checks[name] = check
+        return self
+
+    async def run(self):
+        """(ok, body): body is "ok\\n", or "unavailable: <names>\\n" naming the failed checks (never their errors,
+        which can hold internals; those go to the log)."""
+        failed = []
+        for name, check in self.checks.items():
+            try:
+                result = check()
+                if inspect.isawaitable(result):
+                    await result
+            except Exception as e:
+                failed.append(name)
+                log.warning("health check failed check=%s: %s", name, type(e).__name__)
+        if failed:
+            return False, "unavailable: %s\n" % ",".join(failed)
+        return True, "ok\n"

cdtkit/http/logging.py

@@ -0,0 +1,62 @@
+"""Operational logs on stdout in logfmt (standard, chapter "declarations", "Metrics and logs"): an ISO UTC time, the
+level, the component, the message, then `key=value` context. Users by internal ID only; never secrets or tokens.
+
+One event is one line: an exception adds `error="<Type>: <first message line>"`, and its causes for chained exceptions
+(`<-`), never the multi-line traceback (magic-minutes #56: the traceback's last line was often blank, so `error` said
+nothing).
+"""
+import logging
+import sys
+import time
+
+MAX_ERROR = 500
+CAUSES = 3   # the exception and at most two of its causes
+QUIET = ("httpx", "httpcore", "psycopg.pool")
+
+
+def _first_line(exc):
+    try:
+        text = str(exc)
+    except Exception:   # an exception whose str() fails still gets its line, with the type alone
+        return ""
+    for line in text.splitlines():
+        if line.strip():
+            return " ".join(line.split())
+    return ""
+
+
+def error_text(exc):
+    """The exception's type and first non-empty message line, then its causes', on one line; "" without one."""
+    parts, seen = [], set()
+    while exc is not None and id(exc) not in seen and len(parts) < CAUSES:
+        seen.add(id(exc))
+        first = _first_line(exc)
+        parts.append(type(exc).__name__ + (": " + first if first else ""))
+        exc = exc.__cause__ if exc.__cause__ is not None else (None if exc.__suppress_context__ else exc.__context__)
+    return " <- ".join(parts)[:MAX_ERROR]
+
+
+def _clean(text):
+    return " ".join(str(text).replace('"', "'").splitlines())
+
+
+class Logfmt(logging.Formatter):
+    def format(self, record):
+        line = 'time=%s level=%s component=%s msg="%s"' % (
+            time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(record.created)), record.levelname.lower(), record.name,
+            _clean(record.getMessage()))
+        exc = record.exc_info[1] if isinstance(record.exc_info, tuple) else None
+        if exc is not None:
+            line += ' error="%s"' % _clean(error_text(exc))
+        return line
+
+
+def setup(level=logging.INFO, quiet=QUIET):
+    """One handler on the root logger, whatever ran before (calling it again replaces it, never adds a second)."""
+    handler = logging.StreamHandler(sys.stdout)
+    handler.setFormatter(Logfmt())
+    root = logging.getLogger()
+    root.handlers[:] = [handler]
+    root.setLevel(level)
+    for noisy in quiet:
+        logging.getLogger(noisy).setLevel(logging.WARNING)

cdtkit/jobs/__init__.py

@@ -0,0 +1,122 @@
+"""The job queue: procrastinate in the app's own Postgres, one queue per job type (standard, chapter "declarations").
+
+The parameters (concurrency, attempts, backoff, timeouts, stop grace) come from TOOLKIT_JOB_TYPES, which the toolkit
+renders from deploy/workers.json (`cdtkit.config.job_types`); the app's `defaults` only cover local runs and tests.
+Payloads carry internal IDs only, never personal data.
+
+- `make_app(conninfo, job_types, handlers, defaults)`: the procrastinate app with one task per job type, each on its own
+  queue (so its concurrency holds) with the kit's `Backoff`.
+- A handler whose row is gone (deleted while the job waited, an account erased) raises `NoRow`: the job ends
+  successfully and logs `skipped=no-row`, with no retry and no failure hook. Generic on purpose: what the row belongs to
+  (an owner, a group) is the app's business until cdt-kit 0.2.0.
+- `on_final_failure(job_id, payload)` runs when a job's last attempt raises, so what the user sees never stays "running".
+- The worker loop, heartbeat and health are in `cdtkit.jobs.worker`; dead-worker recovery in `cdtkit.jobs.recovery`;
+  pruning finished jobs in `cdtkit.jobs.prune`.
+"""
+import logging
+import random
+import re
+
+from procrastinate import App, BaseRetryStrategy, PsycopgConnector, RetryDecision
+
+log = logging.getLogger("cdtkit.jobs")
+
+DEFAULT_PARAMS = {"concurrency": 1, "max_attempts": 3, "backoff": "30s", "timeout_item": "1h", "stop_grace": "60s"}
+BACKOFF_CAP = 6 * 3600     # seconds: the longest wait between two attempts
+MAX_ATTEMPTS = 25          # the standard's upper bound for max_attempts
+POLL_SECONDS = 5.0         # workers poll for new jobs instead of holding one LISTEN connection per queue
+UNITS = {"": 1, "s": 1, "m": 60, "h": 3600, "d": 86400}
+
+
+class NoRow(Exception):
+    """Raised by a handler when the row its job works on no longer exists: nothing left to do, the job succeeds."""
+
+
+def seconds(value, default=60):
+    """'90s', '15m', '2h', '1d' (or a bare number of seconds) -> seconds; `default` when empty or not a duration."""
+    m = re.match(r"^\s*(\d+)\s*([smhd]?)\s*$", str(value if value is not None else ""))
+    if not m:
+        return default
+    return int(m.group(1)) * UNITS[m.group(2)]
+
+
+def params(job_types, job_id, defaults=None):
+    """A job type's parameters: the kit's defaults, then the app's `defaults[job_id]`, then TOOLKIT_JOB_TYPES."""
+    p = dict(DEFAULT_PARAMS)
+    p.update((defaults or {}).get(job_id) or {})
+    p.update((job_types or {}).get(job_id) or {})
+    return p
+
+
+class Backoff(BaseRetryStrategy):
+    """procrastinate retry strategy: `max_attempts` runs in total (deploy/workers.json's meaning, at most 25), an
+    exponential wait from `backoff` (backoff, 2x, 4x, ...) capped at `cap` seconds, each shortened by up to `jitter`
+    (a share of it), so jobs that failed together do not retry together."""
+
+    def __init__(self, max_attempts=3, backoff=30, cap=BACKOFF_CAP, jitter=0.5, rng=None):
+        self.max_attempts = max(1, min(int(max_attempts), MAX_ATTEMPTS))
+        self.backoff, self.cap = max(0, int(backoff)), max(0, int(cap))
+        self.jitter = min(max(float(jitter), 0.0), 1.0)
+        self._rng = rng or random.Random()
+
+    def wait(self, failed):
+        """Seconds before the next attempt after `failed` failed attempts (1, 2, ...)."""
+        full = min(self.cap, self.backoff * (2 ** max(0, failed - 1)))
+        return int(full * (1 - self.jitter * self._rng.random()))
+
+    def get_retry_decision(self, *, exception, job):
+        """None (no retry) when this was the job's last attempt; procrastinate's `job.attempts` counts earlier ones."""
+        failed = int(job.attempts or 0) + 1
+        if failed >= self.max_attempts:
+            return None
+        return RetryDecision(retry_in={"seconds": self.wait(failed)})
+
+
+def make_app(conninfo, job_types, handlers=None, defaults=None, job_ids=None, pool_max=2, on_final_failure=None,
+             connector=None):
+    """The procrastinate app with one task per job type (`job_ids`, else the keys of `defaults` and `job_types`).
+
+    `handlers` maps job type -> async function taking the payload as keyword arguments (worker processes only; a web
+    process defers and never runs them). The connection pool holds 1 to `pool_max` connections; `connector` replaces it
+    (tests: `cdtkit.testing.queue`)."""
+    ids = list(job_ids or dict.fromkeys(list((defaults or {}).keys()) + list((job_types or {}).keys())))
+    app = App(connector=connector or PsycopgConnector(conninfo=conninfo, min_size=1, max_size=pool_max))
+    for job_id in ids:
+        p = params(job_types, job_id, defaults)
+        retry = Backoff(max_attempts=int(p.get("max_attempts", 3)), backoff=seconds(p.get("backoff"), 30))
+        func = (handlers or {}).get(job_id) or _not_here
+        app.task(name=job_id, queue=job_id, retry=retry, pass_context=True)(_guarded(job_id, func, retry, on_final_failure))
+    return app
+
+
+def _guarded(job_id, func, retry, on_final_failure):
+    """The handler, with NoRow ending the job and the final-failure hook when its last attempt raises."""
+    async def run(context, **payload):
+        try:
+            return await func(**payload)
+        except NoRow as gone:
+            log.info("job done job_type=%s id=%s skipped=no-row%s", job_id, context.job.id,
+                     " reason=%s" % "_".join(str(gone).split()) if str(gone) else "")
+            return None
+        except Exception as error:
+            if on_final_failure is not None and retry.get_retry_decision(exception=error, job=context.job) is None:
+                try:
+                    await on_final_failure(job_id, payload)
+                except Exception as hook_error:   # never hides the job's own error
+                    log.warning("final failure hook failed job_type=%s: %s", job_id, type(hook_error).__name__)
+            raise
+    run.__name__ = run.__qualname__ = getattr(func, "__name__", job_id)
+    return run
+
+
+async def _not_here(**kwargs):
+    raise RuntimeError("job handlers run in the worker")
+
+
+def defer(app, job_id, **payload):
+    """Queue a job on its own queue; returns its ID."""
+    return app.configure_task(job_id, queue=job_id).defer(**payload)
+
+
+async def defer_async(app, job_id, **payload):
+    return await app.configure_task(job_id, queue=job_id).defer_async(**payload)

cdtkit/jobs/prune.py

@@ -0,0 +1,59 @@
+"""Finished jobs are removed after `failed_retention` (deploy/workers.json; hours or days, default 30 days).
+
+Successful jobs are deleted as they finish (`delete_jobs="successful"` in `cdtkit.jobs.worker`); failed, cancelled and
+aborted jobs stay for `failed_retention` so their errors can be read, then go. Time-driven work is a declared job, so
+the app runs this from a schedule of its own (deploy/schedules.json), never from a loop in its code:
+
+    python -m cdtkit.jobs.prune [--service worker]   # DATABASE_*, DATABASE_PASSWORD_FILE and TOOLKIT_JOB_TYPES
+"""
+import asyncio
+import logging
+import sys
+
+from procrastinate import App, PsycopgConnector
+
+from cdtkit import config, jobs
+
+log = logging.getLogger("cdtkit.prune")
+DEFAULT_RETENTION = "30d"
+
+
+def retention_hours(entry):
+    """`failed_retention` of a TOOLKIT_JOB_TYPES entry in whole hours (at least 1)."""
+    value = (entry or {}).get("failed_retention") or DEFAULT_RETENTION
+    return max(1, jobs.seconds(value, jobs.seconds(DEFAULT_RETENTION)) // 3600)
+
+
+async def prune_finished(app, job_ids, entry):
+    """Delete the jobs of `job_ids` that finished (failed, cancelled, aborted; successful ones too) more than
+    `failed_retention` ago. Returns the hours used."""
+    hours = retention_hours(entry)
+    for job_id in job_ids:
+        await app.job_manager.delete_old_jobs(nb_hours=hours, queue=job_id, include_failed=True,
+                                              include_cancelled=True, include_aborted=True)
+    log.info("finished jobs pruned queues=%s older_than_hours=%d", ",".join(job_ids), hours)
+    return hours
+
+
+async def _main(service):
+    entry = config.worker_entry(service=service)
+    job_ids = list((entry.get("job_types") or {}).keys())
+    if not job_ids:
+        raise config.ConfigError("TOOLKIT_JOB_TYPES names no job types for %s" % service)
+    conninfo = config.database().conninfo("cdtkit-prune")
+    app = App(connector=PsycopgConnector(conninfo=conninfo, min_size=1, max_size=1))
+    async with app.open_async():
+        await prune_finished(app, job_ids, entry)
+
+
+def main(argv=None):
+    from cdtkit.http.logging import setup
+    args = list(sys.argv[1:] if argv is None else argv)
+    service = args[args.index("--service") + 1] if "--service" in args else "worker"
+    setup()
+    asyncio.run(_main(service))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

cdtkit/jobs/recovery.py

@@ -0,0 +1,108 @@
+"""Jobs survive a dead worker (magic-minutes #54, made generic).
+
+procrastinate tracks its workers with heartbeats but never re-queues the job of a worker that died or lost the database:
+the job stays `doing` forever. Each worker service therefore checks its own queues itself, at start and every few
+minutes (part of the queue's health, not a scheduled job):
+- a `doing` job whose worker stopped beating (or is gone) is put back in the queue, as its retry would have been; when it
+  has no attempt left it ends failed and `on_given_up(job_id, payload)` runs, so what the user sees says it was
+  interrupted;
+- a `doing` job running past its own time limit (`timeout_item`) plus a margin ends failed the same way: the handler's
+  timeout should have ended it, so its worker is stuck;
+- job types in `never_give_up` (an account's erasure must happen) are re-queued whatever their attempts.
+Jobs are idempotent (deploy/workers.json), so running one again is safe.
+"""
+import asyncio
+import inspect
+import logging
+from datetime import datetime, timezone
+
+from procrastinate.jobs import Status
+
+from cdtkit import jobs
+
+log = logging.getLogger("cdtkit.recovery")
+HEARTBEAT_EVERY = 10    # seconds between a worker's heartbeats (procrastinate's update_heartbeat_interval)
+STALLED_AFTER = 120     # seconds without a heartbeat before a worker counts as dead (stalled_worker_timeout)
+CHECK_EVERY = 300       # seconds between two checks
+TIMEOUT_MARGIN = 300    # seconds a job may run past its timeout_item before it counts as stuck
+OVERDUE_SQL = (
+    "SELECT j.id, j.args, j.attempts FROM procrastinate_jobs j WHERE j.status = 'doing' AND j.queue_name = %(queue)s"
+    " AND (SELECT max(e.at) FROM procrastinate_events e WHERE e.job_id = j.id AND e.type = 'started')"
+    " < now() - make_interval(secs => %(seconds)s)")
+
+
+class Interrupted(Exception):
+    """Stands for the death of a job's worker when its retry strategy is asked whether another attempt is left."""
+
+
+class Recovery:
+    """The checks of one worker service over the queues it runs (one queue per job type).
+
+    `on_given_up(job_id, payload)` (sync or async) marks what the user sees about a job that was given up; it may be
+    None. `defaults` are the app's job type defaults (as for `cdtkit.jobs.make_app`)."""
+
+    def __init__(self, app, job_ids, job_types, defaults=None, on_given_up=None, never_give_up=()):
+        self.app, self.job_ids, self.job_types, self.defaults = app, list(job_ids), job_types, defaults
+        self.on_given_up, self.never_give_up = on_given_up, tuple(never_give_up)
+
+    def _out_of_attempts(self, job_id, job):
+        """As if the job had raised: its own retry strategy decides whether an attempt is left."""
+        retry = self.app.tasks[job_id].retry_strategy
+        return retry is None or retry.get_retry_decision(exception=Interrupted(), job=job) is None
+
+    async def _mark(self, job_id, payload):
+        if self.on_given_up is None:
+            return
+        result = self.on_given_up(job_id, payload)
+        if inspect.isawaitable(result):
+            await result
+
+    async def _give_up(self, job_id, job_db_id, payload, why):
+        await self.app.job_manager.finish_job_by_id_async(job_db_id, status=Status.FAILED, delete_job=False)
+        await self._mark(job_id, payload)
+        log.warning("job given up id=%s queue=%s reason=%s", job_db_id, job_id, why.replace(" ", "_"))
+
+    async def _settle(self, action, job_db_id, job_id):
+        """One job's action; a job that left `doing` meanwhile (it finished) is skipped, never the whole pass."""
+        try:
+            await action()
+            return True
+        except Exception as e:
+            log.info("recovery skipped id=%s queue=%s: %s", job_db_id, job_id, type(e).__name__)
+            return False
+
+    async def check(self):
+        """One pass over the service's queues; returns (requeued, given_up)."""
+        manager, requeued, given_up = self.app.job_manager, 0, 0
+        now = datetime.now(timezone.utc)
+        for job_id in self.job_ids:
+            for job in await manager.get_stalled_jobs(queue=job_id, seconds_since_heartbeat=STALLED_AFTER):
+                if job_id not in self.never_give_up and self._out_of_attempts(job_id, job):
+                    why = "its worker died, no attempt left"
+                    if await self._settle(lambda j=job: self._give_up(job_id, j.id, j.task_kwargs, why), job.id, job_id):
+                        given_up += 1
+                elif await self._settle(lambda j=job: manager.retry_job(j), job.id, job_id):
+                    log.warning("stalled job requeued id=%s queue=%s attempts=%d", job.id, job_id, job.attempts + 1)
+                    requeued += 1
+            p = jobs.params(self.job_types, job_id, self.defaults)
+            limit = jobs.seconds(p.get("timeout_item"), 3600) + TIMEOUT_MARGIN
+            rows = await self.app.connector.execute_query_all_async(OVERDUE_SQL, queue=job_id, seconds=limit)
+            for row in rows:
+                if job_id in self.never_give_up:
+                    action = (lambda r=row: manager.retry_job_by_id_async(r["id"], retry_at=now))
+                    requeued += await self._settle(action, row["id"], job_id)
+                    continue
+                action = (lambda r=row: self._give_up(job_id, r["id"], r["args"], "past its time limit"))
+                given_up += await self._settle(action, row["id"], job_id)
+        return requeued, given_up
+
+    async def watch(self, every=CHECK_EVERY):
+        """At start, then every `every` seconds, until cancelled; a failed pass is logged and tried again later."""
+        while True:
+            try:
+                await self.check()
+            except asyncio.CancelledError:
+                raise
+            except Exception as e:
+                log.warning("recovery check failed: %s", type(e).__name__)
+            await asyncio.sleep(every)

cdtkit/jobs/worker.py

@@ -0,0 +1,96 @@
+"""A worker service's process: one procrastinate worker per queue, a heartbeat for the container health check and the
+dead-worker recovery, until SIGTERM (graceful within each job type's `stop_grace`).
+
+    async def main():
+        app = jobs.make_app(conninfo, job_types, HANDLERS, DEFAULTS, job_ids=MY_JOB_IDS)
+        migrations.migrate(conninfo, MIGRATIONS, queue_app=app)
+        await worker.run(app, MY_JOB_IDS, job_types, DEFAULTS, ping=db.ping, on_given_up=mark_failed)
+
+    python -m cdtkit.jobs.worker --health      # exit 0 while the heartbeat is fresh (the container's health check)
+"""
+import asyncio
+import inspect
+import logging
+import os
+import signal
+import sys
+import time
+
+from procrastinate.worker import Worker
+
+from cdtkit import jobs
+from cdtkit.jobs import recovery
+
+log = logging.getLogger("cdtkit.worker")
+HEARTBEAT = "/tmp/worker-heartbeat"
+HEARTBEAT_EVERY = 20    # seconds; the health check fails after four missed beats
+
+
+async def heartbeat(ping=None, path=HEARTBEAT, every=HEARTBEAT_EVERY):
+    """Writes the time into `path` every `every` seconds after `ping()` (sync or async, e.g. the database's SELECT 1)
+    answered; a failed ping skips the beat, so the health check fails while the database is unreachable."""
+    while True:
+        try:
+            if ping is not None:
+                result = await asyncio.to_thread(ping) if not inspect.iscoroutinefunction(ping) else await ping()
+                if inspect.isawaitable(result):
+                    await result
+            with open(path, "w") as f:
+                f.write(str(time.time()))
+        except asyncio.CancelledError:
+            raise
+        except Exception as e:   # a missed beat makes the health check fail; the loop keeps trying
+            log.warning("heartbeat failed: %s", type(e).__name__)
+        await asyncio.sleep(every)
+
+
+def health(path=HEARTBEAT, every=HEARTBEAT_EVERY):
+    """0 while the heartbeat is fresh (younger than four beats), else 1."""
+    try:
+        age = time.time() - os.path.getmtime(path)
+    except OSError:
+        return 1
+    return 0 if age < every * 4 else 1
+
+
+def workers(app, job_ids, job_types, defaults=None, name_prefix="w-"):
+    """One procrastinate Worker per job type, with its concurrency and stop grace, polling (no LISTEN connection), the
+    successful jobs deleted, and the heartbeat timings the recovery check uses."""
+    out = []
+    for job_id in job_ids:
+        p = jobs.params(job_types, job_id, defaults)
+        out.append(Worker(app, queues=[job_id], name=name_prefix + job_id, concurrency=int(p.get("concurrency", 1)),
+                          shutdown_graceful_timeout=jobs.seconds(p.get("stop_grace"), 60),
+                          install_signal_handlers=False, delete_jobs="successful",
+                          listen_notify=False, fetch_job_polling_interval=jobs.POLL_SECONDS,
+                          # a worker counts as dead after the recovery check's quiet time, never procrastinate's default
+                          # 30 s, when a starting worker would prune a live worker's row
+                          update_heartbeat_interval=recovery.HEARTBEAT_EVERY,
+                          stalled_worker_timeout=recovery.STALLED_AFTER))
+    return out
+
+
+async def run(app, job_ids, job_types, defaults=None, ping=None, on_given_up=None, never_give_up=(), name_prefix="w-",
+              heartbeat_path=HEARTBEAT):
+    """Run the service's queues until SIGTERM or SIGINT, with the heartbeat and the recovery watch."""
+    async with app.open_async():
+        ws = workers(app, job_ids, job_types, defaults, name_prefix)
+        log.info("worker ready: queues %s", ",".join(job_ids))
+        loop = asyncio.get_running_loop()
+        for sig in (signal.SIGTERM, signal.SIGINT):   # one stop for every queue's worker (graceful, stop_grace)
+            loop.add_signal_handler(sig, lambda: [w.stop() for w in ws])
+        beat = asyncio.create_task(heartbeat(ping, heartbeat_path))
+        watch = asyncio.create_task(recovery.Recovery(app, job_ids, job_types, defaults, on_given_up,
+                                                      never_give_up).watch())
+        try:
+            await asyncio.gather(*(w.run() for w in ws))
+        finally:
+            beat.cancel()
+            watch.cancel()
+
+
+if __name__ == "__main__":
+    if "--health" in sys.argv:
+        sys.exit(health())
+    print(__doc__.strip().splitlines()[-1].strip(), file=sys.stderr)
+    sys.exit(2)

cdtkit/migrations.py

@@ -0,0 +1,188 @@
+"""Forward-only SQL migrations, applied at app start, and the expand-only lint (standard, chapter "database").
+
+`migrate(conninfo, directory)` applies `<directory>/*.sql` in name order, each once, each in its own transaction, and
+records it in `schema_migrations (name, applied_at)`. A session advisory lock serialises the processes of a stack that
+start together (web and workers): the second waits, then finds nothing left to do. Idempotent; never rolls back.
+
+`lint(directory)` reports lines that contract the schema (drops, renames, type changes, new NOT NULL, UNIQUE or CHECK
+constraints on existing data). A contract is allowed only in a cleanup release that declares it in
+`deploy/package.json` "contracts"; marking the statement's line with `-- cdt: contract <reason>` says it was declared.
+
+    python -m cdtkit.migrations lint <directory> [--after <name>]   # exit 1 on findings
+    python -m cdtkit.migrations apply <directory>                   # DATABASE_* and DATABASE_PASSWORD_FILE
+"""
+import logging
+import os
+import re
+import sys
+
+log = logging.getLogger("cdtkit.migrations")
+
+LOCK_ID = 7316001   # the advisory lock's key; one database per app, so one fixed key is enough (magic-minutes' value)
+TABLE_SQL = ("CREATE TABLE IF NOT EXISTS schema_migrations (name text PRIMARY KEY,"
+             " applied_at timestamptz NOT NULL DEFAULT now())")
+CONTRACT_MARK = re.compile(r"--\s*cdt:\s*contract\b", re.I)
+CONTRACTS = (   # (pattern on one statement, what it does)
+    (r"\bDROP\s+(?P<k>TABLE|COLUMN|INDEX|VIEW|MATERIALIZED\s+VIEW|TYPE|SCHEMA|SEQUENCE|CONSTRAINT|DEFAULT|TRIGGER|FUNCTION)\b",
+     "drops %s"),
+    (r"\bRENAME\b", "renames"),
+    (r"\bALTER\s+(COLUMN\s+)?\S+\s+(SET\s+DATA\s+)?TYPE\b", "changes a column type"),
+    (r"\bSET\s+NOT\s+NULL\b", "adds NOT NULL to an existing column"),
+    (r"\bADD\s+(CONSTRAINT\s+\S+\s+)?(?P<k>UNIQUE|CHECK|PRIMARY\s+KEY|EXCLUDE)\b", "adds a %s constraint to an existing table"),
+    (r"\bCREATE\s+UNIQUE\s+INDEX\b", "adds a unique index"),
+    (r"\bTRUNCATE\b", "empties a table"),
+    (r"\bALTER\s+TYPE\s+\S+\s+DROP\s+VALUE\b", "removes an enum value"),
+)
+ADD_COLUMN = re.compile(r"\bADD\s+(COLUMN\s+)?(IF\s+NOT\s+EXISTS\s+)?(\S+)\s+[^,;]*", re.I)
+
+
+def files(directory):
+    return sorted(f for f in os.listdir(directory) if f.endswith(".sql"))
+
+
+def migrate(conninfo, directory, queue_app=None, lock_id=LOCK_ID):
+    """Apply the pending migrations of `directory`; then, with a procrastinate app, its queue schema when missing.
+    Returns the names applied (empty when the database was current)."""
+    import psycopg   # the core's only database driver; imported here so `lint` runs without it
+
+    applied = []
+    with psycopg.connect(conninfo, autocommit=True) as c:
+        c.execute("SELECT pg_advisory_lock(%s)", (lock_id,))
+        try:
+            c.execute(TABLE_SQL)
+            done = {r[0] for r in c.execute("SELECT name FROM schema_migrations").fetchall()}
+            for name in files(directory):
+                if name in done:
+                    continue
+                with open(os.path.join(directory, name), encoding="utf-8") as f:
+                    sql = f.read()
+                with c.transaction():
+                    c.execute(sql)
+                    c.execute("INSERT INTO schema_migrations (name) VALUES (%s)", (name,))
+                applied.append(name)
+                log.info("migration applied: %s", name)
+            if queue_app is not None:
+                exists = c.execute("SELECT to_regclass('procrastinate_jobs') IS NOT NULL").fetchone()[0]
+                if not exists:
+                    with c.transaction():
+                        c.execute(queue_app.schema_manager.get_schema())
+                    log.info("job queue schema applied (procrastinate)")
+        finally:
+            c.execute("SELECT pg_advisory_unlock(%s)", (lock_id,))
+    return applied
+
+
+def _strip_comments(sql):
+    """The SQL without comments and string literals (their text is not schema), keeping the line breaks."""
+    out, i, n = [], 0, len(sql)
+    while i < n:
+        if sql.startswith("--", i):
+            j = sql.find("\n", i)
+            i = n if j < 0 else j
+        elif sql.startswith("/*", i):
+            j = sql.find("*/", i + 2)
+            j = n if j < 0 else j + 2
+            out.append("\n" * sql.count("\n", i, j))
+            i = j
+        elif sql[i] == "'":
+            j = i + 1
+            while j < n and not (sql[j] == "'" and not sql.startswith("''", j)):
+                j += 2 if sql.startswith("''", j) else 1
+            out.append("''" + "\n" * sql.count("\n", i, j))
+            i = j + 1
+        else:
+            out.append(sql[i])
+            i += 1
+    return "".join(out)
+
+
+def _statements(sql):
+    """(first line number, statement text, raw lines) per statement; `$$` bodies are kept whole."""
+    clean = _strip_comments(sql)
+    raw_lines = sql.splitlines()
+    result, start, i, line, depth = [], 0, 0, 1, False
+    begin_line = 1
+    while i < len(clean):
+        if clean.startswith("$$", i):
+            depth = not depth
+            i += 2
+            continue
+        ch = clean[i]
+        if ch == "\n":
+            line += 1
+        if ch == ";" and not depth:
+            text = clean[start:i]
+            if text.strip():
+                first = begin_line + text[:len(text) - len(text.lstrip())].count("\n")
+                result.append((first, text, raw_lines[first - 1:line]))
+            start, begin_line = i + 1, line
+        i += 1
+    text = clean[start:]
+    if text.strip():
+        first = begin_line + text[:len(text) - len(text.lstrip())].count("\n")
+        result.append((first, text, raw_lines[first - 1:]))
+    return result
+
+
+def _findings(statement):
+    s = " ".join(statement.split())
+    upper = s.upper()
+    found = []
+    if upper.startswith(("CREATE FUNCTION", "CREATE OR REPLACE FUNCTION", "CREATE PROCEDURE", "CREATE OR REPLACE PROCEDURE")):
+        return found
+    for pattern, what in CONTRACTS:
+        m = re.search(pattern, s, re.I)
+        if m:
+            kind = " ".join((m.groupdict().get("k") or "").lower().split())
+            if kind == "default" and what.startswith("drops"):
+                continue   # dropping a default relaxes nothing the stable version needs
+            found.append(what % kind if "%s" in what else what)
+    if upper.startswith("ALTER TABLE"):
+        for m in ADD_COLUMN.finditer(s):
+            if m.group(3).upper() in ("CONSTRAINT", "UNIQUE", "CHECK", "PRIMARY", "FOREIGN", "EXCLUDE"):
+                continue
+            column = m.group(0).upper()
+            if "NOT NULL" in column and " DEFAULT " not in " " + column + " ":
+                found.append("adds a NOT NULL column without a default")
+    return found
+
+
+def lint(directory, after=None):
+    """Contract lines of `directory`'s migrations (only files named after `after`, when given): a list of
+    (file, line, what) for statements not marked `-- cdt: contract <reason>`."""
+    out = []
+    for name in files(directory):
+        if after and name <= after:
+            continue
+        with open(os.path.join(directory, name), encoding="utf-8") as f:
+            sql = f.read()
+        for line, statement, raw in _statements(sql):
+            if any(CONTRACT_MARK.search(r) for r in raw):
+                continue
+            for what in _findings(statement):
+                out.append((name, line, what))
+    return out
+
+
+def main(argv=None):
+    args = list(sys.argv[1:] if argv is None else argv)
+    if len(args) < 2 or args[0] not in ("lint", "apply"):
+        print(__doc__.strip().splitlines()[-2].strip(), file=sys.stderr)
+        print(__doc__.strip().splitlines()[-1].strip(), file=sys.stderr)
+        return 2
+    if args[0] == "lint":
+        after = args[args.index("--after") + 1] if "--after" in args else None
+        found = lint(args[1], after)
+        for name, line, what in found:
+            print("%s:%d: %s (expand only: declare it in deploy/package.json \"contracts\" and mark the line"
+                  " -- cdt: contract <reason>)" % (name, line, what))
+        return 1 if found else 0
+    from cdtkit import config
+    from cdtkit.http.logging import setup
+    setup()
+    migrate(config.database().conninfo("cdtkit-migrate"), args[1])
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

cdtkit/testing/__init__.py

@@ -0,0 +1,7 @@
+"""Test stand-ins (extra `cdt-kit[test]`): a Postgres (`cdtkit.testing.postgres`), the queue (`cdtkit.testing.queue`)
+and the pytest fixtures (`cdtkit.testing.plugin`, registered as a pytest plugin by the extra).
+
+Fixtures: `cdt_postgres` (one server per test session) and `cdt_database` (a fresh empty database per test, a
+`cdtkit.config.Database`; skipped when no Postgres is available). The broker stand-in arrives with identity (0.2.0),
+the ClamAV stub with `cdtkit.scan`.
+"""

cdtkit/testing/plugin.py

@@ -0,0 +1,28 @@
+"""pytest fixtures of the kit (entry point `pytest11` of the extra `cdt-kit[test]`)."""
+import pytest
+
+from cdtkit.testing.postgres import PostgresServer, Unavailable
+
+
+@pytest.fixture(scope="session")
+def cdt_postgres():
+    try:
+        server = PostgresServer()
+    except Unavailable as e:
+        pytest.skip(str(e))
+    try:
+        yield server
+    finally:
+        server.close()
+
+
+@pytest.fixture
+def cdt_database(cdt_postgres):
+    db = cdt_postgres.database()
+    try:
+        yield db
+    finally:
+        try:
+            cdt_postgres.drop(db)
+        except Exception:
+            pass

cdtkit/testing/postgres.py

@@ -0,0 +1,93 @@
+"""A Postgres for tests: an existing server named by CDT_TEST_DATABASE_* , else a throwaway local cluster.
+
+- `CDT_TEST_DATABASE_HOST` (and optional `_PORT`, `_USER`, `_PASSWORD`, `_NAME`): a server the tests may create and
+  drop databases on (the kit's selftest service uses its sidecar this way).
+- Otherwise `initdb` and `pg_ctl` (from PATH, `PG_BIN`, or /usr/lib/postgresql/<major>/bin) start a cluster in a temporary
+  folder, listening only on a Unix socket there, stopped and removed afterwards.
+
+`PostgresServer().database()` creates a fresh, empty database per test and returns a `cdtkit.config.Database`.
+"""
+import glob
+import os
+import shutil
+import subprocess
+import tempfile
+import uuid
+
+from cdtkit.config import Database
+
+
+class Unavailable(RuntimeError):
+    """No server configured and no local Postgres binaries: tests that need a database are skipped."""
+
+
+def _bin_dir():
+    if os.environ.get("PG_BIN"):
+        return os.environ["PG_BIN"]
+    found = shutil.which("pg_ctl")
+    if found:
+        return os.path.dirname(found)
+    dirs = sorted(glob.glob("/usr/lib/postgresql/*/bin"), key=lambda d: int(d.split("/")[-2]) if d.split("/")[-2].isdigit() else 0)
+    if dirs:
+        return dirs[-1]
+    raise Unavailable("no Postgres: set CDT_TEST_DATABASE_HOST or install the server (initdb, pg_ctl)")
+
+
+class PostgresServer:
+    def __init__(self):
+        self.tmp, self.created = None, []
+        env = os.environ
+        if env.get("CDT_TEST_DATABASE_HOST"):
+            self.admin = Database(env["CDT_TEST_DATABASE_HOST"], int(env.get("CDT_TEST_DATABASE_PORT") or 5432),
+                                  env.get("CDT_TEST_DATABASE_NAME") or "postgres",
+                                  env.get("CDT_TEST_DATABASE_USER") or "postgres",
+                                  env.get("CDT_TEST_DATABASE_PASSWORD") or "")
+        else:
+            self.admin = self._start()
+
+    def _start(self):
+        bindir = _bin_dir()
+        self.tmp = tempfile.mkdtemp(prefix="cdtkit-pg-")
+        data, sock = os.path.join(self.tmp, "data"), os.path.join(self.tmp, "sock")
+        os.mkdir(sock)
+        user = "cdtkit"
+        run = dict(check=True, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
+        subprocess.run([os.path.join(bindir, "initdb"), "-D", data, "-U", user, "-A", "trust", "--no-sync"], **run)
+        opts = "-c listen_addresses='' -k %s -c fsync=off -c synchronous_commit=off -c full_page_writes=off" % sock
+        subprocess.run([os.path.join(bindir, "pg_ctl"), "-D", data, "-o", opts, "-w", "-l",
+                        os.path.join(self.tmp, "log"), "start"], **run)
+        self._pg_ctl = (os.path.join(bindir, "pg_ctl"), data)
+        return Database(sock, 5432, "postgres", user, "")
+
+    def conninfo(self, db=None):
+        d = db or self.admin
+        info = "host=%s port=%d dbname=%s user=%s" % (d.host, d.port, d.name, d.user)
+        return info + (" password=%s" % d.password if d.password else "")
+
+    def database(self):
+        """A new empty database owned by the admin user."""
+        import psycopg
+        name = "t_" + uuid.uuid4().hex[:12]
+        with psycopg.connect(self.conninfo(), autocommit=True) as c:
+            c.execute('CREATE DATABASE "%s"' % name)
+        self.created.append(name)
+        return Database(self.admin.host, self.admin.port, name, self.admin.user, self.admin.password)
+
+    def drop(self, db):
+        import psycopg
+        with psycopg.connect(self.conninfo(), autocommit=True) as c:
+            c.execute('DROP DATABASE IF EXISTS "%s" WITH (FORCE)' % db.name)
+
+    def close(self):
+        if self.tmp:
+            pg_ctl, data = self._pg_ctl
+            subprocess.run([pg_ctl, "-D", data, "-m", "immediate", "stop"], stdout=subprocess.DEVNULL,
+                           stderr=subprocess.DEVNULL)
+            shutil.rmtree(self.tmp, ignore_errors=True)
+            self.tmp = None
+        else:
+            for name in self.created:
+                try:
+                    self.drop(Database(self.admin.host, self.admin.port, name, self.admin.user, self.admin.password))
+                except Exception:
+                    pass

cdtkit/testing/queue.py

@@ -0,0 +1,35 @@
+"""The queue stand-in: the kit's procrastinate app on procrastinate's in-memory connector, and a helper that runs the
+queued jobs until none is left (retries made due at once), so a test checks a handler's outcome without a worker.
+
+    app = memory_app(job_types, HANDLERS, DEFAULTS)
+    async with app.open_async():
+        await jobs.defer_async(app, "convert", meeting_id=1)
+        await drain(app)
+        assert statuses(app) == {"succeeded": 1}
+"""
+from collections import Counter
+
+from procrastinate.testing import InMemoryConnector
+
+from cdtkit import jobs
+
+
+def memory_app(job_types=None, handlers=None, defaults=None, job_ids=None, on_final_failure=None):
+    return jobs.make_app("", job_types or {}, handlers, defaults, job_ids, on_final_failure=on_final_failure,
+                         connector=InMemoryConnector())
+
+
+async def drain(app, rounds=30):
+    """Run every queue's jobs until none is due, making retries due at once (at most `rounds` passes)."""
+    for _ in range(rounds):
+        pending = [j for j in app.connector.jobs.values() if j["status"] == "todo"]
+        if not pending:
+            return
+        for j in pending:
+            j["scheduled_at"] = None
+        await app.run_worker_async(wait=False, install_signal_handlers=False, listen_notify=False)
+
+
+def statuses(app):
+    """{status: count} of the jobs the connector holds."""
+    return dict(Counter(j["status"] for j in app.connector.jobs.values()))