simple-module-background-tasks 0.0.36__tar.gz → 0.0.38__tar.gz

{simple_module_background_tasks-0.0.36 → simple_module_background_tasks-0.0.38}/.gitignore

@@ -7,8 +7,6 @@ build/
 .venv/
 *.egg
 
-# UV
-uv.lock
 
 # Node
 node_modules/
@@ -64,3 +62,5 @@ host/client_app/.playwright-cli/*
 .superpowers/
 .qa/
 .verify/
+/openapi.json
+qa-shots/

{simple_module_background_tasks-0.0.36 → simple_module_background_tasks-0.0.38}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: simple_module_background_tasks
-Version: 0.0.36
+Version: 0.0.38
 Summary: Celery + Redis task queue with admin UI for monitoring and retrying failed/stuck tasks
 Project-URL: Homepage, https://github.com/antosubash/simple_module_python
 Project-URL: Repository, https://github.com/antosubash/simple_module_python
@@ -23,10 +23,10 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.12
 Requires-Dist: celery[redis]>=5.4
 Requires-Dist: redis>=5
-Requires-Dist: simple-module-core==0.0.36
-Requires-Dist: simple-module-db==0.0.36
-Requires-Dist: simple-module-hosting==0.0.36
-Requires-Dist: simple-module-settings==0.0.36
+Requires-Dist: simple-module-core==0.0.38
+Requires-Dist: simple-module-db==0.0.38
+Requires-Dist: simple-module-hosting==0.0.38
+Requires-Dist: simple-module-settings==0.0.38
 Description-Content-Type: text/markdown
 
 # simple_module_background_tasks
@@ -46,6 +46,8 @@ Requires a Redis broker. The broker / result-backend URLs are module settings (`
 - Zero-config task discovery — any installed module that ships a `tasks.py` has its tasks autodiscovered (`celery.autodiscover_tasks` imports `<package>.tasks` for every installed module). No per-module registration hook.
 - Admin UI at `/admin/background-tasks` — list recent runs, retry failed, inspect tracebacks (gated by the `background_tasks.view` permission).
 - `build_celery(settings)` factory in `background_tasks.celery_app`, plus `bind_task_context` / `get_log_context` / `install_log_filter` exported from the package root (import name `background_tasks`, distribution name `simple_module_background_tasks`).
+- `enqueue_after_commit(db, task, *args, **kwargs)` — publish a task only once the session's transaction commits, never on rollback.
+- `heartbeat()` — called from a long task body, keeps the stuck sweep from flagging it; `@shared_task(stuck_after=seconds)` sets a per-task stuck threshold. See the module docs for both.
 
 ## Usage
 
@@ -80,6 +82,27 @@ uv run celery -A scripts.run_worker:celery worker -l info
 uv run celery -A scripts.run_worker:celery beat   -l info
 ```
 
+## Routing, tuning and per-worker module subsets
+
+A module can ship a light `task_config.py` (`TASK_ROUTES`, `QUEUES`, optional
+`BEAT_SCHEDULE`) that is read without importing `tasks.py`. The
+`task_routes_override` setting re-routes tasks at runtime, and the Workers page
+lists the effective routes. Tuning settings (`visibility_timeout_seconds`,
+`worker_max_tasks_per_child`, `task_time_limit`, `worker_concurrency`, ...) and a
+raw `SM_BG_TASKS_CELERY_OVERRIDES` (env only, unsafe keys refused) default to Celery's own defaults.
+`SM_BG_TASKS_WORKER_QUEUES=gpu,default make worker` consumes only those queues
+and imports only the modules whose `QUEUES` overlap (see `SM_BG_TASKS_INCLUDE_MODULES` /
+`SM_BG_TASKS_EXCLUDE_MODULES`; env only, not DB settings). Full details in `docs/modules/background_tasks.md`.
+
+## Durable events
+
+This module is the transport behind `bus.publish_durable(event, db)`: the event
+is written to an outbox table in the caller's transaction and delivered by a
+worker to every `bus.subscribe_durable(...)` handler, at least once, with
+backoff, after the commit. Writes a handler makes through `ctx.db` land exactly
+once. Undelivered and dead rows are on `/admin/background-tasks/outbox`. Full
+details in `docs/modules/background_tasks.md` § Durable events.
+
 ## Worker log context
 
 Every worker log line automatically carries the Celery task identifiers

{simple_module_background_tasks-0.0.36 → simple_module_background_tasks-0.0.38}/README.md

@@ -15,6 +15,8 @@ Requires a Redis broker. The broker / result-backend URLs are module settings (`
 - Zero-config task discovery — any installed module that ships a `tasks.py` has its tasks autodiscovered (`celery.autodiscover_tasks` imports `<package>.tasks` for every installed module). No per-module registration hook.
 - Admin UI at `/admin/background-tasks` — list recent runs, retry failed, inspect tracebacks (gated by the `background_tasks.view` permission).
 - `build_celery(settings)` factory in `background_tasks.celery_app`, plus `bind_task_context` / `get_log_context` / `install_log_filter` exported from the package root (import name `background_tasks`, distribution name `simple_module_background_tasks`).
+- `enqueue_after_commit(db, task, *args, **kwargs)` — publish a task only once the session's transaction commits, never on rollback.
+- `heartbeat()` — called from a long task body, keeps the stuck sweep from flagging it; `@shared_task(stuck_after=seconds)` sets a per-task stuck threshold. See the module docs for both.
 
 ## Usage
 
@@ -49,6 +51,27 @@ uv run celery -A scripts.run_worker:celery worker -l info
 uv run celery -A scripts.run_worker:celery beat   -l info
 ```
 
+## Routing, tuning and per-worker module subsets
+
+A module can ship a light `task_config.py` (`TASK_ROUTES`, `QUEUES`, optional
+`BEAT_SCHEDULE`) that is read without importing `tasks.py`. The
+`task_routes_override` setting re-routes tasks at runtime, and the Workers page
+lists the effective routes. Tuning settings (`visibility_timeout_seconds`,
+`worker_max_tasks_per_child`, `task_time_limit`, `worker_concurrency`, ...) and a
+raw `SM_BG_TASKS_CELERY_OVERRIDES` (env only, unsafe keys refused) default to Celery's own defaults.
+`SM_BG_TASKS_WORKER_QUEUES=gpu,default make worker` consumes only those queues
+and imports only the modules whose `QUEUES` overlap (see `SM_BG_TASKS_INCLUDE_MODULES` /
+`SM_BG_TASKS_EXCLUDE_MODULES`; env only, not DB settings). Full details in `docs/modules/background_tasks.md`.
+
+## Durable events
+
+This module is the transport behind `bus.publish_durable(event, db)`: the event
+is written to an outbox table in the caller's transaction and delivered by a
+worker to every `bus.subscribe_durable(...)` handler, at least once, with
+backoff, after the commit. Writes a handler makes through `ctx.db` land exactly
+once. Undelivered and dead rows are on `/admin/background-tasks/outbox`. Full
+details in `docs/modules/background_tasks.md` § Durable events.
+
 ## Worker log context
 
 Every worker log line automatically carries the Celery task identifiers

simple_module_background_tasks-0.0.38/background_tasks/__init__.py

@@ -0,0 +1,39 @@
+"""BackgroundTasks module — Celery + Redis task queue with admin UI."""
+
+from typing import TYPE_CHECKING, Any
+
+from background_tasks.after_commit import enqueue_after_commit
+from background_tasks.log_context import (
+    bind_task_context,
+    get_log_context,
+    install_log_filter,
+)
+
+if TYPE_CHECKING:
+    from background_tasks.task_heartbeat import heartbeat
+    from background_tasks.worker_settings import settings_for
+
+_LAZY_HEARTBEAT = "heartbeat"
+_LAZY_SETTINGS = "settings_for"
+
+
+def __getattr__(name: str) -> Any:
+    if name == _LAZY_HEARTBEAT:
+        from background_tasks.task_heartbeat import heartbeat
+
+        return heartbeat
+    if name == _LAZY_SETTINGS:
+        from background_tasks.worker_settings import settings_for
+
+        return settings_for
+    raise AttributeError(name)
+
+
+__all__ = [
+    "bind_task_context",
+    "enqueue_after_commit",
+    "get_log_context",
+    "heartbeat",
+    "install_log_filter",
+    "settings_for",
+]

simple_module_background_tasks-0.0.38/background_tasks/_lifecycle.py

@@ -0,0 +1,174 @@
+"""Startup/shutdown wiring for the BackgroundTasks module.
+
+``BackgroundTasksModule`` keeps the ``on_startup`` / ``on_shutdown`` hooks and
+delegates here, so the module file stays a declaration of what the module
+contributes and this file owns the order things are installed and torn down.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from fastapi import FastAPI
+
+logger = logging.getLogger(__name__)
+
+
+async def startup(app: FastAPI, module_name: str) -> None:
+    """Build the Celery app, install signal handlers, hand-off to worker."""
+    import asyncio
+
+    from simple_module_hosting.log_sinks import redact
+
+    from background_tasks.celery_app import build_celery
+    from background_tasks.db_settings import worker_database_options
+    from background_tasks.log_sink import install_log_sink
+    from background_tasks.signals import bind_event_bus
+    from background_tasks.sync_db import configure_database, set_database_url
+    from background_tasks.tenant_context import set_default_tenant
+
+    services = app.state.background_tasks
+    # Pin the sync engine to the same URL the host's async settings
+    # resolved — pydantic-settings reads ``.env`` but never propagates
+    # to ``os.environ``, so signals would otherwise fall back to the
+    # SQLite default and silently drop ``TaskExecution`` rows.
+    host = app.state.sm.settings
+    multi = bool(getattr(host, "multi_tenant", False))
+    default_tenant = None if multi else getattr(host, "default_tenant", "") or None
+    set_database_url(host.database_url, tenant_strict=multi, default_tenant=default_tenant)
+    configure_database(worker_database_options(host, services.settings))
+    set_default_tenant(default_tenant)
+    # build_celery imports `signals` for side effects and runs
+    # `autodiscover_tasks` across every installed module.
+    services.celery = build_celery(services.settings)
+    # Let signal handlers hop onto the API loop to publish events.
+    # In a standalone worker process this bind never runs, so signals
+    # stay a silent no-op — that's the documented cross-process limit.
+    bind_event_bus(app.state.sm.event_bus, asyncio.get_running_loop())
+    install_durable_transport(app)
+    install_log_sink(services.settings)
+    logger.info(
+        "BackgroundTasks: Celery app ready (broker=%s, queue=%s)",
+        redact(services.settings.broker_url),
+        services.settings.task_default_queue,
+    )
+
+    await start_invalidation_transport(app)
+
+    # Registered here, not in register_health_checks: the Celery app does
+    # not exist until this hook builds it, and the check must follow later
+    # settings changes rather than pinning the boot-time instance.
+    from simple_module_core.health import HealthCheck
+
+    from background_tasks.health import CHECK_REDIS, build_redis_check
+
+    app.state.sm.health_registry.add(
+        HealthCheck(
+            name=CHECK_REDIS,
+            check=build_redis_check(app),
+            module=module_name,
+            # On demand only, like the mailer and storage checks. An
+            # earlier version had this on the readiness probe, reasoning
+            # that Redis is infrastructure this app owns rather than a
+            # third party. That conflates two different questions: whether
+            # this process can serve requests, and whether the system is
+            # fully functional. Failing readiness pulls the web tier out of
+            # the load balancer, turning "background jobs are backed up"
+            # into "the site is down" — and it made /health/ready report
+            # unhealthy on every deployment without Redis, which CI caught.
+            # The wizard's connection step and the settings screen's "Test
+            # connection" both run it explicitly, which is what it is for.
+            probe=False,
+        )
+    )
+
+
+def install_durable_transport(app: FastAPI) -> None:
+    """Make ``bus.publish_durable`` write to this module's outbox (GH #459)."""
+    from background_tasks.constants import OutboxMode
+    from background_tasks.outbox_sweep import start_sweep
+    from background_tasks.outbox_transport import OutboxTransport
+    from background_tasks.outbox_worker import use_bus
+
+    bus = app.state.sm.event_bus
+    services = app.state.background_tasks
+    mode = (
+        OutboxMode.INPROCESS
+        if services.celery.conf.task_always_eager
+        else OutboxMode(services.settings.outbox_mode)
+    )
+    transport = OutboxTransport(services=services, bus=bus, db_state=app.state.sm.db, mode=mode)
+    bus.durable_transport = transport
+    if mode == OutboxMode.INPROCESS:
+        start_sweep(services, app.state.sm.db.session_factory, bus, transport.policy)
+    # An eager in-process dispatch delivers with the app's bus, which already
+    # carries every module's durable subscriptions.
+    use_bus(bus)
+
+
+async def start_invalidation_transport(app: FastAPI) -> None:
+    """Hand the framework bus the Redis connection this module configures.
+
+    The transport is installed here rather than in a registration hook
+    because it needs a running event loop for its listener task, and it is
+    installed by *this* module because it owns the Redis URL — a cache-owning
+    module opening its own connection is the duplication GH #318 exists to
+    avoid.
+
+    A worker process that runs Celery without the FastAPI app never reaches
+    this, so it publishes nothing and hears nothing. That is the same
+    documented cross-process limit as ``bind_event_bus`` above: a task that
+    invalidates a cache should do it through an API call or accept the TTL.
+    """
+    from background_tasks.invalidation import RedisInvalidationTransport
+
+    services = app.state.background_tasks
+    s = services.settings
+    if not s.broadcast_invalidations:
+        logger.info(
+            "BackgroundTasks: cross-worker cache invalidation disabled "
+            "(broadcast_invalidations=false) — per-process caches expire on their own TTL"
+        )
+        return
+
+    bus = app.state.sm.invalidation
+    transport = RedisInvalidationTransport(bus, s.broker_url, s.invalidation_channel)
+    await transport.start()
+    bus.set_transport(transport)
+    services.invalidation_transport = transport
+    logger.info(
+        "BackgroundTasks: invalidation transport on %r for channel(s) %s",
+        s.invalidation_channel,
+        ", ".join(bus.channels) or "(none subscribed)",
+    )
+
+
+async def shutdown(app: FastAPI) -> None:
+    from background_tasks.log_sink import uninstall_log_sink
+    from background_tasks.outbox_sweep import stop_sweep
+    from background_tasks.outbox_worker import reset_worker_state
+    from background_tasks.signals import unbind_event_bus
+    from background_tasks.sync_db import dispose_sync_engine
+
+    await stop_sweep(app.state.background_tasks)
+    bus = getattr(app.state.sm, "event_bus", None)
+    if bus is not None:
+        drain = getattr(bus.durable_transport, "drain", None)
+        if drain is not None:
+            # Eager deliveries still running on this loop.
+            await drain()
+        bus.durable_transport = None
+    reset_worker_state()
+    services = app.state.background_tasks
+    if services.invalidation_transport is not None:
+        app.state.sm.invalidation.clear_transport()
+        await services.invalidation_transport.stop()
+        services.invalidation_transport = None
+    if services.celery is not None:
+        services.celery.close()
+    unbind_event_bus()
+    # Before the engine goes: the final flush still needs it.
+    uninstall_log_sink()
+    dispose_sync_engine()

{simple_module_background_tasks-0.0.36 → simple_module_background_tasks-0.0.38}/background_tasks/_signal_bus.py

@@ -7,6 +7,7 @@ re-exports :func:`bind_event_bus` / :func:`unbind_event_bus`.
 from __future__ import annotations
 
 import asyncio
+import concurrent.futures
 import logging
 from typing import Any
 
@@ -53,7 +54,7 @@ def publish_from_signal(event: Any) -> None:
     future.add_done_callback(_log_publish_failure)
 
 
-def _log_publish_failure(future: asyncio.Future[Any]) -> None:
+def _log_publish_failure(future: concurrent.futures.Future[Any]) -> None:
     if future.cancelled():
         return
     exc = future.exception()

{simple_module_background_tasks-0.0.36 → simple_module_background_tasks-0.0.38}/background_tasks/_signal_support.py

@@ -7,14 +7,20 @@ These helpers are intentionally side-effect-free except for the DB write in
 
 from __future__ import annotations
 
+import logging
 import traceback as tb_mod
+from collections.abc import Callable
 from datetime import UTC, datetime
 from typing import Any
 
 from sqlalchemy import select
 from sqlalchemy.orm import Session
 
+from background_tasks.constants import INTERNAL_TASK_SWEEP_OUTBOX, TaskStatus
 from background_tasks.models import TaskExecution
+from background_tasks.sync_db import sync_session
+
+logger = logging.getLogger(__name__)
 
 
 def now_utc() -> datetime:
@@ -47,7 +53,7 @@ def upsert_by_celery_id(
     """
     row: TaskExecution | None = None
     if celery_task_id:
-        stmt = select(TaskExecution).where(TaskExecution.celery_task_id == celery_task_id)
+        stmt = select(TaskExecution).where(TaskExecution.celery_task_id == celery_task_id)  # ty: ignore[invalid-argument-type]
         row = session.execute(stmt).scalar_one_or_none()
     if row is None:
         row = TaskExecution(celery_task_id=celery_task_id, **defaults)
@@ -103,3 +109,48 @@ def render_traceback(einfo: Any, exception: BaseException | None) -> str | None:
     if exception is not None:
         return "".join(tb_mod.format_exception(type(exception), exception, exception.__traceback__))
     return None
+
+
+# Beat ticks that fire every few seconds and are a no-op almost every time;
+# recording each run buried real work (QA, GH #459). Only a run that went wrong
+# is worth a row.
+_UNRECORDED_WHEN_QUIET = frozenset({INTERNAL_TASK_SWEEP_OUTBOX})
+_QUIET_STATUSES = frozenset({None, TaskStatus.PENDING, TaskStatus.RUNNING, TaskStatus.SUCCESS})
+
+
+def apply_signal(
+    handler: str,
+    *,
+    celery_task_id: str | None,
+    defaults: dict[str, Any],
+    after: Callable[[TaskExecution], None] | None = None,
+    tenant_id: str | None = None,
+    owner_user_id: str | None = None,
+) -> TaskExecution | None:
+    """Open a sync session, upsert by celery_task_id, log on failure.
+
+    Returns the upserted row (or ``None`` if the session raised) so callers
+    can read DB-assigned fields like ``id`` without a second round trip.
+
+    ``tenant_id`` and ``owner_user_id`` (the message's) are written only when
+    known, so a signal whose message carries neither never blanks a row already
+    stamped.
+    """
+    if (
+        defaults.get("task_name") in _UNRECORDED_WHEN_QUIET
+        and defaults.get("status") in _QUIET_STATUSES
+    ):
+        return None
+    if tenant_id:
+        defaults = {**defaults, "tenant_id": tenant_id}
+    if owner_user_id:
+        defaults = {**defaults, "owner_user_id": owner_user_id}
+    try:
+        with sync_session() as session:
+            row = upsert_by_celery_id(session, celery_task_id=celery_task_id, defaults=defaults)
+            if after is not None:
+                after(row)
+            return row
+    except Exception:
+        logger.exception("%s failed for task_id=%s", handler, celery_task_id)
+        return None

simple_module_background_tasks-0.0.38/background_tasks/after_commit.py

@@ -0,0 +1,137 @@
+"""Publish to the broker only once the transaction that justifies it commits.
+
+A task enqueued inside a transaction races that transaction: the worker can
+pick the message up before the row it refers to is visible (eager mode runs
+it *inside* ``send_task``), and if the commit then fails the task is already
+queued against nothing. Both are fixed the same way — hold the publish until
+the commit has landed, and drop it if the transaction rolls back (GH #452).
+
+Two delivery paths, picked per session:
+
+- **A request's session** (``get_db``) — the publish goes through
+  :meth:`~simple_module_db.RequestSession.on_commit`, which the framework runs
+  after the request's commit and before the response is sent, and discards on
+  rollback or commit failure.
+- **Any other session** — a worker's sync session, a CLI script's own
+  ``async with session.begin()`` — has nobody to run ``on_commit`` callbacks,
+  so the publish hangs off SQLAlchemy's ``after_commit`` event instead and is
+  dropped when the outermost transaction ends without committing.
+
+The broker id is reserved up front and returned, so the caller can write it
+onto its own row in the same transaction: the row names the task before the
+task exists, never the other way round.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import uuid
+from collections.abc import Callable, Sequence
+from typing import Any
+
+from simple_module_db import RequestSession, bind_current_tenant, mark_written
+from simple_module_db.transaction import SESSION_START_KEY
+from sqlalchemy import event
+from sqlalchemy.ext.asyncio import AsyncSession
+from sqlalchemy.orm import Session, SessionTransaction
+
+
+class _LazyCurrentApp:
+    """``celery.current_app``, imported on first use so importing this module stays light."""
+
+    def __getattr__(self, name: str) -> Any:
+        from celery import current_app as _current_app
+
+        return getattr(_current_app, name)
+
+
+current_app: Any = _LazyCurrentApp()
+
+logger = logging.getLogger(__name__)
+
+type Publish = Callable[[], object]
+
+_PENDING_KEY = "background_tasks_after_commit"
+_HOOKED_KEY = "background_tasks_after_commit_hooked"
+
+
+def enqueue_after_commit(db: AsyncSession | Session, task: Any, *args: Any, **kwargs: Any) -> str:
+    """Send *task* once *db*'s transaction commits; never if it rolls back.
+
+    *task* is a Celery task (``send_receipt``), a signature
+    (``send_receipt.s().set(queue="mail")``, for routing options), or a
+    registered task name. ``*args`` / ``**kwargs`` are the task's own
+    arguments. Returns the Celery task id the message will carry.
+
+    Publishes as the tenant bound *now*, so a task enqueued in a request runs
+    for that request's tenant even when the commit happens elsewhere.
+    """
+    task_id = str(uuid.uuid4())
+
+    def publish() -> None:
+        if isinstance(task, str):
+            current_app.send_task(task, args=list(args), kwargs=kwargs, task_id=task_id)
+        else:
+            task.apply_async(args=args, kwargs=kwargs, task_id=task_id)
+
+    after_commit(db, [bind_current_tenant(publish)])
+    return task_id
+
+
+def after_commit(
+    db: AsyncSession | Session, publishes: Sequence[Publish], *, offload: bool = False
+) -> None:
+    """Run each of *publishes* after *db* commits, in order. Internal.
+
+    *offload* sends the batch to a thread when the request path runs it: kombu
+    publishes block, and hundreds of them on the event loop stall every other
+    request. Left off for a single publish so Celery's eager mode — which runs
+    the task body inside the publish — stays on the loop's thread. One failed
+    publish is logged and does not stop the rest: the rows are already
+    committed, and an unsent ``pending`` row is visible and retryable.
+    """
+    batch = list(publishes)
+    if isinstance(db, RequestSession) and SESSION_START_KEY in db.info:
+        # A request that wrote nothing else is rolled back and its ``on_commit``
+        # callbacks discarded, which would drop this publish silently. Queueing a
+        # task is itself work that must survive to the commit.
+        mark_written(db)
+        if offload:
+            db.on_commit(lambda: asyncio.to_thread(_run_all, batch))
+        else:
+            db.on_commit(lambda: _run_all(batch))
+        return
+    sync = db.sync_session if isinstance(db, AsyncSession) else db
+    sync.info.setdefault(_PENDING_KEY, []).extend(batch)
+    _hook(sync)
+
+
+def _run_all(batch: Sequence[Publish]) -> None:
+    for publish in batch:
+        try:
+            publish()
+        except Exception:
+            logger.exception("background_tasks.after_commit.publish_failed")
+
+
+def _hook(session: Session) -> None:
+    """Attach the commit/rollback listeners to *session* once."""
+    if session.info.get(_HOOKED_KEY):
+        return
+    session.info[_HOOKED_KEY] = True
+
+    def on_commit(sess: Session) -> None:
+        _run_all(sess.info.pop(_PENDING_KEY, []))
+
+    def on_end(sess: Session, transaction: SessionTransaction) -> None:
+        # ``after_commit`` already took the batch on the commit path; anything
+        # still queued when the outermost transaction ends was rolled back.
+        if transaction.parent is None:
+            sess.info.pop(_PENDING_KEY, None)
+
+    event.listen(session, "after_commit", on_commit)
+    event.listen(session, "after_transaction_end", on_end)
+
+
+__all__ = ["enqueue_after_commit"]

{simple_module_background_tasks-0.0.36 → simple_module_background_tasks-0.0.38}/background_tasks/celery_app.py

@@ -23,12 +23,18 @@ from celery import Celery
 from celery.schedules import schedule
 from simple_module_core.discovery import ENTRY_POINT_GROUP
 
+from background_tasks.config_guards import celery_overrides_from_env
 from background_tasks.constants import (
+    BEAT_PURGE_OLD,
+    BEAT_SWEEP_OUTBOX,
+    BEAT_SWEEP_STUCK,
     INTERNAL_TASK_PURGE_OLD,
+    INTERNAL_TASK_SWEEP_OUTBOX,
     INTERNAL_TASK_SWEEP_STUCK,
     MODULE_NAME,
 )
 from background_tasks.settings import BackgroundTasksSettings
+from background_tasks.task_routing import load_task_configs, merge_routes, select_packages
 
 logger = logging.getLogger(__name__)
 
@@ -86,25 +92,70 @@ def _collect_module_beat_schedules(packages: list[str]) -> dict:
     return merged
 
 
+# setting name -> Celery conf key (they differ only for the memory cap, which
+# Celery names without the unit although it is KiB).
+_TUNING_KEYS = {
+    "worker_max_tasks_per_child": "worker_max_tasks_per_child",
+    "worker_max_memory_per_child_kb": "worker_max_memory_per_child",
+    "task_reject_on_worker_lost": "task_reject_on_worker_lost",
+    "task_soft_time_limit": "task_soft_time_limit",
+    "task_time_limit": "task_time_limit",
+    "worker_concurrency": "worker_concurrency",
+}
+
+
+def _tuning_conf(settings: BackgroundTasksSettings) -> dict:
+    """Celery conf for the optional tuning fields; ``None`` keeps Celery's default."""
+    conf = {
+        key: getattr(settings, name)
+        for name, key in _TUNING_KEYS.items()
+        if getattr(settings, name) is not None
+    }
+    if settings.visibility_timeout_seconds is not None:
+        conf["broker_transport_options"] = {
+            "visibility_timeout": settings.visibility_timeout_seconds
+        }
+    return conf
+
+
+def describe_routes(settings: BackgroundTasksSettings) -> list[dict[str, str]]:
+    """Effective ``[{pattern, queue, module}]`` routes, for the Workers admin page."""
+    configs = load_task_configs(_discover_task_packages())
+    return merge_routes(configs, settings.task_routes_override)[1]
+
+
 def build_celery(settings: BackgroundTasksSettings) -> Celery:
     """Construct a Celery app wired to the project's Redis broker."""
     celery = Celery(MODULE_NAME)
 
-    packages = _discover_task_packages()
+    all_packages = _discover_task_packages()
+    # Routes and beat entries come from the light ``task_config`` of *every*
+    # module; only the heavy ``tasks.py`` imports are narrowed to this
+    # process's subset (GH #450).
+    configs = load_task_configs(all_packages)
+    packages = select_packages(all_packages, configs, settings)
+    task_routes, _ = merge_routes(configs, settings.task_routes_override)
 
-    # The two internal entries always ship; modules contribute more via their
+    # The internal entries always ship; modules contribute more via their
     # own ``tasks.BEAT_SCHEDULE``. ``setdefault`` keeps the built-ins authoritative
     # if a module reuses one of their entry names. See GH #199.
     beat_schedule = {
-        "background-tasks-sweep-stuck": {
+        BEAT_SWEEP_STUCK: {
             "task": INTERNAL_TASK_SWEEP_STUCK,
             "schedule": schedule(settings.stuck_sweep_interval_seconds),
         },
-        "background-tasks-purge-old": {
+        BEAT_PURGE_OLD: {
             "task": INTERNAL_TASK_PURGE_OLD,
             "schedule": schedule(settings.purge_interval_seconds),
         },
+        # Delivers durable events whose post-commit dispatch never ran (a crash
+        # between commit and publish) and retries failed ones (GH #459).
+        BEAT_SWEEP_OUTBOX: {
+            "task": INTERNAL_TASK_SWEEP_OUTBOX,
+            "schedule": schedule(settings.outbox_sweep_interval_seconds),
+        },
     }
+    builtin_beat = frozenset(beat_schedule)
     for name, entry in _collect_module_beat_schedules(packages).items():
         if name in beat_schedule:
             logger.warning(
@@ -112,6 +163,15 @@ def build_celery(settings: BackgroundTasksSettings) -> Celery:
             )
             continue
         beat_schedule[name] = entry
+    # ``task_config.BEAT_SCHEDULE`` wins over the same entry in ``tasks.py``.
+    for pkg in sorted(configs):
+        for name, entry in configs[pkg].beat_schedule.items():
+            if name in builtin_beat:
+                logger.warning(
+                    "task_config beat entry %r clashes with a built-in; built-in wins", name
+                )
+                continue
+            beat_schedule[name] = entry
 
     celery.conf.update(
         broker_url=settings.broker_url,
@@ -138,7 +198,13 @@ def build_celery(settings: BackgroundTasksSettings) -> Celery:
         broker_connection_retry_on_startup=True,
         beat_schedule=beat_schedule,
         beat_scheduler="celery.beat:PersistentScheduler",
+        task_routes=task_routes,
     )
+    celery.conf.update(_tuning_conf(settings))
+    overrides = celery_overrides_from_env()
+    if overrides:
+        logger.info("Applying celery_overrides: %s", sorted(overrides))
+        celery.conf.update(overrides)
 
     # Autodiscover across every installed simple_module — a module just ships
     # a `tasks.py` and the worker picks it up.