simple-module-background-tasks 0.0.35__tar.gz → 0.0.37__tar.gz

{simple_module_background_tasks-0.0.35 → simple_module_background_tasks-0.0.37}/.gitignore

@@ -64,3 +64,4 @@ host/client_app/.playwright-cli/*
 .superpowers/
 .qa/
 .verify/
+/openapi.json

{simple_module_background_tasks-0.0.35 → simple_module_background_tasks-0.0.37}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: simple_module_background_tasks
-Version: 0.0.35
+Version: 0.0.37
 Summary: Celery + Redis task queue with admin UI for monitoring and retrying failed/stuck tasks
 Project-URL: Homepage, https://github.com/antosubash/simple_module_python
 Project-URL: Repository, https://github.com/antosubash/simple_module_python
@@ -23,10 +23,10 @@ Classifier: Typing :: Typed
 Requires-Python: >=3.12
 Requires-Dist: celery[redis]>=5.4
 Requires-Dist: redis>=5
-Requires-Dist: simple-module-core==0.0.35
-Requires-Dist: simple-module-db==0.0.35
-Requires-Dist: simple-module-hosting==0.0.35
-Requires-Dist: simple-module-settings==0.0.35
+Requires-Dist: simple-module-core==0.0.37
+Requires-Dist: simple-module-db==0.0.37
+Requires-Dist: simple-module-hosting==0.0.37
+Requires-Dist: simple-module-settings==0.0.37
 Description-Content-Type: text/markdown
 
 # simple_module_background_tasks
@@ -46,6 +46,8 @@ Requires a Redis broker. The broker / result-backend URLs are module settings (`
 - Zero-config task discovery — any installed module that ships a `tasks.py` has its tasks autodiscovered (`celery.autodiscover_tasks` imports `<package>.tasks` for every installed module). No per-module registration hook.
 - Admin UI at `/admin/background-tasks` — list recent runs, retry failed, inspect tracebacks (gated by the `background_tasks.view` permission).
 - `build_celery(settings)` factory in `background_tasks.celery_app`, plus `bind_task_context` / `get_log_context` / `install_log_filter` exported from the package root (import name `background_tasks`, distribution name `simple_module_background_tasks`).
+- `enqueue_after_commit(db, task, *args, **kwargs)` — publish a task only once the session's transaction commits, never on rollback.
+- `heartbeat()` — called from a long task body, keeps the stuck sweep from flagging it; `@shared_task(stuck_after=seconds)` sets a per-task stuck threshold. See the module docs for both.
 
 ## Usage
 
@@ -80,6 +82,27 @@ uv run celery -A scripts.run_worker:celery worker -l info
 uv run celery -A scripts.run_worker:celery beat   -l info
 ```
 
+## Routing, tuning and per-worker module subsets
+
+A module can ship a light `task_config.py` (`TASK_ROUTES`, `QUEUES`, optional
+`BEAT_SCHEDULE`) that is read without importing `tasks.py`. The
+`task_routes_override` setting re-routes tasks at runtime, and the Workers page
+lists the effective routes. Tuning settings (`visibility_timeout_seconds`,
+`worker_max_tasks_per_child`, `task_time_limit`, `worker_concurrency`, ...) and a
+raw `SM_BG_TASKS_CELERY_OVERRIDES` (env only, unsafe keys refused) default to Celery's own defaults.
+`SM_BG_TASKS_WORKER_QUEUES=gpu,default make worker` consumes only those queues
+and imports only the modules whose `QUEUES` overlap (see `SM_BG_TASKS_INCLUDE_MODULES` /
+`SM_BG_TASKS_EXCLUDE_MODULES`; env only, not DB settings). Full details in `docs/modules/background_tasks.md`.
+
+## Durable events
+
+This module is the transport behind `bus.publish_durable(event, db)`: the event
+is written to an outbox table in the caller's transaction and delivered by a
+worker to every `bus.subscribe_durable(...)` handler, at least once, with
+backoff, after the commit. Writes a handler makes through `ctx.db` land exactly
+once. Undelivered and dead rows are on `/admin/background-tasks/outbox`. Full
+details in `docs/modules/background_tasks.md` § Durable events.
+
 ## Worker log context
 
 Every worker log line automatically carries the Celery task identifiers

{simple_module_background_tasks-0.0.35 → simple_module_background_tasks-0.0.37}/README.md

@@ -15,6 +15,8 @@ Requires a Redis broker. The broker / result-backend URLs are module settings (`
 - Zero-config task discovery — any installed module that ships a `tasks.py` has its tasks autodiscovered (`celery.autodiscover_tasks` imports `<package>.tasks` for every installed module). No per-module registration hook.
 - Admin UI at `/admin/background-tasks` — list recent runs, retry failed, inspect tracebacks (gated by the `background_tasks.view` permission).
 - `build_celery(settings)` factory in `background_tasks.celery_app`, plus `bind_task_context` / `get_log_context` / `install_log_filter` exported from the package root (import name `background_tasks`, distribution name `simple_module_background_tasks`).
+- `enqueue_after_commit(db, task, *args, **kwargs)` — publish a task only once the session's transaction commits, never on rollback.
+- `heartbeat()` — called from a long task body, keeps the stuck sweep from flagging it; `@shared_task(stuck_after=seconds)` sets a per-task stuck threshold. See the module docs for both.
 
 ## Usage
 
@@ -49,6 +51,27 @@ uv run celery -A scripts.run_worker:celery worker -l info
 uv run celery -A scripts.run_worker:celery beat   -l info
 ```
 
+## Routing, tuning and per-worker module subsets
+
+A module can ship a light `task_config.py` (`TASK_ROUTES`, `QUEUES`, optional
+`BEAT_SCHEDULE`) that is read without importing `tasks.py`. The
+`task_routes_override` setting re-routes tasks at runtime, and the Workers page
+lists the effective routes. Tuning settings (`visibility_timeout_seconds`,
+`worker_max_tasks_per_child`, `task_time_limit`, `worker_concurrency`, ...) and a
+raw `SM_BG_TASKS_CELERY_OVERRIDES` (env only, unsafe keys refused) default to Celery's own defaults.
+`SM_BG_TASKS_WORKER_QUEUES=gpu,default make worker` consumes only those queues
+and imports only the modules whose `QUEUES` overlap (see `SM_BG_TASKS_INCLUDE_MODULES` /
+`SM_BG_TASKS_EXCLUDE_MODULES`; env only, not DB settings). Full details in `docs/modules/background_tasks.md`.
+
+## Durable events
+
+This module is the transport behind `bus.publish_durable(event, db)`: the event
+is written to an outbox table in the caller's transaction and delivered by a
+worker to every `bus.subscribe_durable(...)` handler, at least once, with
+backoff, after the commit. Writes a handler makes through `ctx.db` land exactly
+once. Undelivered and dead rows are on `/admin/background-tasks/outbox`. Full
+details in `docs/modules/background_tasks.md` § Durable events.
+
 ## Worker log context
 
 Every worker log line automatically carries the Celery task identifiers

{simple_module_background_tasks-0.0.35 → simple_module_background_tasks-0.0.37}/background_tasks/__init__.py

@@ -1,13 +1,19 @@
 """BackgroundTasks module — Celery + Redis task queue with admin UI."""
 
+from background_tasks.after_commit import enqueue_after_commit
 from background_tasks.log_context import (
     bind_task_context,
     get_log_context,
     install_log_filter,
 )
+from background_tasks.task_heartbeat import heartbeat
+from background_tasks.worker_settings import settings_for
 
 __all__ = [
     "bind_task_context",
+    "enqueue_after_commit",
     "get_log_context",
+    "heartbeat",
     "install_log_filter",
+    "settings_for",
 ]

simple_module_background_tasks-0.0.37/background_tasks/_signal_bus.py

@@ -0,0 +1,61 @@
+"""Bridge from Celery's sync signal thread to the app's async event bus.
+
+Split out of :mod:`.signals` to keep it under the 300-line cap; ``signals``
+re-exports :func:`bind_event_bus` / :func:`unbind_event_bus`.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+from typing import Any
+
+from simple_module_core.events import EventBus
+
+logger = logging.getLogger("background_tasks.signals")
+
+
+_bus: EventBus | None = None
+_loop: asyncio.AbstractEventLoop | None = None
+
+
+def bind_event_bus(bus: EventBus, loop: asyncio.AbstractEventLoop) -> None:
+    """Bind an event bus + its running loop so signals can publish events.
+
+    Signals fire on the Celery sync thread; `run_coroutine_threadsafe`
+    bridges back to ``loop`` so handlers run on the API event loop
+    regardless of which thread triggered the signal.
+    """
+    global _bus, _loop
+    _bus = bus
+    _loop = loop
+
+
+def unbind_event_bus() -> None:
+    """Drop the bound bus — called from ``on_shutdown`` so tests stay isolated."""
+    global _bus, _loop
+    _bus = None
+    _loop = None
+
+
+def publish_from_signal(event: Any) -> None:
+    """Dispatch ``event`` onto the bound bus without blocking the signal thread."""
+    if _bus is None or _loop is None:
+        return
+    try:
+        future = asyncio.run_coroutine_threadsafe(_bus.publish(event), _loop)
+    except RuntimeError:
+        # Loop has stopped (shutdown race). The DB row is already written.
+        logger.debug("Event bus loop is not running; skipping %s", type(event).__name__)
+        return
+    # Surface subscriber exceptions — run_coroutine_threadsafe otherwise only
+    # logs them when the Future is GC'd, which happens far from the failure.
+    future.add_done_callback(_log_publish_failure)
+
+
+def _log_publish_failure(future: asyncio.Future[Any]) -> None:
+    if future.cancelled():
+        return
+    exc = future.exception()
+    if exc is not None:
+        logger.error("Event publish raised: %s", exc, exc_info=exc)

simple_module_background_tasks-0.0.37/background_tasks/after_commit.py

@@ -0,0 +1,126 @@
+"""Publish to the broker only once the transaction that justifies it commits.
+
+A task enqueued inside a transaction races that transaction: the worker can
+pick the message up before the row it refers to is visible (eager mode runs
+it *inside* ``send_task``), and if the commit then fails the task is already
+queued against nothing. Both are fixed the same way — hold the publish until
+the commit has landed, and drop it if the transaction rolls back (GH #452).
+
+Two delivery paths, picked per session:
+
+- **A request's session** (``get_db``) — the publish goes through
+  :meth:`~simple_module_db.RequestSession.on_commit`, which the framework runs
+  after the request's commit and before the response is sent, and discards on
+  rollback or commit failure.
+- **Any other session** — a worker's sync session, a CLI script's own
+  ``async with session.begin()`` — has nobody to run ``on_commit`` callbacks,
+  so the publish hangs off SQLAlchemy's ``after_commit`` event instead and is
+  dropped when the outermost transaction ends without committing.
+
+The broker id is reserved up front and returned, so the caller can write it
+onto its own row in the same transaction: the row names the task before the
+task exists, never the other way round.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import logging
+import uuid
+from collections.abc import Callable, Sequence
+from typing import Any
+
+from celery import current_app
+from simple_module_db import RequestSession, bind_current_tenant, mark_written
+from simple_module_db.transaction import SESSION_START_KEY
+from sqlalchemy import event
+from sqlalchemy.ext.asyncio import AsyncSession
+from sqlalchemy.orm import Session, SessionTransaction
+
+logger = logging.getLogger(__name__)
+
+type Publish = Callable[[], object]
+
+_PENDING_KEY = "background_tasks_after_commit"
+_HOOKED_KEY = "background_tasks_after_commit_hooked"
+
+
+def enqueue_after_commit(db: AsyncSession | Session, task: Any, *args: Any, **kwargs: Any) -> str:
+    """Send *task* once *db*'s transaction commits; never if it rolls back.
+
+    *task* is a Celery task (``send_receipt``), a signature
+    (``send_receipt.s().set(queue="mail")``, for routing options), or a
+    registered task name. ``*args`` / ``**kwargs`` are the task's own
+    arguments. Returns the Celery task id the message will carry.
+
+    Publishes as the tenant bound *now*, so a task enqueued in a request runs
+    for that request's tenant even when the commit happens elsewhere.
+    """
+    task_id = str(uuid.uuid4())
+
+    def publish() -> None:
+        if isinstance(task, str):
+            current_app.send_task(task, args=list(args), kwargs=kwargs, task_id=task_id)
+        else:
+            task.apply_async(args=args, kwargs=kwargs, task_id=task_id)
+
+    after_commit(db, [bind_current_tenant(publish)])
+    return task_id
+
+
+def after_commit(
+    db: AsyncSession | Session, publishes: Sequence[Publish], *, offload: bool = False
+) -> None:
+    """Run each of *publishes* after *db* commits, in order. Internal.
+
+    *offload* sends the batch to a thread when the request path runs it: kombu
+    publishes block, and hundreds of them on the event loop stall every other
+    request. Left off for a single publish so Celery's eager mode — which runs
+    the task body inside the publish — stays on the loop's thread. One failed
+    publish is logged and does not stop the rest: the rows are already
+    committed, and an unsent ``pending`` row is visible and retryable.
+    """
+    batch = list(publishes)
+    if isinstance(db, RequestSession) and SESSION_START_KEY in db.info:
+        # A request that wrote nothing else is rolled back and its ``on_commit``
+        # callbacks discarded, which would drop this publish silently. Queueing a
+        # task is itself work that must survive to the commit.
+        mark_written(db)
+        if offload:
+            db.on_commit(lambda: asyncio.to_thread(_run_all, batch))
+        else:
+            db.on_commit(lambda: _run_all(batch))
+        return
+    sync = db.sync_session if isinstance(db, AsyncSession) else db
+    sync.info.setdefault(_PENDING_KEY, []).extend(batch)
+    _hook(sync)
+
+
+def _run_all(batch: Sequence[Publish]) -> None:
+    for publish in batch:
+        try:
+            publish()
+        except Exception:
+            logger.exception("background_tasks.after_commit.publish_failed")
+
+
+def _hook(session: Session) -> None:
+    """Attach the commit/rollback listeners to *session* once."""
+    if session.info.get(_HOOKED_KEY):
+        return
+    session.info[_HOOKED_KEY] = True
+
+    def on_commit(sess: Session) -> None:
+        _run_all(sess.info.pop(_PENDING_KEY, []))
+
+    def on_end(sess: Session, transaction: SessionTransaction) -> None:
+        # ``after_commit`` already took the batch on the commit path; anything
+        # still queued when the outermost transaction ends was rolled back.
+        if transaction.parent is None:
+            sess.info.pop(_PENDING_KEY, None)
+
+    event.listen(session, "after_commit", on_commit)
+    event.listen(session, "after_transaction_end", on_end)
+
+
+__all__ = ["enqueue_after_commit"]

{simple_module_background_tasks-0.0.35 → simple_module_background_tasks-0.0.37}/background_tasks/celery_app.py

@@ -23,12 +23,18 @@ from celery import Celery
 from celery.schedules import schedule
 from simple_module_core.discovery import ENTRY_POINT_GROUP
 
+from background_tasks.config_guards import celery_overrides_from_env
 from background_tasks.constants import (
+    BEAT_PURGE_OLD,
+    BEAT_SWEEP_OUTBOX,
+    BEAT_SWEEP_STUCK,
     INTERNAL_TASK_PURGE_OLD,
+    INTERNAL_TASK_SWEEP_OUTBOX,
     INTERNAL_TASK_SWEEP_STUCK,
     MODULE_NAME,
 )
 from background_tasks.settings import BackgroundTasksSettings
+from background_tasks.task_routing import load_task_configs, merge_routes, select_packages
 
 logger = logging.getLogger(__name__)
 
@@ -86,25 +92,70 @@ def _collect_module_beat_schedules(packages: list[str]) -> dict:
     return merged
 
 
+# setting name -> Celery conf key (they differ only for the memory cap, which
+# Celery names without the unit although it is KiB).
+_TUNING_KEYS = {
+    "worker_max_tasks_per_child": "worker_max_tasks_per_child",
+    "worker_max_memory_per_child_kb": "worker_max_memory_per_child",
+    "task_reject_on_worker_lost": "task_reject_on_worker_lost",
+    "task_soft_time_limit": "task_soft_time_limit",
+    "task_time_limit": "task_time_limit",
+    "worker_concurrency": "worker_concurrency",
+}
+
+
+def _tuning_conf(settings: BackgroundTasksSettings) -> dict:
+    """Celery conf for the optional tuning fields; ``None`` keeps Celery's default."""
+    conf = {
+        key: getattr(settings, name)
+        for name, key in _TUNING_KEYS.items()
+        if getattr(settings, name) is not None
+    }
+    if settings.visibility_timeout_seconds is not None:
+        conf["broker_transport_options"] = {
+            "visibility_timeout": settings.visibility_timeout_seconds
+        }
+    return conf
+
+
+def describe_routes(settings: BackgroundTasksSettings) -> list[dict[str, str]]:
+    """Effective ``[{pattern, queue, module}]`` routes, for the Workers admin page."""
+    configs = load_task_configs(_discover_task_packages())
+    return merge_routes(configs, settings.task_routes_override)[1]
+
+
 def build_celery(settings: BackgroundTasksSettings) -> Celery:
     """Construct a Celery app wired to the project's Redis broker."""
     celery = Celery(MODULE_NAME)
 
-    packages = _discover_task_packages()
+    all_packages = _discover_task_packages()
+    # Routes and beat entries come from the light ``task_config`` of *every*
+    # module; only the heavy ``tasks.py`` imports are narrowed to this
+    # process's subset (GH #450).
+    configs = load_task_configs(all_packages)
+    packages = select_packages(all_packages, configs, settings)
+    task_routes, _ = merge_routes(configs, settings.task_routes_override)
 
-    # The two internal entries always ship; modules contribute more via their
+    # The internal entries always ship; modules contribute more via their
     # own ``tasks.BEAT_SCHEDULE``. ``setdefault`` keeps the built-ins authoritative
     # if a module reuses one of their entry names. See GH #199.
     beat_schedule = {
-        "background-tasks-sweep-stuck": {
+        BEAT_SWEEP_STUCK: {
             "task": INTERNAL_TASK_SWEEP_STUCK,
             "schedule": schedule(settings.stuck_sweep_interval_seconds),
         },
-        "background-tasks-purge-old": {
+        BEAT_PURGE_OLD: {
             "task": INTERNAL_TASK_PURGE_OLD,
             "schedule": schedule(settings.purge_interval_seconds),
         },
+        # Delivers durable events whose post-commit dispatch never ran (a crash
+        # between commit and publish) and retries failed ones (GH #459).
+        BEAT_SWEEP_OUTBOX: {
+            "task": INTERNAL_TASK_SWEEP_OUTBOX,
+            "schedule": schedule(settings.outbox_sweep_interval_seconds),
+        },
     }
+    builtin_beat = frozenset(beat_schedule)
     for name, entry in _collect_module_beat_schedules(packages).items():
         if name in beat_schedule:
             logger.warning(
@@ -112,6 +163,15 @@ def build_celery(settings: BackgroundTasksSettings) -> Celery:
             )
             continue
         beat_schedule[name] = entry
+    # ``task_config.BEAT_SCHEDULE`` wins over the same entry in ``tasks.py``.
+    for pkg in sorted(configs):
+        for name, entry in configs[pkg].beat_schedule.items():
+            if name in builtin_beat:
+                logger.warning(
+                    "task_config beat entry %r clashes with a built-in; built-in wins", name
+                )
+                continue
+            beat_schedule[name] = entry
 
     celery.conf.update(
         broker_url=settings.broker_url,
@@ -138,7 +198,13 @@ def build_celery(settings: BackgroundTasksSettings) -> Celery:
         broker_connection_retry_on_startup=True,
         beat_schedule=beat_schedule,
         beat_scheduler="celery.beat:PersistentScheduler",
+        task_routes=task_routes,
     )
+    celery.conf.update(_tuning_conf(settings))
+    overrides = celery_overrides_from_env()
+    if overrides:
+        logger.info("Applying celery_overrides: %s", sorted(overrides))
+        celery.conf.update(overrides)
 
     # Autodiscover across every installed simple_module — a module just ships
     # a `tasks.py` and the worker picks it up.

simple_module_background_tasks-0.0.37/background_tasks/config_guards.py

@@ -0,0 +1,220 @@
+"""Guards for operator-supplied Celery configuration.
+
+Celery conf can name serializers (``pickle``), dotted paths it imports on boot
+(``imports``, ``worker_pool``, ``beat_scheduler`` ...) and signing material, so
+a raw conf dict is a code-execution surface for whoever can write it. Two
+inputs reach ``build_celery``:
+
+- ``SM_BG_TASKS_CELERY_OVERRIDES`` (JSON object): environment only, so it is
+  controlled by whoever deploys the process and never by an admin-UI editor.
+  It is read here rather than being a ``BackgroundTasksSettings`` field, because
+  every field of that class is stored in, and hydrated from, the database.
+  Dangerous keys are refused even from here.
+- ``SM_BG_TASKS_WORKER_QUEUES`` / ``_INCLUDE_MODULES`` / ``_EXCLUDE_MODULES``
+  (comma-separated): which queues and modules *this one worker process*
+  serves. Environment only for the same reason: one stored value would
+  re-scope every worker. All three go through the helpers below.
+- ``task_routes_override`` (DB-backed, admin-editable): only names queues, and
+  is reduced to plain queue names and route dicts with a closed key set. A route
+  dict is merged into ``apply_async`` options, so ``serializer``, ``link`` and
+  friends must not be expressible.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import os
+import re
+from dataclasses import dataclass
+from typing import Any
+
+from background_tasks.constants import ENV_PREFIX
+
+logger = logging.getLogger(__name__)
+
+ENV_CELERY_OVERRIDES = f"{ENV_PREFIX}CELERY_OVERRIDES"
+ENV_WORKER_QUEUES = f"{ENV_PREFIX}WORKER_QUEUES"
+ENV_INCLUDE_MODULES = f"{ENV_PREFIX}INCLUDE_MODULES"
+ENV_EXCLUDE_MODULES = f"{ENV_PREFIX}EXCLUDE_MODULES"
+
+# Conf keys that choose a (de)serializer or load code by dotted path / file.
+_DENIED_KEYS = frozenset(
+    {
+        "accept_content",
+        "task_serializer",
+        "result_serializer",
+        "result_accept_content",
+        "event_serializer",
+        "imports",
+        "include",
+        "worker_pool",
+        "worker_autoscaler",
+        "worker_consumer",
+        "worker_timer",
+        "beat_scheduler",
+        "beat_schedule",
+        "beat_schedule_filename",
+        "task_routes",
+        "task_annotations",
+        "task_queues",
+        "task_cls",
+        "broker_transport",
+        "result_backend_transport_options",
+        "worker_state_db",
+    }
+)
+_DENIED_SUBSTRINGS = ("hijack", "serializer", "accept_content", "pickle")
+_DENIED_PREFIXES = ("security_", "worker_pool_", "pool_")
+_DENIED_SUFFIXES = ("_cls", "_class")
+
+
+def is_denied_key(key: str) -> bool:
+    """True for a conf key that must not be settable from configuration."""
+    # Celery also resolves the old ``CELERY_*`` spellings.
+    k = key.strip().lower().removeprefix("celery_")
+    return (
+        k in _DENIED_KEYS
+        or k.startswith(_DENIED_PREFIXES)
+        or k.endswith(_DENIED_SUFFIXES)
+        or any(s in k for s in _DENIED_SUBSTRINGS)
+    )
+
+
+def filter_celery_overrides(raw: object, *, source: str = "celery_overrides") -> dict[str, Any]:
+    """Drop (and log) denied keys; non-dict input yields ``{}``."""
+    if not isinstance(raw, dict):
+        if raw not in (None, {}):
+            logger.error("%s must be a JSON object; ignoring it", source)
+        return {}
+    safe: dict[str, Any] = {}
+    for key, value in raw.items():
+        if not isinstance(key, str) or is_denied_key(key):
+            logger.error(
+                "%s: refusing Celery conf key %r (serializer, import and code-loading "
+                "keys cannot be set from configuration); ignoring it",
+                source,
+                key,
+            )
+            continue
+        safe[key] = value
+    return safe
+
+
+def _env_text(name: str) -> str:
+    return os.environ.get(name, "").strip()
+
+
+def env_csv(name: str) -> list[str]:
+    """A comma-separated environment variable as a list (``[]`` when unset)."""
+    return [part.strip() for part in _env_text(name).split(",") if part.strip()]
+
+
+def env_json_object(name: str) -> dict[str, Any]:
+    """A JSON-object environment variable (``{}`` when unset or unusable, with a log)."""
+    raw = _env_text(name)
+    if not raw:
+        return {}
+    try:
+        parsed = json.loads(raw)
+    except ValueError as exc:
+        logger.error("%s is not valid JSON (%s); ignoring it", name, exc)
+        return {}
+    if not isinstance(parsed, dict):
+        logger.error("%s must be a JSON object; ignoring it", name)
+        return {}
+    return parsed
+
+
+def celery_overrides_from_env() -> dict[str, Any]:
+    """The vetted ``SM_BG_TASKS_CELERY_OVERRIDES`` mapping (``{}`` when unset)."""
+    return filter_celery_overrides(
+        env_json_object(ENV_CELERY_OVERRIDES), source=ENV_CELERY_OVERRIDES
+    )
+
+
+@dataclass(frozen=True)
+class WorkerScope:
+    """What this process serves; see :func:`worker_scope_from_env`."""
+
+    queues: frozenset[str] = frozenset()
+    include_modules: frozenset[str] = frozenset()
+    exclude_modules: frozenset[str] = frozenset()
+
+
+def worker_scope_from_env() -> WorkerScope:
+    """Per-process queue/module subset (GH #450), from the environment only."""
+    return WorkerScope(
+        queues=frozenset(env_csv(ENV_WORKER_QUEUES)),
+        include_modules=frozenset(env_csv(ENV_INCLUDE_MODULES)),
+        exclude_modules=frozenset(env_csv(ENV_EXCLUDE_MODULES)),
+    )
+
+
+_QUEUE_RE = re.compile(r"^[A-Za-z0-9_.:\-]{1,255}$")
+_ROUTE_KEYS = frozenset({"queue", "exchange", "routing_key", "priority"})
+
+
+def _safe_route(value: object) -> object | None:
+    if isinstance(value, str):
+        return value if _QUEUE_RE.match(value) else None
+    if not isinstance(value, dict) or set(value) - _ROUTE_KEYS:
+        return None
+    queue = value.get("queue")
+    if not isinstance(queue, str) or not _QUEUE_RE.match(queue):
+        return None
+    for key in ("exchange", "routing_key"):
+        if key in value and not (isinstance(value[key], str) and _QUEUE_RE.match(value[key])):
+            return None
+    priority = value.get("priority")
+    if "priority" in value and (isinstance(priority, bool) or not isinstance(priority, int)):
+        return None
+    return dict(value)
+
+
+def sanitize_task_routes(raw: object) -> dict[str, Any]:
+    """Keep only ``{pattern: queue-name | {queue, exchange, routing_key, priority}}``."""
+    if not isinstance(raw, dict):
+        return {}
+    clean: dict[str, Any] = {}
+    for pattern, value in raw.items():
+        route = _safe_route(value)
+        if not isinstance(pattern, str) or not pattern.strip() or route is None:
+            logger.error(
+                "task_routes_override: refusing entry %r (needs a task pattern and a queue "
+                "name or a route dict using only %s); ignoring it",
+                pattern,
+                sorted(_ROUTE_KEYS),
+            )
+            continue
+        clean[pattern] = route
+    return clean
+
+
+def parse_task_routes(raw: object) -> dict[str, Any]:
+    """Strict form of :func:`sanitize_task_routes` for user input.
+
+    Accepts a mapping or a JSON string of one (what the admin settings text box
+    submits) and raises ``ValueError`` naming the first bad entry instead of
+    silently dropping it, so the editor can show why a save was refused.
+    """
+    if isinstance(raw, str):
+        if not raw.strip():
+            return {}
+        try:
+            raw = json.loads(raw)
+        except ValueError as exc:
+            raise ValueError(f"task_routes_override is not valid JSON: {exc}") from exc
+    if raw is None:
+        return {}
+    if not isinstance(raw, dict):
+        raise ValueError(
+            'task_routes_override must be a JSON object such as {"reports.*": "heavy"}'
+        )
+    for pattern, value in raw.items():
+        if not isinstance(pattern, str) or not pattern.strip() or _safe_route(value) is None:
+            raise ValueError(
+                f"task_routes_override entry {pattern!r} needs a task pattern and a queue "
+                f"name or a route dict using only {sorted(_ROUTE_KEYS)}"
+            )
+    return dict(raw)