django-lambda-tasks 0.4.8__tar.gz → 0.4.9__tar.gz

{django_lambda_tasks-0.4.8 → django_lambda_tasks-0.4.9}/.kiro/steering/product.md

@@ -246,7 +246,16 @@ Implementation lives in `lambda_tasks/local_executor.py`:
 - `get_pool()` — lazily creates and returns the shared `ProcessPoolExecutor`
 - `submit_task(*, message_json: str)` — generates a UUID4 message_id and submits to the pool
 - `_execute_in_worker(*, message_json: str, message_id: str)` — worker entry point; deserializes and calls `execute_immediately()`
-- `_pool_initializer()` — calls `django.setup()` once per worker
+- `_pool_initializer()` — calls `django.setup()` once per worker and sets `SIGINT` to `SIG_IGN` so workers ignore Ctrl+C
+- `_install_shutdown_handlers()` — installs main-thread `SIGINT`/`SIGTERM` handlers that release the pool, then chain to the previous handler
+
+### Shutdown and the Ctrl+C semaphore-leak race
+
+`runserver` runs the development server in an autoreloader **child** process spawned via `subprocess.run()`. On Ctrl+C the terminal delivers `SIGINT` to the whole process group; the autoreloader **parent** unwinds out of `subprocess.run` and immediately calls `process.kill()` (`SIGKILL`) on the child. The pool's POSIX semaphores must be unlinked before that `SIGKILL` lands, otherwise multiprocessing's `resource_tracker` prints `There appear to be N leaked semaphore objects to clean up at shutdown`.
+
+Relying on `atexit` alone loses this race in applications with a heavy shutdown sequence (many `atexit` handlers, open DB connections), because `atexit` runs only after the full interpreter unwind — `SIGKILL` arrives first. To win the race, `LambdaTasksConfig.ready()` calls `_install_shutdown_handlers()` when `LOCAL_WORKERS > 0`. The handler shuts the pool down as its **first** action, then chains to the previously installed handler so normal shutdown behaviour (`KeyboardInterrupt`, autoreloader exit) is preserved. `atexit` registration remains as a fallback for non-signal exits. Handler installation is idempotent and only effective on the main thread (`signal.signal` raises off the main thread, in which case it is a no-op).
+
+`_shutdown_pool()` does **not** use `pool.shutdown(wait=True)`. `wait=True` blocks on joining the worker processes, and a worker that ran a heavy `django.setup()` is slow to exit — `concurrent.futures` does not unlink the pool's queue semaphores until the workers actually die, so the parent's near-instant `SIGKILL` still wins and the semaphores leak. Instead `_shutdown_pool()` **terminates the worker processes first** (a near-instant signal to children we own), then calls `pool.shutdown(wait=False, cancel_futures=True)`. With the workers already gone, the queue semaphores are unlinked in milliseconds — fast enough to beat the parent's `SIGKILL`.
 
 ## Logging
 

{django_lambda_tasks-0.4.8 → django_lambda_tasks-0.4.9}/.kiro/steering/structure.md

@@ -41,13 +41,14 @@ django-lambda-tasks/
 
 - `decorators.py` — defines `@lambda_task`; enforces kwargs-only at decoration time
 - `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
-- `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker
+- `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker (and sets `SIGINT` to `SIG_IGN` so workers ignore Ctrl+C); `_install_shutdown_handlers()` installs main-thread `SIGINT`/`SIGTERM` handlers that release the pool before chaining to the previous handler
 - `handler.py` — Lambda entry point; cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
 - `environment_loader.py` — loads env vars from a Secrets Manager secret at cold start; validates flat JSON format; idempotent via `_loaded` sentinel
 - `secret_loader.py` — resolves `LAMBDA_TASKS_SECRET_*` env vars from Secrets Manager before Django starts; validates format, detects conflicts, batches API calls; idempotent via `_loaded` sentinel
 - `logging.py` — `task_logger` singleton; `message_id` set/cleared around each task execution
 - `settings.py` — `LambdaTasksSettings` instantiated fresh per use (reads live Django settings)
 - `admin.py` — Django admin registration for `TaskRecord`
+- `apps.py` — Django `AppConfig`; `ready()` installs the local-executor shutdown signal handlers when `LOCAL_WORKERS > 0`
 - `tasks.py` — built-in maintenance tasks; `cleanup_task_records` deletes old `TaskRecord` rows
 
 ## Conventions

{django_lambda_tasks-0.4.8 → django_lambda_tasks-0.4.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: django-lambda-tasks
-Version: 0.4.8
+Version: 0.4.9
 Summary: Run async tasks in a lambda function
 Requires-Python: >=3.10
 Requires-Dist: awslambdaric

django_lambda_tasks-0.4.9/lambda_tasks/apps.py

@@ -0,0 +1,22 @@
+"""Django AppConfig for the lambda_tasks library."""
+
+from django.apps import AppConfig
+
+
+class LambdaTasksConfig(AppConfig):
+    name = "lambda_tasks"
+    verbose_name = "Lambda Tasks"
+
+    def ready(self) -> None:
+        """Install pool shutdown signal handlers in async-local mode.
+
+        Only relevant when LOCAL_WORKERS > 0 (development async execution). In
+        that mode the process pool's POSIX semaphores must be released promptly
+        on Ctrl+C, before Django's autoreloader parent SIGKILLs this child. See
+        ``local_executor._install_shutdown_handlers`` for the full rationale.
+        """
+        from lambda_tasks.local_executor import _install_shutdown_handlers
+        from lambda_tasks.settings import LambdaTasksSettings
+
+        if LambdaTasksSettings().LOCAL_WORKERS > 0:
+            _install_shutdown_handlers()

django_lambda_tasks-0.4.9/lambda_tasks/local_executor.py

@@ -0,0 +1,151 @@
+"""Process pool executor for async local task execution."""
+
+import atexit
+import logging
+import signal
+import threading
+import uuid
+from concurrent.futures import Future, ProcessPoolExecutor
+
+from lambda_tasks.settings import LambdaTasksSettings
+
+logger = logging.getLogger(__name__)
+
+_pool: ProcessPoolExecutor | None = None
+_handlers_installed: bool = False
+
+
+def _pool_initializer() -> None:
+    """Run once per worker process.
+
+    Ignores SIGINT so that Ctrl+C is handled exclusively by the parent
+    process, which releases the pool via its SIGINT/SIGTERM handler (and
+    atexit as a fallback). This prevents workers from being killed
+    mid-operation and leaking semaphores.
+
+    Then sets up Django for task execution.
+    """
+    signal.signal(signal.SIGINT, signal.SIG_IGN)
+
+    import django
+
+    django.setup()
+
+
+def _shutdown_pool() -> None:
+    """Shut down the pool, releasing its POSIX semaphores promptly.
+
+    On shutdown we must unlink the pool's semaphores quickly, because under
+    ``runserver`` the autoreloader parent SIGKILLs this child almost immediately
+    after Ctrl+C (see ``_install_shutdown_handlers``). ``pool.shutdown(wait=True)``
+    blocks on joining the worker processes — if a worker is slow to exit (e.g.
+    it ran a heavy ``django.setup()``), the semaphores are not unlinked until
+    the worker dies, and the SIGKILL wins the race, leaking the semaphores.
+
+    To avoid that, terminate the worker processes first (a near-instant
+    SIGTERM/SIGKILL to children we own), then shut the pool down without
+    waiting. With the workers already gone, ``concurrent.futures`` releases the
+    queue semaphores immediately.
+    """
+    global _pool
+    if _pool is not None:
+        pool = _pool
+        _pool = None
+        processes = getattr(pool, "_processes", None)
+        if processes:
+            for process in list(processes.values()):
+                process.terminate()
+        pool.shutdown(wait=False, cancel_futures=True)
+
+
+def _install_shutdown_handlers() -> None:
+    """Install SIGINT/SIGTERM handlers that release the pool promptly.
+
+    Under Django's ``runserver`` autoreloader the development server runs in a
+    child process spawned by ``subprocess.run()``. On Ctrl+C the terminal
+    delivers SIGINT to the whole process group; the autoreloader parent unwinds
+    out of ``subprocess.run`` and immediately calls ``process.kill()``
+    (SIGKILL) on this child. That is a race: this process must unlink the
+    pool's POSIX semaphores before the SIGKILL lands, otherwise multiprocessing's
+    ``resource_tracker`` reports them as leaked at shutdown.
+
+    Relying on ``atexit`` loses that race in applications with a heavy shutdown
+    sequence, because ``atexit`` runs only after the full interpreter unwind.
+    Instead we shut the pool down as the very first action of the signal
+    handler, then chain to the previously installed handler so normal shutdown
+    behaviour (KeyboardInterrupt, autoreloader exit) is preserved.
+
+    Idempotent and only effective on the main thread — ``signal.signal`` raises
+    ``ValueError`` off the main thread, in which case this is a no-op.
+    """
+    global _handlers_installed
+    if _handlers_installed:
+        return
+
+    if threading.current_thread() is not threading.main_thread():
+        return
+
+    def make_handler(previous):  # type: ignore[no-untyped-def]
+        def handler(signum, frame):  # type: ignore[no-untyped-def]
+            # Release the pool's semaphores first, before the autoreloader
+            # parent can SIGKILL us.
+            _shutdown_pool()
+            if callable(previous):
+                previous(signum, frame)
+            elif previous == signal.SIG_DFL:
+                signal.signal(signum, signal.SIG_DFL)
+                signal.raise_signal(signum)
+            # previous == SIG_IGN: nothing to chain to.
+
+        return handler
+
+    try:
+        for signum in (signal.SIGINT, signal.SIGTERM):
+            previous = signal.getsignal(signum)
+            signal.signal(signum, make_handler(previous))
+    except ValueError:
+        # Not on the main thread; cannot install signal handlers here.
+        return
+
+    _handlers_installed = True
+
+
+def get_pool() -> ProcessPoolExecutor:
+    """Return the shared ProcessPoolExecutor, creating it on first call."""
+    global _pool
+    if _pool is None:
+        conf = LambdaTasksSettings()
+        _pool = ProcessPoolExecutor(
+            max_workers=conf.LOCAL_WORKERS,
+            initializer=_pool_initializer,
+        )
+        atexit.register(_shutdown_pool)
+    return _pool
+
+
+def _execute_in_worker(*, message_json: str, message_id: str) -> None:
+    """Worker entry point. Deserializes and executes the task.
+
+    Runs in a child process. Django is already set up via the pool initializer.
+    """
+    from lambda_tasks.models import SQSLambdaTaskMessage
+
+    message = SQSLambdaTaskMessage.model_validate_json(message_json)
+    message.execute_immediately(message_id=message_id)
+
+
+def _log_worker_exception(future: Future) -> None:  # type: ignore[type-arg]
+    """Callback attached to each worker future. Logs unhandled exceptions."""
+    exception = future.exception()
+    if exception is not None:
+        logger.error("Worker process raised an exception", exc_info=exception)
+
+
+def submit_task(*, message_json: str) -> None:
+    """Submit a task to the process pool. Fire-and-forget."""
+    pool = get_pool()
+    message_id = str(uuid.uuid4())
+    future = pool.submit(
+        _execute_in_worker, message_json=message_json, message_id=message_id
+    )
+    future.add_done_callback(_log_worker_exception)

{django_lambda_tasks-0.4.8 → django_lambda_tasks-0.4.9}/pyproject.toml

@@ -7,7 +7,7 @@ packages = ["lambda_tasks"]
 
 [project]
 name = "django-lambda-tasks"
-version = "0.4.8"
+version = "0.4.9"
 description = "Run async tasks in a lambda function"
 readme = "README.md"
 requires-python = ">=3.10"

{django_lambda_tasks-0.4.8 → django_lambda_tasks-0.4.9}/tests/test_local_executor.py

@@ -679,3 +679,212 @@ class TestWorkerExceptionIsolation:
 
         # Pool.submit was called twice — pool did not crash
         assert mock_pool.submit.call_count == 2
+
+
+# ---------------------------------------------------------------------------
+# Shutdown signal handlers (Ctrl+C / SIGINT race with the autoreloader parent)
+# ---------------------------------------------------------------------------
+
+import signal as signal_module
+
+
+@pytest.fixture()
+def _reset_signal_handlers():
+    """Save/restore SIGINT and SIGTERM handlers and the installed sentinel."""
+    import lambda_tasks.local_executor
+
+    original_sigint = signal_module.getsignal(signal_module.SIGINT)
+    original_sigterm = signal_module.getsignal(signal_module.SIGTERM)
+    original_flag = lambda_tasks.local_executor._handlers_installed
+    lambda_tasks.local_executor._handlers_installed = False
+    try:
+        yield
+    finally:
+        signal_module.signal(signal_module.SIGINT, original_sigint)
+        signal_module.signal(signal_module.SIGTERM, original_sigterm)
+        lambda_tasks.local_executor._handlers_installed = original_flag
+
+
+class TestShutdownSignalHandlers:
+    """Tests for the SIGINT/SIGTERM handlers that release the pool promptly.
+
+    Under Django's autoreloader the server runs in a child spawned by
+    subprocess.run(); on Ctrl+C the parent SIGKILLs the child. The child must
+    release the pool's semaphores before that SIGKILL lands, so cleanup happens
+    at the front of signal handling rather than via atexit.
+    """
+
+    @pytest.mark.usefixtures("_reset_signal_handlers")
+    def test_installs_sigint_handler(self):
+        """_install_shutdown_handlers() replaces the SIGINT handler with our own."""
+        from lambda_tasks.local_executor import _install_shutdown_handlers
+
+        _install_shutdown_handlers()
+
+        handler = signal_module.getsignal(signal_module.SIGINT)
+        assert callable(handler)
+        assert handler != signal_module.default_int_handler
+
+    @pytest.mark.usefixtures("_reset_signal_handlers")
+    def test_handler_shuts_pool_down_then_chains_to_previous(self):
+        """The installed handler calls _shutdown_pool() then the previous handler."""
+        import lambda_tasks.local_executor as local_executor
+
+        call_order = []
+
+        def previous_handler(signum, frame):
+            call_order.append("previous")
+
+        signal_module.signal(signal_module.SIGINT, previous_handler)
+
+        with patch.object(
+            local_executor,
+            "_shutdown_pool",
+            side_effect=lambda: call_order.append("shutdown"),
+        ) as mock_shutdown:
+            local_executor._install_shutdown_handlers()
+            installed = signal_module.getsignal(signal_module.SIGINT)
+            installed(signal_module.SIGINT, None)
+
+        mock_shutdown.assert_called_once()
+        assert call_order == ["shutdown", "previous"]
+
+    @pytest.mark.usefixtures("_reset_signal_handlers")
+    def test_idempotent_does_not_double_wrap(self):
+        """Calling twice keeps a single wrapper (no nested chaining of our own handler)."""
+        from lambda_tasks.local_executor import _install_shutdown_handlers
+
+        _install_shutdown_handlers()
+        first = signal_module.getsignal(signal_module.SIGINT)
+        _install_shutdown_handlers()
+        second = signal_module.getsignal(signal_module.SIGINT)
+
+        assert first is second
+
+    @pytest.mark.usefixtures("_reset_signal_handlers")
+    def test_noop_outside_main_thread(self):
+        """Installation is a no-op when not on the main thread (signal.signal would raise)."""
+        import threading
+
+        import lambda_tasks.local_executor as local_executor
+
+        results = {}
+
+        def worker():
+            local_executor._install_shutdown_handlers()
+            results["installed"] = local_executor._handlers_installed
+
+        thread = threading.Thread(target=worker)
+        thread.start()
+        thread.join()
+
+        assert results["installed"] is False
+
+
+class TestAppConfigReady:
+    """AppConfig.ready() installs the shutdown handlers only in async-local mode."""
+
+    def test_ready_installs_handlers_when_local_workers_positive(self, settings):
+        from lambda_tasks.apps import LambdaTasksConfig
+
+        settings.LAMBDA_TASKS_LOCAL_WORKERS = 2
+        settings.LAMBDA_TASKS_EAGER = False
+
+        config = LambdaTasksConfig.create("lambda_tasks")
+        with patch(
+            "lambda_tasks.local_executor._install_shutdown_handlers"
+        ) as mock_install:
+            config.ready()
+
+        mock_install.assert_called_once()
+
+    def test_ready_does_not_install_handlers_when_local_workers_zero(self, settings):
+        from lambda_tasks.apps import LambdaTasksConfig
+
+        settings.LAMBDA_TASKS_LOCAL_WORKERS = 0
+        settings.LAMBDA_TASKS_EAGER = False
+
+        config = LambdaTasksConfig.create("lambda_tasks")
+        with patch(
+            "lambda_tasks.local_executor._install_shutdown_handlers"
+        ) as mock_install:
+            config.ready()
+
+        mock_install.assert_not_called()
+
+
+# ---------------------------------------------------------------------------
+# _shutdown_pool() terminate-first behaviour
+# ---------------------------------------------------------------------------
+
+
+class TestShutdownPool:
+    """_shutdown_pool() must release semaphores promptly to win the SIGKILL race.
+
+    pool.shutdown(wait=True) blocks on joining worker processes; if a worker is
+    slow to exit, the autoreloader parent SIGKILLs this process before the
+    semaphores are unlinked. So _shutdown_pool() terminates the workers first,
+    then shuts down without waiting.
+    """
+
+    def test_shutdown_terminates_workers_before_shutdown(self):
+        """_shutdown_pool() calls terminate() on each worker, then shutdown(wait=False)."""
+        import lambda_tasks.local_executor as local_executor
+
+        call_order = []
+
+        process_a = MagicMock()
+        process_a.terminate.side_effect = lambda: call_order.append("terminate_a")
+        process_b = MagicMock()
+        process_b.terminate.side_effect = lambda: call_order.append("terminate_b")
+
+        mock_pool = MagicMock()
+        mock_pool._processes = {"a": process_a, "b": process_b}
+        mock_pool.shutdown.side_effect = lambda **kwargs: call_order.append(
+            f"shutdown:{kwargs}"
+        )
+
+        original_pool = local_executor._pool
+        local_executor._pool = mock_pool
+        try:
+            local_executor._shutdown_pool()
+        finally:
+            local_executor._pool = original_pool
+
+        process_a.terminate.assert_called_once()
+        process_b.terminate.assert_called_once()
+        # Both terminates happen before the shutdown call.
+        assert call_order.index("terminate_a") < call_order.index(
+            "shutdown:{'wait': False, 'cancel_futures': True}"
+        )
+        assert call_order.index("terminate_b") < call_order.index(
+            "shutdown:{'wait': False, 'cancel_futures': True}"
+        )
+        mock_pool.shutdown.assert_called_once_with(wait=False, cancel_futures=True)
+
+    def test_shutdown_clears_module_pool(self):
+        """_shutdown_pool() resets the module-level _pool to None."""
+        import lambda_tasks.local_executor as local_executor
+
+        mock_pool = MagicMock()
+        mock_pool._processes = {}
+
+        original_pool = local_executor._pool
+        local_executor._pool = mock_pool
+        try:
+            local_executor._shutdown_pool()
+            assert local_executor._pool is None
+        finally:
+            local_executor._pool = original_pool
+
+    def test_shutdown_is_noop_when_no_pool(self):
+        """_shutdown_pool() does nothing when no pool exists."""
+        import lambda_tasks.local_executor as local_executor
+
+        original_pool = local_executor._pool
+        local_executor._pool = None
+        try:
+            local_executor._shutdown_pool()  # must not raise
+            assert local_executor._pool is None
+        finally:
+            local_executor._pool = original_pool

django_lambda_tasks-0.4.8/lambda_tasks/apps.py

@@ -1,8 +0,0 @@
-"""Django AppConfig for the lambda_tasks library."""
-
-from django.apps import AppConfig
-
-
-class LambdaTasksConfig(AppConfig):
-    name = "lambda_tasks"
-    verbose_name = "Lambda Tasks"

django_lambda_tasks-0.4.8/lambda_tasks/local_executor.py

@@ -1,78 +0,0 @@
-"""Process pool executor for async local task execution."""
-
-import atexit
-import logging
-import signal
-import uuid
-from concurrent.futures import Future, ProcessPoolExecutor
-
-from lambda_tasks.settings import LambdaTasksSettings
-
-logger = logging.getLogger(__name__)
-
-_pool: ProcessPoolExecutor | None = None
-
-
-def _pool_initializer() -> None:
-    """Run once per worker process.
-
-    Ignores SIGINT so that Ctrl+C is handled exclusively by the parent
-    process, which shuts down the pool cleanly via atexit. This prevents
-    workers from being killed mid-operation and leaking semaphores.
-
-    Then sets up Django for task execution.
-    """
-    signal.signal(signal.SIGINT, signal.SIG_IGN)
-
-    import django
-
-    django.setup()
-
-
-def _shutdown_pool() -> None:
-    """Shut down the pool at interpreter exit to release semaphores."""
-    global _pool
-    if _pool is not None:
-        _pool.shutdown(wait=True, cancel_futures=True)
-        _pool = None
-
-
-def get_pool() -> ProcessPoolExecutor:
-    """Return the shared ProcessPoolExecutor, creating it on first call."""
-    global _pool
-    if _pool is None:
-        conf = LambdaTasksSettings()
-        _pool = ProcessPoolExecutor(
-            max_workers=conf.LOCAL_WORKERS,
-            initializer=_pool_initializer,
-        )
-        atexit.register(_shutdown_pool)
-    return _pool
-
-
-def _execute_in_worker(*, message_json: str, message_id: str) -> None:
-    """Worker entry point. Deserializes and executes the task.
-
-    Runs in a child process. Django is already set up via the pool initializer.
-    """
-    from lambda_tasks.models import SQSLambdaTaskMessage
-
-    message = SQSLambdaTaskMessage.model_validate_json(message_json)
-    message.execute_immediately(message_id=message_id)
-
-
-def _log_worker_exception(future: Future) -> None:  # type: ignore[type-arg]
-    """Callback attached to each worker future. Logs unhandled exceptions."""
-    exception = future.exception()
-    if exception is not None:
-        logger.error("Worker process raised an exception", exc_info=exception)
-
-
-def submit_task(*, message_json: str) -> None:
-    """Submit a task to the process pool. Fire-and-forget."""
-    pool = get_pool()
-    message_id = str(uuid.uuid4())
-    future = pool.submit(
-        _execute_in_worker, message_json=message_json, message_id=message_id
-    )
-    future.add_done_callback(_log_worker_exception)