django-lambda-tasks 0.4.2__tar.gz → 0.4.4__tar.gz

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/.kiro/steering/product.md

@@ -39,7 +39,14 @@ def my_task(*, user_id: int, action: str) -> None:
 
 ## Enqueuing
 
-`execute_on_commit()` uses the decorator `delay` value. There are no per-call overrides.
+`execute_on_commit()` uses the decorator `delay` value by default. Pass `_delay=<seconds>` at call time to override the delay for that specific enqueue:
+
+```python
+my_task.execute_on_commit(user_id=1, action="x")            # uses decorator delay
+my_task.execute_on_commit(user_id=1, action="x", _delay=60) # overrides with 60s
+```
+
+The `_delay` override is validated against the same range `[0, 900]` as the decorator `delay`. It only affects the SQS `DelaySeconds` — it has no effect in eager or async-local mode.
 
 ## ignore_errors
 
@@ -136,10 +143,11 @@ class SQSLambdaTaskMessage(BaseModel):
 - Returns `{"batchItemFailures": [...]}` for partial-batch failure reporting
 - Only pre-execution failures (malformed message, import error, misconfiguration) are reported as `batchItemFailures` — task logic failures are caught and recorded as `FAILED` TaskRecords without raising
 - Recommended SQS queue settings: `maxReceiveCount=1` with a DLQ configured; automatic retries are not useful since task failures are not re-driven by design
-- Cold-start sequence runs inside the handler on the first invocation (not at module import time) to avoid Lambda init-duration timeouts: a temporary `StreamHandler` is attached to the `lambda_tasks` logger, then `resolve_environment()` → `resolve_secrets_into_env()` (handler removed) → conditional `django.setup()`
+- Cold-start sequence runs inside the handler on the first invocation (not at module import time) to avoid Lambda init-duration timeouts: memory limit is set, a temporary `StreamHandler` is attached to the `lambda_tasks` logger, then `resolve_environment()` → `resolve_secrets_into_env()` (handler removed) → conditional `django.setup()`
 - A module-level `_cold_start_done` sentinel ensures the sequence runs only once; subsequent warm invocations skip it
 - Both loaders run unconditionally (outside the `DJANGO_SETTINGS_MODULE` check) — the environment secret may provide that var, and individual secrets may depend on environment-loaded vars
 - A temporary `StreamHandler` is attached to the `lambda_tasks` logger for the duration of the loaders so their log output is visible before Django's `LOGGING` dictConfig has run; it is removed immediately after so that Django's configuration is the sole authority on logging from that point on
+- `resource.setrlimit(RLIMIT_DATA)` is set from `AWS_LAMBDA_FUNCTION_MEMORY_SIZE` so that excessive allocation raises `MemoryError` instead of triggering the OOM killer
 
 ## Environment Loader
 

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/.kiro/steering/structure.md

@@ -42,7 +42,7 @@ django-lambda-tasks/
 - `decorators.py` — defines `@lambda_task`; enforces kwargs-only at decoration time
 - `models.py` — `TaskRecord` (Django ORM), `SQSLambdaTaskMessage` (Pydantic, SQS schema + execution logic), `SQSLambdaTask` (Pydantic, holds message + routing; `_execute()` publishes to SQS, executes eagerly, or submits to the local process pool; `execute_on_commit()` registers `_execute` with `transaction.on_commit`)
 - `local_executor.py` — `ProcessPoolExecutor`-based async local execution; `get_pool()` lazily creates a module-level pool; `submit_task()` fire-and-forget submission; `_execute_in_worker()` deserializes and runs the task in a child process; `_pool_initializer()` calls `django.setup()` per worker
-- `handler.py` — Lambda entry point; cold-start init (temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
+- `handler.py` — Lambda entry point; cold-start init (memory limit → temporary log handler → `resolve_environment()` → `resolve_secrets_into_env()` → handler removed → conditional `django.setup()`) runs inside the handler on first invocation, guarded by `_cold_start_done` sentinel; processes SQS records independently; returns `batchItemFailures`
 - `environment_loader.py` — loads env vars from a Secrets Manager secret at cold start; validates flat JSON format; idempotent via `_loaded` sentinel
 - `secret_loader.py` — resolves `LAMBDA_TASKS_SECRET_*` env vars from Secrets Manager before Django starts; validates format, detects conflicts, batches API calls; idempotent via `_loaded` sentinel
 - `logging.py` — `task_logger` singleton; `message_id` set/cleared around each task execution

django_lambda_tasks-0.4.2/README.md → django_lambda_tasks-0.4.4/PKG-INFO

@@ -1,3 +1,15 @@
+Metadata-Version: 2.4
+Name: django-lambda-tasks
+Version: 0.4.4
+Summary: Run async tasks in a lambda function
+Requires-Python: >=3.10
+Requires-Dist: awslambdaric
+Requires-Dist: boto3
+Requires-Dist: django
+Requires-Dist: pydantic
+Requires-Dist: redis
+Description-Content-Type: text/markdown
+
 # Django Lambda Tasks
 
 A Django library for offloading work to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler that AWS invokes with SQS message batches. Task results, status, and metadata are persisted in the Django database.
@@ -59,6 +71,13 @@ def register(request):
     return HttpResponse("Registered")
 ```
 
+You can override the delay for a specific enqueue by passing `_delay`:
+
+```python
+# Delay this particular invocation by 60 seconds instead of the decorator default
+send_welcome_email.execute_on_commit(user_id=user.id, template="welcome", _delay=60)
+```
+
 ### 3. Configure AWS
 
 See [AWS Lambda and SQS setup](#aws-lambda-and-sqs-setup) for queue and Lambda configuration.
@@ -208,6 +227,24 @@ def my_task(*, arg: str) -> None:
 | `retry_delay` | `int` | `0` | Base delay in seconds when enqueuing a retry. Jitter (1–5s) is always added; result capped at 900. Requires `retry_on` to be non-empty. |
 | `singleton` | `bool` | `False` | Prevent concurrent execution via a Redis lock (see [Singleton tasks](#singleton-tasks)). |
 
+### Per-call delay override
+
+The `delay` decorator parameter sets the default SQS `DelaySeconds` for all invocations of a task. To override it for a specific call, pass `_delay` to `execute_on_commit()` or `serialize()`:
+
+```python
+@lambda_task(delay=0)
+def notify_user(*, user_id: int) -> None:
+    ...
+
+# Uses the decorator default (0 seconds)
+notify_user.execute_on_commit(user_id=1)
+
+# Override: delay this specific invocation by 120 seconds
+notify_user.execute_on_commit(user_id=1, _delay=120)
+```
+
+`_delay` is validated against the same `[0, 900]` range as the decorator `delay`. It only affects the SQS `DelaySeconds` — it has no effect in eager or async-local mode.
+
 ---
 
 ## Serializing a task invocation

django_lambda_tasks-0.4.2/PKG-INFO → django_lambda_tasks-0.4.4/README.md

@@ -1,15 +1,3 @@
-Metadata-Version: 2.4
-Name: django-lambda-tasks
-Version: 0.4.2
-Summary: Run async tasks in a lambda function
-Requires-Python: >=3.10
-Requires-Dist: awslambdaric
-Requires-Dist: boto3
-Requires-Dist: django
-Requires-Dist: pydantic
-Requires-Dist: redis
-Description-Content-Type: text/markdown
-
 # Django Lambda Tasks
 
 A Django library for offloading work to AWS Lambda outside of the request-response cycle. Tasks are defined with a decorator, enqueued to SQS on transaction commit, and executed by a Lambda handler that AWS invokes with SQS message batches. Task results, status, and metadata are persisted in the Django database.
@@ -71,6 +59,13 @@ def register(request):
     return HttpResponse("Registered")
 ```
 
+You can override the delay for a specific enqueue by passing `_delay`:
+
+```python
+# Delay this particular invocation by 60 seconds instead of the decorator default
+send_welcome_email.execute_on_commit(user_id=user.id, template="welcome", _delay=60)
+```
+
 ### 3. Configure AWS
 
 See [AWS Lambda and SQS setup](#aws-lambda-and-sqs-setup) for queue and Lambda configuration.
@@ -220,6 +215,24 @@ def my_task(*, arg: str) -> None:
 | `retry_delay` | `int` | `0` | Base delay in seconds when enqueuing a retry. Jitter (1–5s) is always added; result capped at 900. Requires `retry_on` to be non-empty. |
 | `singleton` | `bool` | `False` | Prevent concurrent execution via a Redis lock (see [Singleton tasks](#singleton-tasks)). |
 
+### Per-call delay override
+
+The `delay` decorator parameter sets the default SQS `DelaySeconds` for all invocations of a task. To override it for a specific call, pass `_delay` to `execute_on_commit()` or `serialize()`:
+
+```python
+@lambda_task(delay=0)
+def notify_user(*, user_id: int) -> None:
+    ...
+
+# Uses the decorator default (0 seconds)
+notify_user.execute_on_commit(user_id=1)
+
+# Override: delay this specific invocation by 120 seconds
+notify_user.execute_on_commit(user_id=1, _delay=120)
+```
+
+`_delay` is validated against the same `[0, 900]` range as the decorator `delay`. It only affects the SQS `DelaySeconds` — it has no effect in eager or async-local mode.
+
 ---
 
 ## Serializing a task invocation

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/lambda_tasks/decorators.py

@@ -154,13 +154,17 @@ class LambdaTaskWrapper:
     def _build_task(self, *, kwargs: dict[str, Any]) -> SQSLambdaTask:
         """Pop overrides, validate kwargs, and build a SQSLambdaTask.
 
-        Mutates *kwargs* in-place (pops ``_n_retries``).
+        Mutates *kwargs* in-place (pops ``_n_retries``, ``_delay``).
         Returns ``SQSLambdaTask``.
 
         Raises:
             pydantic.ValidationError: if the remaining kwargs fail type validation.
+            ValueError: if ``_delay`` is outside the allowed range [0, 900].
         """
         n_retries = kwargs.pop("_n_retries", 0)
+        delay = kwargs.pop("_delay", self._delay)
+
+        self._validate_delay(delay=delay)
 
         self._kwargs_model.model_validate(kwargs)
 
@@ -172,7 +176,7 @@ class LambdaTaskWrapper:
 
         return SQSLambdaTask(
             message=message,
-            delay=self._delay,
+            delay=delay,
             queue=self._queue,
         )
 

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/lambda_tasks/handler.py

@@ -14,6 +14,7 @@ warm invocations skip it.
 
 import logging
 import os
+import resource
 
 import django
 from django.apps import apps
@@ -26,6 +27,36 @@ logger = logging.getLogger(__name__)
 _cold_start_done: bool = False
 
 
+_MEMORY_RESERVED_MB = 128
+_MEMORY_MINIMUM_LIMIT_MB = 64
+
+
+def _set_memory_limit() -> None:
+    """Set RLIMIT_DATA from AWS_LAMBDA_FUNCTION_MEMORY_SIZE if available.
+
+    When running in Lambda, this causes Python to raise MemoryError on
+    excessive allocation instead of being killed by the OOM killer.
+
+    128 MB is reserved for the Python runtime, shared libraries, and OS
+    overhead. The limit is floored at 64 MB so that even small Lambdas
+    get some protection.
+    """
+    memory_mb = os.environ.get("AWS_LAMBDA_FUNCTION_MEMORY_SIZE")
+
+    if memory_mb is None:
+        return
+
+    limit_bytes = (
+        max(int(memory_mb) - _MEMORY_RESERVED_MB, _MEMORY_MINIMUM_LIMIT_MB)
+        * 1024
+        * 1024
+    )
+    resource.setrlimit(resource.RLIMIT_AS, (limit_bytes, limit_bytes))
+    logger.info(
+        f"Set RLIMIT_AS to {limit_bytes} bytes ({memory_mb} MB - {_MEMORY_RESERVED_MB} MB reserved)"
+    )
+
+
 def _perform_cold_start() -> None:
     """Run one-time initialisation: env loading, secrets, Django setup.
 
@@ -49,6 +80,8 @@ def _perform_cold_start() -> None:
     lambda_tasks_logger.addHandler(boot_handler)
     lambda_tasks_logger.setLevel(logging.DEBUG)
 
+    _set_memory_limit()
+
     try:
         resolve_environment()
         resolve_secrets_into_env()

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/pyproject.toml

@@ -7,7 +7,7 @@ packages = ["lambda_tasks"]
 
 [project]
 name = "django-lambda-tasks"
-version = "0.4.2"
+version = "0.4.4"
 description = "Run async tasks in a lambda function"
 readme = "README.md"
 requires-python = ">=3.10"

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/tests/test_decorator.py

@@ -82,13 +82,23 @@ class TestLambdaTaskWrapperOnCommit:
 
     def test_on_commit_accepts_delay_override(self):
         wrapper = LambdaTaskWrapper(sample_task)
-        with pytest.raises(pydantic.ValidationError):
+        captured: list = []
+        with patch(
+            "lambda_tasks.models.transaction.on_commit",
+            side_effect=lambda cb: captured.append(cb),
+        ):
             wrapper.execute_on_commit(x=1, _delay=10)
+        assert captured[0].__self__.delay == 10
 
     def test_on_commit_accepts_delay_and_task_kwargs(self):
         wrapper = LambdaTaskWrapper(sample_task)
-        with pytest.raises(pydantic.ValidationError):
+        captured: list = []
+        with patch(
+            "lambda_tasks.models.transaction.on_commit",
+            side_effect=lambda cb: captured.append(cb),
+        ):
             wrapper.execute_on_commit(x=1, y="test", _delay=5)
+        assert captured[0].__self__.delay == 5
 
 
 # ---------------------------------------------------------------------------

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/tests/test_decorators.py

@@ -251,21 +251,30 @@ def test_queue_property_defaults_to_default():
 # ---------------------------------------------------------------------------
 
 
-def test_passing_delay_kwarg_to_execute_on_commit_raises_validation_error():
-    """Passing _delay to execute_on_commit raises pydantic.ValidationError. Requirement 4.1, 4.2"""
+def test_passing_delay_kwarg_to_execute_on_commit_overrides_decorator_delay():
+    """Passing _delay to _build_task overrides the decorator delay. Requirement 4.1, 4.2"""
     wrapper = LambdaTaskWrapper(_make_func(), delay=10)
-    with pytest.raises(pydantic.ValidationError):
-        wrapper._build_task(kwargs={"x": 1, "_delay": 5})
+    task = wrapper._build_task(kwargs={"x": 1, "_delay": 5})
+    assert task.delay == 5
 
 
 def test_build_task_uses_decorator_delay_not_call_time_override():
-    """_build_task uses the decorator delay, not a call-time override. Requirement 4.3"""
+    """_build_task uses the decorator delay when no _delay override is given. Requirement 4.3"""
     decorator_delay = 42
     wrapper = LambdaTaskWrapper(_make_func(), delay=decorator_delay)
     task = wrapper._build_task(kwargs={"x": 1})
     assert task.delay == decorator_delay
 
 
+def test_build_task_delay_override_validates_range():
+    """_delay outside [0, 900] raises ValueError."""
+    wrapper = LambdaTaskWrapper(_make_func(), delay=10)
+    with pytest.raises(ValueError):
+        wrapper._build_task(kwargs={"x": 1, "_delay": -1})
+    with pytest.raises(ValueError):
+        wrapper._build_task(kwargs={"x": 1, "_delay": 901})
+
+
 # ---------------------------------------------------------------------------
 # Unit tests: lambda_task forwarding retry_delay (Requirements 1.1, 1.2)
 # ---------------------------------------------------------------------------

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/tests/test_deferred_enqueue.py

@@ -44,9 +44,9 @@ def test_to_json_uses_decorator_defaults_for_delay_and_queue():
 
 
 def test_to_json_uses_call_site_override_for_delay():
-    """_delay at call site is no longer supported — it raises pydantic.ValidationError."""
-    with pytest.raises(ValidationError):
-        _wrapper.serialize(x=1, _delay=30)
+    """_delay at call site overrides the decorator default."""
+    result = _wrapper.serialize(x=1, _delay=30)
+    assert result["delay"] == 30
 
 
 def test_to_json_task_name_matches_module_qualname():

django_lambda_tasks-0.4.4/tests/test_memory_limit.py

@@ -0,0 +1,131 @@
+"""
+Unit tests for memory limit enforcement in the Lambda handler.
+
+The handler sets resource.RLIMIT_AS during cold start so that runaway
+memory allocation raises MemoryError instead of triggering the Lambda OOM
+killer.
+"""
+
+import resource
+from unittest.mock import patch
+
+import lambda_tasks.handler as handler_module
+
+
+class TestMemoryLimitSetDuringColdStart:
+    def test_rlimit_as_set_from_env_var(self, monkeypatch):
+        """When AWS_LAMBDA_FUNCTION_MEMORY_SIZE is set, RLIMIT_AS is configured with 128 MB reserved."""
+        monkeypatch.delenv("DJANGO_SETTINGS_MODULE", raising=False)
+        monkeypatch.setenv("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", "512")
+        monkeypatch.setattr("lambda_tasks.handler.resolve_environment", lambda: None)
+        monkeypatch.setattr(
+            "lambda_tasks.handler.resolve_secrets_into_env", lambda: None
+        )
+
+        calls: list[tuple] = []
+
+        def fake_setrlimit(which: int, limits: tuple[int, int]) -> None:
+            calls.append((which, limits))
+
+        monkeypatch.setattr(resource, "setrlimit", fake_setrlimit)
+
+        handler_module._cold_start_done = False
+        handler_module.handler(event={"Records": []}, context=None)
+
+        expected = (512 - 128) * 1024 * 1024
+        assert calls == [(resource.RLIMIT_AS, (expected, expected))]
+
+    def test_rlimit_as_not_set_when_env_var_missing(self, monkeypatch):
+        """When AWS_LAMBDA_FUNCTION_MEMORY_SIZE is not set, RLIMIT_AS is unchanged."""
+        monkeypatch.delenv("DJANGO_SETTINGS_MODULE", raising=False)
+        monkeypatch.delenv("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", raising=False)
+        monkeypatch.setattr("lambda_tasks.handler.resolve_environment", lambda: None)
+        monkeypatch.setattr(
+            "lambda_tasks.handler.resolve_secrets_into_env", lambda: None
+        )
+
+        calls: list[tuple] = []
+
+        def fake_setrlimit(which: int, limits: tuple[int, int]) -> None:
+            calls.append((which, limits))
+
+        monkeypatch.setattr(resource, "setrlimit", fake_setrlimit)
+
+        handler_module._cold_start_done = False
+        handler_module.handler(event={"Records": []}, context=None)
+
+        assert calls == []
+
+    def test_memory_limit_set_before_loaders(self, monkeypatch):
+        """The memory limit is set before resolve_environment runs."""
+        monkeypatch.delenv("DJANGO_SETTINGS_MODULE", raising=False)
+        monkeypatch.setenv("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", "256")
+
+        call_order: list[str] = []
+
+        def fake_setrlimit(which: int, limits: tuple[int, int]) -> None:
+            call_order.append("setrlimit")
+
+        def fake_resolve_environment() -> None:
+            call_order.append("resolve_environment")
+
+        def fake_resolve_secrets() -> None:
+            call_order.append("resolve_secrets_into_env")
+
+        monkeypatch.setattr(resource, "setrlimit", fake_setrlimit)
+        monkeypatch.setattr(
+            "lambda_tasks.handler.resolve_environment", fake_resolve_environment
+        )
+        monkeypatch.setattr(
+            "lambda_tasks.handler.resolve_secrets_into_env", fake_resolve_secrets
+        )
+
+        handler_module._cold_start_done = False
+        handler_module.handler(event={"Records": []}, context=None)
+
+        assert call_order == [
+            "setrlimit",
+            "resolve_environment",
+            "resolve_secrets_into_env",
+        ]
+
+    def test_memory_limit_is_logged(self, monkeypatch, caplog):
+        """Setting the memory limit emits an info log."""
+        import logging
+
+        monkeypatch.delenv("DJANGO_SETTINGS_MODULE", raising=False)
+        monkeypatch.setenv("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", "1024")
+        monkeypatch.setattr("lambda_tasks.handler.resolve_environment", lambda: None)
+        monkeypatch.setattr(
+            "lambda_tasks.handler.resolve_secrets_into_env", lambda: None
+        )
+        monkeypatch.setattr(resource, "setrlimit", lambda *args: None)
+
+        handler_module._cold_start_done = False
+
+        with caplog.at_level(logging.INFO, logger="lambda_tasks.handler"):
+            handler_module.handler(event={"Records": []}, context=None)
+
+        assert any("RLIMIT_AS" in msg for msg in caplog.messages)
+
+    def test_rlimit_floored_at_minimum_when_memory_too_small(self, monkeypatch):
+        """When AWS_LAMBDA_FUNCTION_MEMORY_SIZE <= reserved, RLIMIT_AS is set to the 64 MB minimum."""
+        monkeypatch.delenv("DJANGO_SETTINGS_MODULE", raising=False)
+        monkeypatch.setenv("AWS_LAMBDA_FUNCTION_MEMORY_SIZE", "128")
+        monkeypatch.setattr("lambda_tasks.handler.resolve_environment", lambda: None)
+        monkeypatch.setattr(
+            "lambda_tasks.handler.resolve_secrets_into_env", lambda: None
+        )
+
+        calls: list[tuple] = []
+
+        def fake_setrlimit(which: int, limits: tuple[int, int]) -> None:
+            calls.append((which, limits))
+
+        monkeypatch.setattr(resource, "setrlimit", fake_setrlimit)
+
+        handler_module._cold_start_done = False
+        handler_module.handler(event={"Records": []}, context=None)
+
+        expected = 64 * 1024 * 1024
+        assert calls == [(resource.RLIMIT_AS, (expected, expected))]

{django_lambda_tasks-0.4.2 → django_lambda_tasks-0.4.4}/tests/test_models.py

@@ -1649,17 +1649,22 @@ def test_normal_execute_on_commit_uses_decorator_delay(settings, decorator_delay
     assert mock_client.send_message.call_args.kwargs["DelaySeconds"] == decorator_delay
 
 
-def test_passing_delay_to_execute_on_commit_raises_validation_error():
-    """Passing _delay to execute_on_commit raises ValidationError (extra='forbid' on kwargs model).
+def test_passing_delay_to_execute_on_commit_overrides_decorator_delay():
+    """Passing _delay to execute_on_commit overrides the decorator delay.
     Validates: Requirements 4.1, 4.2"""
-    from pydantic import ValidationError
+    from unittest.mock import patch
 
-    @lambda_task
+    @lambda_task(delay=10)
     def _task_simple(*, x: int) -> None:
         pass
 
-    with pytest.raises(ValidationError):
+    captured: list = []
+    with patch(
+        "lambda_tasks.models.transaction.on_commit",
+        side_effect=lambda cb: captured.append(cb),
+    ):
         _task_simple.execute_on_commit(x=1, _delay=5)
+    assert captured[0].__self__.delay == 5
 
 
 # ---------------------------------------------------------------------------