async-lambda-unstable 0.6.13__tar.gz → 0.6.15__tar.gz

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: async-lambda-unstable
-Version: 0.6.13
+Version: 0.6.15
 Summary: A framework for creating AWS Lambda Async Workflows. - Unstable Branch
 Author-email: "Nuclei, Inc" <engineering@nuclei.ai>
 Requires-Dist: click>=8.0.0
@@ -249,6 +249,32 @@ and merging is done thru concatenation.
 
 [The value is passed to the `Layers` property on `SAM::Serverless::Function`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-layers)
 
+### image_uri
+
+```
+"ACCOUNT.dkr.ecr.REGION.amazonaws.com/REPOSITORY:TAG"
+```
+
+Package the functions as a Lambda container image instead of a zip bundle, which raises the
+package size limit from 250 MB (unzipped) to 10 GB.
+
+When set, every function gets `PackageType: Image`, `ImageUri`, and `ImageConfig.Command` set to
+`<module>.lambda_handler` in place of `CodeUri`, `Handler`, and `Runtime`, and `async-lambda build`
+writes only the template (no `deployment.zip`). The image is built and pushed separately; it must
+be based on an AWS Lambda Python base image (or implement the Lambda runtime interface) and contain
+the app module, `src/`, and its dependencies.
+
+`async-lambda build app --image-uri <uri>` sets this for a single build, overriding the config (including
+stage-level values), so a CI pipeline can pin the image it just pushed.
+
+Image functions are named `<name>-<task_id>-image`. Lambda cannot change an existing function's package
+type, so adding or removing `image_uri` replaces each function, and CloudFormation cannot replace a
+function under the same custom name. The suffix lets the new function be created before the old one is
+deleted, so switching needs no manual deletion or downtime. Queue and schedule names are unchanged.
+
+Lambda layers and SnapStart are not supported on container images, so `layers` and `snap_start`
+raise an error at build time when `image_uri` is set.
+
 ### subnet_ids
 
 ```
@@ -437,6 +463,27 @@ Two caveats:
 - A task with task-specific `policies` keeps its own generated role, since the shared role is built from app-wide and stage-wide config only.
 - Every app-wide policy must be either a managed policy ARN or a `{"Statement": [...]}` document. SAM policy templates can't be expressed in a plain IAM role and will fail the build.
 
+### Consolidated Queue-Age Alarms
+
+`managed_queue_extras` attaches a copy of each extra resource to *every* managed queue, so using it for a CloudWatch alarm costs one stack resource per queue lane.
+
+Setting `queue_age_alarm` in the build config emits a small number of grouped alarms instead, each watching `ApproximateAgeOfOldestMessage` across up to ten queues:
+
+```json
+{
+  "queue_age_alarm": {
+    "threshold": 5000,
+    "period": 86400,
+    "alarm_actions": ["arn:aws:sns:us-east-1:123456789012:alarms"],
+    "ok_actions": ["arn:aws:sns:us-east-1:123456789012:alarms"]
+  }
+}
+```
+
+Only `threshold` is required and `period` defaults to one day. Coverage is every managed queue lane plus the app DLQ. An app with 104 queues gets 11 alarms rather than 104.
+
+CloudWatch rejects an alarm carrying more than 10 metrics.
+
 ## Known Limitations
 
 - Not all Lambda configuration options are supported (see code for extension points)

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/README.md

@@ -237,6 +237,32 @@ and merging is done thru concatenation.
 
 [The value is passed to the `Layers` property on `SAM::Serverless::Function`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-function.html#sam-function-layers)
 
+### image_uri
+
+```
+"ACCOUNT.dkr.ecr.REGION.amazonaws.com/REPOSITORY:TAG"
+```
+
+Package the functions as a Lambda container image instead of a zip bundle, which raises the
+package size limit from 250 MB (unzipped) to 10 GB.
+
+When set, every function gets `PackageType: Image`, `ImageUri`, and `ImageConfig.Command` set to
+`<module>.lambda_handler` in place of `CodeUri`, `Handler`, and `Runtime`, and `async-lambda build`
+writes only the template (no `deployment.zip`). The image is built and pushed separately; it must
+be based on an AWS Lambda Python base image (or implement the Lambda runtime interface) and contain
+the app module, `src/`, and its dependencies.
+
+`async-lambda build app --image-uri <uri>` sets this for a single build, overriding the config (including
+stage-level values), so a CI pipeline can pin the image it just pushed.
+
+Image functions are named `<name>-<task_id>-image`. Lambda cannot change an existing function's package
+type, so adding or removing `image_uri` replaces each function, and CloudFormation cannot replace a
+function under the same custom name. The suffix lets the new function be created before the old one is
+deleted, so switching needs no manual deletion or downtime. Queue and schedule names are unchanged.
+
+Lambda layers and SnapStart are not supported on container images, so `layers` and `snap_start`
+raise an error at build time when `image_uri` is set.
+
 ### subnet_ids
 
 ```
@@ -425,6 +451,27 @@ Two caveats:
 - A task with task-specific `policies` keeps its own generated role, since the shared role is built from app-wide and stage-wide config only.
 - Every app-wide policy must be either a managed policy ARN or a `{"Statement": [...]}` document. SAM policy templates can't be expressed in a plain IAM role and will fail the build.
 
+### Consolidated Queue-Age Alarms
+
+`managed_queue_extras` attaches a copy of each extra resource to *every* managed queue, so using it for a CloudWatch alarm costs one stack resource per queue lane.
+
+Setting `queue_age_alarm` in the build config emits a small number of grouped alarms instead, each watching `ApproximateAgeOfOldestMessage` across up to ten queues:
+
+```json
+{
+  "queue_age_alarm": {
+    "threshold": 5000,
+    "period": 86400,
+    "alarm_actions": ["arn:aws:sns:us-east-1:123456789012:alarms"],
+    "ok_actions": ["arn:aws:sns:us-east-1:123456789012:alarms"]
+  }
+}
+```
+
+Only `threshold` is required and `period` defaults to one day. Coverage is every managed queue lane plus the app DLQ. An app with 104 queues gets 11 alarms rather than 104.
+
+CloudWatch rejects an alarm carrying more than 10 metrics.
+
 ## Known Limitations
 
 - Not all Lambda configuration options are supported (see code for extension points)

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/async_lambda/__init__.py

@@ -22,4 +22,4 @@ from .models.events.managed_sqs_event import ManagedSQSEvent as ManagedSQSEvent
 from .models.events.scheduled_event import ScheduledEvent as ScheduledEvent
 from .models.events.unmanaged_sqs_event import UnmanagedSQSEvent as UnmanagedSQSEvent
 
-__version__ = "0.6.13"
+__version__ = "0.6.15"

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/async_lambda/build_config.py

@@ -14,6 +14,46 @@ def make_default_tags() -> Dict[str, str]:
     return {"framework": "async-lambda", "framework-version": __version__}
 
 
+@dataclass
+class QueueAgeAlarmConfig:
+    """
+    Settings for the consolidated queue-age alarms.
+
+    One alarm covers a group of queues rather than one alarm per queue.
+
+    Attributes:
+        threshold (int): Oldest-message age, in seconds, above which the alarm fires.
+        period (int): Seconds of data each datapoint covers.
+        alarm_actions (List[str]): ARNs notified when an alarm starts firing.
+        ok_actions (List[str]): ARNs notified when an alarm recovers.
+    """
+
+    threshold: int
+    period: int
+    alarm_actions: List[str]
+    ok_actions: List[str]
+
+    @classmethod
+    def new(cls, config: dict) -> "QueueAgeAlarmConfig":
+        """
+        Creates a new instance from a configuration dictionary.
+
+        Args:
+            config (dict): provided the values of QueueAgeAlarmConfig
+
+        Returns:
+            QueueAgeAlarmConfig: A new instance configured with the provided options.
+        """
+        if "threshold" not in config:
+            raise ValueError("queue_age_alarm requires a `threshold`.")
+        return cls(
+            threshold=config["threshold"],
+            period=config.get("period", 86400),
+            alarm_actions=list(config.get("alarm_actions", list())),
+            ok_actions=list(config.get("ok_actions", list())),
+        )
+
+
 @dataclass
 class AsyncLambdaBuildConfig:
     """
@@ -26,6 +66,7 @@ class AsyncLambdaBuildConfig:
         subnet_ids (Set[str]): Set of subnet IDs for VPC configuration.
         security_group_ids (Set[str]): Set of security group IDs for VPC configuration.
         managed_queue_extras (List[dict]): Additional configuration for managed queues.
+        queue_age_alarm (Optional[QueueAgeAlarmConfig]): Settings for the consolidated queue-age alarms. No alarms are emitted when this is unset.
         method_settings (List[dict]): API Gateway method settings.
         tags (Dict[str, str]): Tags to assign to the Lambda function.
         logging_config (Dict[str, str]): Logging configuration options.
@@ -34,6 +75,7 @@ class AsyncLambdaBuildConfig:
         certificate_arn (Optional[str]): ARN of the SSL certificate for the domain.
         hosted_zone_id (Optional[str]): Hosted zone name for the Lambda/API.
         auto_create_acm_certificate (bool): Whether or not to auto create a certificate in ACM for the domain name. Must be used in conjunction with hosted_zone_id and domain_name to be used.
+        image_uri (Optional[str]): URI of a Lambda container image. When set, functions are packaged as container images instead of a zip bundle.
 
     Methods:
         new(config: dict) -> AsyncLambdaBuildConfig:
@@ -57,6 +99,8 @@ class AsyncLambdaBuildConfig:
     certificate_arn: Optional[str] = None
     hosted_zone_id: Optional[str] = None
     auto_create_acm_certificate: Optional[bool] = None
+    queue_age_alarm: Optional[QueueAgeAlarmConfig] = None
+    image_uri: Optional[str] = None
 
     @classmethod
     def new(cls, config: dict) -> "AsyncLambdaBuildConfig":
@@ -71,6 +115,7 @@ class AsyncLambdaBuildConfig:
                 - subnet_ids (set or list): Set or list of subnet IDs for VPC configuration.
                 - security_group_ids (set or list): Set or list of security group IDs for VPC configuration.
                 - managed_queue_extras (list): Additional managed queue configuration.
+                - queue_age_alarm (dict): Settings for the consolidated queue-age alarms.
                 - method_settings (list): List of method settings for API Gateway.
                 - tags (dict): Tags to assign to the Lambda function.
                 - logging_config (dict): Logging configuration options.
@@ -78,6 +123,7 @@ class AsyncLambdaBuildConfig:
                 - tls_version (str, optional): TLS version to use.
                 - certificate_arn (str, optional): ARN of the SSL certificate.
                 - hosted_zone_id (str, optional): Hosted zone ID for the Lambda/API.
+                - image_uri (str, optional): URI of the Lambda container image.
 
         Returns:
             AsyncLambdaBuildConfig: A new instance configured with the provided options.
@@ -89,6 +135,11 @@ class AsyncLambdaBuildConfig:
             subnet_ids=set(config.get("subnet_ids", set())),
             security_group_ids=set(config.get("security_group_ids", set())),
             managed_queue_extras=list(config.get("managed_queue_extras", list())),
+            queue_age_alarm=(
+                QueueAgeAlarmConfig.new(config["queue_age_alarm"])
+                if config.get("queue_age_alarm") is not None
+                else None
+            ),
             method_settings=list(config.get("method_settings", list())),
             tags=config.get("tags", dict()),
             logging_config=config.get("logging_config", dict()),
@@ -97,6 +148,7 @@ class AsyncLambdaBuildConfig:
             certificate_arn=config.get("certificate_arn"),
             hosted_zone_id=config.get("hosted_zone_id"),
             auto_create_acm_certificate=config.get("auto_create_acm_certificate"),
+            image_uri=config.get("image_uri"),
         )
 
     def merge(self, other: "AsyncLambdaBuildConfig"):
@@ -118,6 +170,10 @@ class AsyncLambdaBuildConfig:
             self.hosted_zone_id = other.hosted_zone_id
         if other.auto_create_acm_certificate is not None:
             self.auto_create_acm_certificate = other.auto_create_acm_certificate
+        if other.queue_age_alarm is not None:
+            self.queue_age_alarm = other.queue_age_alarm
+        if other.image_uri is not None:
+            self.image_uri = other.image_uri
 
     @property
     def function_properties(self):

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/async_lambda/cli.py

@@ -108,8 +108,16 @@ def import_module_get_controller(module_name: str, config, stage: Optional[str])
 @click.option(
     "-e", "--extras", help="The path to a json file to merge the output template with."
 )
+@click.option(
+    "--image-uri",
+    help="Package the functions as this Lambda container image instead of a zip bundle. Overrides `image_uri` in the config.",
+)
 def build(
-    module: str, output: str, stage: Optional[str] = None, extras: Optional[str] = None
+    module: str,
+    output: str,
+    stage: Optional[str] = None,
+    extras: Optional[str] = None,
+    image_uri: Optional[str] = None,
 ):
     """
     Builds/generates the SAM template for the given module.
@@ -119,6 +127,10 @@ def build(
     config_file = dir.joinpath(".async_lambda/config.json")
     if config_file.exists():
         config = json.loads(config_file.read_bytes())
+    if image_uri is not None:
+        config["image_uri"] = image_uri
+        for stage_config in config.get("stages", {}).values():
+            stage_config.pop("image_uri", None)
     controller = import_module_get_controller(
         module_name=module, config=config, stage=stage
     )
@@ -157,6 +169,11 @@ def build(
 
     if os.path.exists(".async_lambda/build"):
         shutil.rmtree(".async_lambda/build")
+
+    if get_build_config_for_stage(config, stage).image_uri is not None:
+        click.echo("Container image configured, skipping zip bundle.")
+        return
+
     os.makedirs(".async_lambda/build/packages", exist_ok=True)
 
     if dir.joinpath("requirements.txt").exists():

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/async_lambda/controller.py

@@ -20,7 +20,11 @@ from typing import (
 from uuid import uuid4
 
 from . import env
-from .build_config import AsyncLambdaBuildConfig, get_build_config_for_stage
+from .build_config import (
+    AsyncLambdaBuildConfig,
+    QueueAgeAlarmConfig,
+    get_build_config_for_stage,
+)
 from .client import get_s3_client, get_scheduler_client, get_sqs_client
 from .config import config
 from .middleware import MET, RT, MiddlewareFunction, MiddlewareRegistration
@@ -46,6 +50,9 @@ logger = logging.getLogger(__name__)
 
 _SQS_MAX_DELAY_SECONDS = 900
 
+# CloudWatch rejects an alarm carrying more than 10 metrics.
+_MAX_METRICS_PER_ALARM = 10
+
 BaseEventT = TypeVar("BaseEventT", bound=BaseEvent)
 APIEventT = TypeVar("APIEventT", bound=APIEvent)
 ManagedSQSEventT = TypeVar("ManagedSQSEventT", bound=ManagedSQSEvent)
@@ -231,6 +238,92 @@ class AsyncLambdaController:
             + f"{config.name}-*"
         }
 
+    def _alarmable_queue_logical_ids(self) -> List[str]:
+        """
+        Logical IDs of every queue the queue-age alarms should watch.
+
+        Covers each managed queue lane plus the app DLQ. The delay DLQ is left out
+        because it is named deterministically and so can be alarmed on directly.
+        """
+        logical_ids = []
+        for task in self.tasks.values():
+            if task.trigger_type not in MANAGED_SQS_TASK_TYPES:
+                continue
+            for lane_index in range(task.get_lane_count()):
+                logical_ids.append(task.get_managed_queue_logical_id(lane=lane_index))
+        logical_ids.append("AsyncLambdaDLQ")
+        return logical_ids
+
+    @staticmethod
+    def _build_queue_age_alarms(
+        queue_logical_ids: List[str],
+        alarm_config: QueueAgeAlarmConfig,
+    ) -> Dict[str, dict]:
+        """
+        Builds alarms watching the oldest message age across every managed queue.
+
+        Queues are grouped rather than alarmed individually.
+
+        Each alarm watches `MAX(METRICS())` over its own group. `METRICS()` resolves to just that alarm's
+        metrics.
+
+        Grouping is required because CloudWatch caps an alarm at 10 metrics.
+        """
+        alarms = {}
+        for alarm_index, group_start in enumerate(
+            range(0, len(queue_logical_ids), _MAX_METRICS_PER_ALARM)
+        ):
+            group = queue_logical_ids[
+                group_start : group_start + _MAX_METRICS_PER_ALARM
+            ]
+            metrics: List[dict] = [
+                {
+                    "Id": f"q{metric_index}",
+                    "ReturnData": False,
+                    "MetricStat": {
+                        "Metric": {
+                            "Namespace": "AWS/SQS",
+                            "MetricName": "ApproximateAgeOfOldestMessage",
+                            "Dimensions": [
+                                {
+                                    "Name": "QueueName",
+                                    "Value": {
+                                        "Fn::GetAtt": [queue_logical_id, "QueueName"]
+                                    },
+                                }
+                            ],
+                        },
+                        "Period": alarm_config.period,
+                        "Stat": "Maximum",
+                    },
+                }
+                for metric_index, queue_logical_id in enumerate(group)
+            ]
+            metrics.append(
+                {
+                    "Id": "oldest_message_age",
+                    "Expression": "MAX(METRICS())",
+                    "ReturnData": True,
+                }
+            )
+            alarms[f"AsyncLambdaQueueAgeAlarm{alarm_index}"] = {
+                "Type": "AWS::CloudWatch::Alarm",
+                "Properties": {
+                    "ActionsEnabled": True,
+                    "AlarmActions": alarm_config.alarm_actions,
+                    "OKActions": alarm_config.ok_actions,
+                    "AlarmDescription": (
+                        f"Oldest message age across {len(group)} {config.name} queues"
+                    ),
+                    "ComparisonOperator": "GreaterThanThreshold",
+                    "EvaluationPeriods": 1,
+                    "Metrics": metrics,
+                    "Threshold": alarm_config.threshold,
+                    "TreatMissingData": "notBreaching",
+                },
+            }
+        return alarms
+
     @classmethod
     def _build_shared_function_role(
         cls,
@@ -527,6 +620,26 @@ class AsyncLambdaController:
         Generates the SAM Template for this project.
         """
         build_config = get_build_config_for_stage(config_dict, stage)
+        if build_config.image_uri is not None and len(build_config.layers) > 0:
+            raise Exception(
+                "Lambda layers cannot be used with container images (image_uri is set)."
+            )
+        if build_config.image_uri is not None and any(
+            task.snap_start for task in self.tasks.values()
+        ):
+            raise Exception(
+                "SnapStart cannot be used with container images (image_uri is set)."
+            )
+        # SAM Globals does not accept PackageType or ImageConfig, so image functions
+        # carry their package properties individually (see AsyncLambdaTask.get_sam_template).
+        package_properties = (
+            {}
+            if build_config.image_uri is not None
+            else {
+                "Handler": f"{module}.lambda_handler",
+                "Runtime": config.runtime,
+            }
+        )
         s3_bucket_properties = {}
         if config.s3_payload_retention:
             s3_bucket_properties["LifecycleConfiguration"] = {
@@ -545,8 +658,7 @@ class AsyncLambdaController:
             "Globals": {
                 "Function": {
                     "Tags": build_config.tags,
-                    "Handler": f"{module}.lambda_handler",
-                    "Runtime": config.runtime,
+                    **package_properties,
                     "Environment": {
                         "Variables": {
                             "ASYNC_LAMBDA_PAYLOAD_S3_BUCKET": {
@@ -736,6 +848,7 @@ class AsyncLambdaController:
                 task_ref_policy_ids=list(task_ref_policies.keys()),
                 config_dict=config_dict,
                 stage=stage,
+                module=module,
             ).items():
                 template["Resources"][logical_id] = resource
 
@@ -744,6 +857,14 @@ class AsyncLambdaController:
                 self._dlq_extras_replace_references(extra)
             )
 
+        if build_config.queue_age_alarm is not None:
+            template["Resources"].update(
+                self._build_queue_age_alarms(
+                    queue_logical_ids=self._alarmable_queue_logical_ids(),
+                    alarm_config=build_config.queue_age_alarm,
+                )
+            )
+
         if has_api_tasks:
             properties: dict = {
                 "StageName": "prod",

{async_lambda_unstable-0.6.13 → async_lambda_unstable-0.6.15}/async_lambda/models/task.py

@@ -14,7 +14,11 @@ from typing import (
 )
 
 from .. import env
-from ..build_config import get_build_config_for_task, get_override_build_config_for_task
+from ..build_config import (
+    get_build_config_for_stage,
+    get_build_config_for_task,
+    get_override_build_config_for_task,
+)
 from ..config import config
 from ..util import make_cf_tags
 
@@ -28,6 +32,8 @@ from .events.managed_sqs_event import ManagedSQSEvent
 from .events.scheduled_event import ScheduledEvent
 from .events.unmanaged_sqs_event import UnmanagedSQSEvent
 
+IMAGE_FUNCTION_NAME_SUFFIX = "-image"
+
 
 class TaskTriggerType(Enum):
     """
@@ -548,6 +554,7 @@ class AsyncLambdaTask(Generic[EventType, RT]):
         task_ref_policy_ids: List[str],
         config_dict: dict,
         stage: Optional[str] = None,
+        module: Optional[str] = None,
     ) -> dict:
         """
         Generate an AWS SAM (Serverless Application Model) template for the current async lambda task.
@@ -568,6 +575,19 @@ class AsyncLambdaTask(Generic[EventType, RT]):
         override_config = get_override_build_config_for_task(
             config=config_dict, task_id=self.task_id, stage=stage
         )
+        if override_config.image_uri is not None:
+            raise Exception(
+                f"Task {self.task_id}: image_uri can only be set at the app or stage level, "
+                "because every task is packaged the same way."
+            )
+        image_uri = get_build_config_for_stage(config_dict, stage).image_uri
+        if image_uri is not None:
+            if len(build_config.layers) > 0:
+                raise Exception(
+                    f"Task {self.task_id}: Lambda layers cannot be used with container images (image_uri is set)."
+                )
+            if module is None:
+                raise Exception("A module name is required to build image functions.")
         events = self.get_template_events()
         policy_sqs_resources = self.get_policy_sqs_resources()
 
@@ -622,8 +642,20 @@ class AsyncLambdaTask(Generic[EventType, RT]):
             "FunctionName": self.get_function_name(),
             "Timeout": self.timeout,
             "Events": events,
-            "CodeUri": ".async_lambda/build/deployment.zip",
         }
+        if image_uri is not None:
+            # Lambda cannot change a function's package type in place, so switching between zip
+            # and image replaces the function, and a replacement cannot reuse the old name.
+            function_properties["FunctionName"] = (
+                f"{self.get_function_name()}{IMAGE_FUNCTION_NAME_SUFFIX}"
+            )
+            function_properties["PackageType"] = "Image"
+            function_properties["ImageUri"] = image_uri
+            function_properties["ImageConfig"] = {
+                "Command": [f"{module}.lambda_handler"]
+            }
+        else:
+            function_properties["CodeUri"] = ".async_lambda/build/deployment.zip"
 
         # cannot use shared role if the task has its own policies
         if config.shared_function_role and len(override_config.policies) == 0: