async-lambda-unstable 0.3.2__tar.gz → 0.3.5__tar.gz

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: async-lambda-unstable
-Version: 0.3.2
+Version: 0.3.5
 Summary: A framework for creating AWS Lambda Async Workflows. - Unstable Branch
 Author-email: "Nuclei, Inc" <engineering@nuclei.ai>
 Description-Content-Type: text/markdown
@@ -47,7 +47,7 @@ All task decorators share common arguments for configuring the underlying lambda
 - `memory: int = 128` Sets the memory allocation for the function.
 - `timeout: int = 60` Sets the timeout for the function (max 900 seconds).
 - `ephemeral_storage: int = 512` Sets the ephemeral storage allocation for the function.
-- `maximum_concurrency: Optional[int] = None` Sets the maximum concurrency value for the SQS trigger for the function. (only applies to `async_task` and `sqs_task` tasks.)
+- `maximum_concurrency: Optional[int | List[int]] = None` Sets the maximum concurrency value for the SQS trigger for the function. (only applies to `async_task` and `sqs_task` tasks.) When using the `lanes` feature, this can be a list of maximum concurrency for each lane. The length of the list must equal the # of lanes.
 
 ## Async Task
 
@@ -82,6 +82,34 @@ def task_1(event: ManagedSQSEvent):
     app.async_invoke("Task1", {})
 ```
 
+### Lanes
+
+Sometimes you may want multiple "lanes" for events to travel through, especially when you have constrained throughput with `maximum_concurrency`. Utilize the `lanes` feature to open up multiple paths to an `async-task`. This can be useful if you have a large backlog of messages you need to process, but you don't want to interrupt the normal message flow.
+
+The # of lanes can be controlled at the controller, sub-controller, and/or task level. With the configuration propagating down the tree, but it can be overridden at any of the levels. The # of lanes can be set with the `lane_count` parameter.
+
+By default all usages of `async_invoke` will place the message in the default lane (`0`). To change this specify `lane=` in the `async_invoke` call. By default, any further calls of `async_invoke` down the call stack will continue to put the messages into the same lane if it is available. You can turn of this behavior by setting `propagate_lane_assignment=False` at the controller level.
+
+For example, we will use a payload field to determine which lane processing should occur in. We will set the maximum concurrency for the default lane at 10, and for the other lane at `2`.
+
+```python
+app = AsyncLambdaController(lane_count=2)
+
+@app.async_task("SwitchBoard")
+def switch_board(event: ManagedSQSEvent):
+    value = event.payload['value']
+    lane = 0
+    if value > 50_000:
+        lane = 1
+    app.async_invoke("ProcessingTask", event.payload, lane=lane)
+
+@app.async_task("ProcessingTask", maximum_concurrency=[10, 2])
+def processing_task(event: ManagedSQSEvent):
+    ...
+```
+
+`async-lambda` creates `n` queues and lambda triggers per `async-task` where `n = lane_count`. All of the `n` queues are still consumed by a single lambda function.
+
 ## Unmanaged SQS Task
 
 Unmanaged SQS tasks consume from any arbitrary SQS queue (1 message per invocation).
@@ -243,6 +271,20 @@ so that you can reference the extras and the associated managed sqs resource by
 - `$QUEUEID"` will be replaced with the `LogicalId` of the associated Managed SQS queue.
 - `$EXTRA<index>` will be replaced with the `LogicalId` of the extra at the specified index.
 
+## `method_settings`
+
+**This config value can only be set at the app or stage level.**
+
+```
+[
+    {...}
+]
+```
+
+If your `async-lambda` app contains any `api_task` tasks, then a `AWS::Serverless::Api` resource is created.
+
+The value is passed into the [`MethodSettings`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-methodsettings) property of the `AWS::Serverless::Api`. The spec for `MethodSetting` can be found [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apigateway-stage-methodsetting.html).
+
 ## `domain_name`
 
 **This config value can only be set at the app or stage level.**
@@ -281,6 +323,22 @@ If your `async-lambda` app contains any `api_task` tasks, then a `AWS::Serverles
 
 This config value will set the [`CertificateArn`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-api-domainconfiguration.html) field of the [`Domain` property](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-domain)
 
+## `tags`
+
+```
+{
+    "TAG_NAME": "TAG_VALUE"
+}
+```
+
+This config value will set the `Tags` field of all resources created by async-lambda. This will not set the field on `managed_queue_extras` resources.
+
+The keys `framework` and `framework-version` will always be set and the system values will override any values set by the user.
+
+For managed queues the tags `async-lambda-queue-type` will be set to `dlq`, `dlq-task`, or `managed` depending on the queue type.
+
+For `async_task` queues (non dlq-task) the `async-lambda-lane` will be set.
+
 # Building an `async-lambda` app
 
 **When the app is packaged for lambda, only the main module, and the `vendor` and `src` directories are included.**

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/README.md

@@ -39,7 +39,7 @@ All task decorators share common arguments for configuring the underlying lambda
 - `memory: int = 128` Sets the memory allocation for the function.
 - `timeout: int = 60` Sets the timeout for the function (max 900 seconds).
 - `ephemeral_storage: int = 512` Sets the ephemeral storage allocation for the function.
-- `maximum_concurrency: Optional[int] = None` Sets the maximum concurrency value for the SQS trigger for the function. (only applies to `async_task` and `sqs_task` tasks.)
+- `maximum_concurrency: Optional[int | List[int]] = None` Sets the maximum concurrency value for the SQS trigger for the function. (only applies to `async_task` and `sqs_task` tasks.) When using the `lanes` feature, this can be a list of maximum concurrency for each lane. The length of the list must equal the # of lanes.
 
 ## Async Task
 
@@ -74,6 +74,34 @@ def task_1(event: ManagedSQSEvent):
     app.async_invoke("Task1", {})
 ```
 
+### Lanes
+
+Sometimes you may want multiple "lanes" for events to travel through, especially when you have constrained throughput with `maximum_concurrency`. Utilize the `lanes` feature to open up multiple paths to an `async-task`. This can be useful if you have a large backlog of messages you need to process, but you don't want to interrupt the normal message flow.
+
+The # of lanes can be controlled at the controller, sub-controller, and/or task level. With the configuration propagating down the tree, but it can be overridden at any of the levels. The # of lanes can be set with the `lane_count` parameter.
+
+By default all usages of `async_invoke` will place the message in the default lane (`0`). To change this specify `lane=` in the `async_invoke` call. By default, any further calls of `async_invoke` down the call stack will continue to put the messages into the same lane if it is available. You can turn of this behavior by setting `propagate_lane_assignment=False` at the controller level.
+
+For example, we will use a payload field to determine which lane processing should occur in. We will set the maximum concurrency for the default lane at 10, and for the other lane at `2`.
+
+```python
+app = AsyncLambdaController(lane_count=2)
+
+@app.async_task("SwitchBoard")
+def switch_board(event: ManagedSQSEvent):
+    value = event.payload['value']
+    lane = 0
+    if value > 50_000:
+        lane = 1
+    app.async_invoke("ProcessingTask", event.payload, lane=lane)
+
+@app.async_task("ProcessingTask", maximum_concurrency=[10, 2])
+def processing_task(event: ManagedSQSEvent):
+    ...
+```
+
+`async-lambda` creates `n` queues and lambda triggers per `async-task` where `n = lane_count`. All of the `n` queues are still consumed by a single lambda function.
+
 ## Unmanaged SQS Task
 
 Unmanaged SQS tasks consume from any arbitrary SQS queue (1 message per invocation).
@@ -235,6 +263,20 @@ so that you can reference the extras and the associated managed sqs resource by
 - `$QUEUEID"` will be replaced with the `LogicalId` of the associated Managed SQS queue.
 - `$EXTRA<index>` will be replaced with the `LogicalId` of the extra at the specified index.
 
+## `method_settings`
+
+**This config value can only be set at the app or stage level.**
+
+```
+[
+    {...}
+]
+```
+
+If your `async-lambda` app contains any `api_task` tasks, then a `AWS::Serverless::Api` resource is created.
+
+The value is passed into the [`MethodSettings`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-methodsettings) property of the `AWS::Serverless::Api`. The spec for `MethodSetting` can be found [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apigateway-stage-methodsetting.html).
+
 ## `domain_name`
 
 **This config value can only be set at the app or stage level.**
@@ -273,6 +315,22 @@ If your `async-lambda` app contains any `api_task` tasks, then a `AWS::Serverles
 
 This config value will set the [`CertificateArn`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-api-domainconfiguration.html) field of the [`Domain` property](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-domain)
 
+## `tags`
+
+```
+{
+    "TAG_NAME": "TAG_VALUE"
+}
+```
+
+This config value will set the `Tags` field of all resources created by async-lambda. This will not set the field on `managed_queue_extras` resources.
+
+The keys `framework` and `framework-version` will always be set and the system values will override any values set by the user.
+
+For managed queues the tags `async-lambda-queue-type` will be set to `dlq`, `dlq-task`, or `managed` depending on the queue type.
+
+For `async_task` queues (non dlq-task) the `async-lambda-lane` will be set.
+
 # Building an `async-lambda` app
 
 **When the app is packaged for lambda, only the main module, and the `vendor` and `src` directories are included.**

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda/__init__.py

@@ -9,4 +9,4 @@ from .models.events.managed_sqs_event import ManagedSQSEvent as ManagedSQSEvent
 from .models.events.scheduled_event import ScheduledEvent as ScheduledEvent
 from .models.events.unmanaged_sqs_event import UnmanagedSQSEvent as UnmanagedSQSEvent
 
-__version__ = "0.3.2"
+__version__ = "0.3.5"

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda/build_config.py

@@ -2,6 +2,12 @@ from dataclasses import dataclass
 from typing import Dict, List, Optional, Set, Union
 
 
+def make_default_tags() -> Dict[str, str]:
+    from . import __version__
+
+    return {"framework": "async-lambda", "framework-version": __version__}
+
+
 @dataclass
 class AsyncLambdaBuildConfig:
     environment_variables: Dict[str, str]
@@ -10,6 +16,8 @@ class AsyncLambdaBuildConfig:
     subnet_ids: Set[str]
     security_group_ids: Set[str]
     managed_queue_extras: List[dict]
+    method_settings: List[dict]
+    tags: Dict[str, str]
     domain_name: Optional[str] = None
     tls_version: Optional[str] = None
     certificate_arn: Optional[str] = None
@@ -23,6 +31,8 @@ class AsyncLambdaBuildConfig:
             subnet_ids=set(config.get("subnet_ids", set())),
             security_group_ids=set(config.get("security_group_ids", set())),
             managed_queue_extras=list(config.get("managed_queue_extras", list())),
+            method_settings=list(config.get("method_settings", list())),
+            tags=config.get("tags", dict()),
             domain_name=config.get("domain_name"),
             tls_version=config.get("tls_version"),
             certificate_arn=config.get("certificate_arn"),
@@ -35,6 +45,7 @@ class AsyncLambdaBuildConfig:
         self.subnet_ids.update(other.subnet_ids)
         self.security_group_ids.update(other.security_group_ids)
         self.managed_queue_extras += other.managed_queue_extras
+        self.tags.update(other.tags)
         if other.domain_name is not None:
             self.domain_name = other.domain_name
         if other.tls_version is not None:
@@ -52,6 +63,7 @@ def get_build_config_for_stage(
         stage_config = config.setdefault("stages", {}).setdefault(stage, {})
         build_config.merge(AsyncLambdaBuildConfig.new(stage_config))
 
+    build_config.tags.update(make_default_tags())
     return build_config
 
 
@@ -72,5 +84,5 @@ def get_build_config_for_task(
                 stage, {}
             )
             build_config.merge(AsyncLambdaBuildConfig.new(task_stage_config))
-
+    build_config.tags.update(make_default_tags())
     return build_config

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda/controller.py

@@ -17,21 +17,32 @@ from .models.events.unmanaged_sqs_event import UnmanagedSQSEvent
 from .models.mock.mock_context import MockLambdaContext
 from .models.mock.mock_event import MockSQSLambdaEvent
 from .models.task import AsyncLambdaTask, TaskTriggerType
+from .util import make_cf_tags
 
 logger = logging.getLogger(__name__)
 
 
 class AsyncLambdaController:
     is_sub: bool
+    lane_count: Optional[int] = None
+    propagate_lane_assignment: Optional[bool] = None
     tasks: Dict[str, AsyncLambdaTask]
     current_task_id: Optional[str] = None
+    current_lane: Optional[int] = None
     current_invocation_id: Optional[str] = None
     parent_controller: Optional["AsyncLambdaController"] = None
     dlq_task_id: Optional[str] = None
 
-    def __init__(self, is_sub: bool = False):
+    def __init__(
+        self,
+        is_sub: bool = False,
+        lane_count: Optional[int] = None,
+        propagate_lane_assignment: Optional[bool] = None,
+    ):
         self.tasks = dict()
         self.is_sub = is_sub
+        self.lane_count = lane_count
+        self.propagate_lane_assignment = propagate_lane_assignment
 
     def add_task(self, task: AsyncLambdaTask):
         """
@@ -43,6 +54,21 @@ class AsyncLambdaController:
             )
         self.tasks[task.task_id] = task
 
+    def get_lane_count(self) -> int:
+        if self.lane_count is not None:
+            return self.lane_count
+
+        if self.parent_controller is not None:
+            return self.parent_controller.get_lane_count()
+        return 1
+
+    def should_propagate_lane_assignment(self) -> bool:
+        if self.propagate_lane_assignment is not None:
+            return self.propagate_lane_assignment
+        if self.parent_controller is not None:
+            return self.parent_controller.should_propagate_lane_assignment()
+        return True
+
     def get_task(self, task_id: str) -> Optional[AsyncLambdaTask]:
         """
         Retrieve a task by task_id from this or any parent controllers.
@@ -81,16 +107,26 @@ class AsyncLambdaController:
         """
         Generates the SAM Template for this project.
         """
+        build_config = get_build_config_for_stage(config_dict, stage)
         template = {
             "AWSTemplateFormatVersion": "2010-09-09",
             "Transform": "AWS::Serverless-2016-10-31",
             "Resources": {
                 "AsyncLambdaPayloadBucket": {
                     "Type": "AWS::S3::Bucket",
+                    "Properties": {"Tags": make_cf_tags(build_config.tags)},
                 },
                 "AsyncLambdaDLQ": {
                     "Type": "AWS::SQS::Queue",
-                    "Properties": {"MessageRetentionPeriod": 1_209_600},  # 14 days
+                    "Properties": {
+                        "MessageRetentionPeriod": 1_209_600,  # 14 days
+                        "Tags": make_cf_tags(
+                            {
+                                **build_config.tags,
+                                "async-lambda-queue-type": "dlq",
+                            }
+                        ),
+                    },
                 },
             },
         }
@@ -104,14 +140,19 @@ class AsyncLambdaController:
             ).items():
                 template["Resources"][logical_id] = resource
 
-        build_config = get_build_config_for_stage(config_dict, stage)
         for extra_index, extra in enumerate(build_config.managed_queue_extras):
             template["Resources"][
                 self._dlq_extra_logical_id(extra_index)
             ] = self._dlq_extras_replace_references(extra)
 
         if has_api_tasks:
-            properties: dict = {"StageName": "prod"}
+            properties: dict = {
+                "StageName": "prod",
+                "PropagateTags": True,
+                "Tags": build_config.tags,
+            }
+            if len(build_config.method_settings) > 0:
+                properties["MethodSettings"] = build_config.method_settings
             if build_config.domain_name is not None:
                 properties["Domain"] = {
                     "DomainName": build_config.domain_name,
@@ -150,6 +191,17 @@ class AsyncLambdaController:
         """
         self.current_task_id = task_id
 
+    def set_current_lane(self, lane: int):
+        """
+        Set the current lane
+        """
+        self.current_lane = lane
+
+    def get_current_lane(self) -> int:
+        if self.current_lane is None:
+            return 0
+        return self.current_lane
+
     def set_current_invocation_id(self, invocation_id: str):
         """
         Set the current_invocation_id
@@ -160,6 +212,7 @@ class AsyncLambdaController:
         """
         Direct the invocation to the task executor.
         """
+        self.current_lane = None
         if task_id is None:
             task_id = env.get_current_task_id()
         task = self.tasks[task_id]
@@ -168,6 +221,16 @@ class AsyncLambdaController:
 
         if task.trigger_type == TaskTriggerType.MANAGED_SQS:
             _event = ManagedSQSEvent(*args)
+            lane_count = task.get_lane_count()
+            if lane_count == 1:
+                self.set_current_lane(lane=0)
+            else:
+                for lane_index in range(lane_count):
+                    if _event.event_source_arn == task.get_managed_queue_arn(
+                        lane=lane_index
+                    ):
+                        self.set_current_lane(lane=lane_index)
+                        break
             self.set_current_invocation_id(_event.invocation_id)
         elif task.trigger_type == TaskTriggerType.UNMANAGED_SQS:
             _event = UnmanagedSQSEvent(*args)
@@ -192,6 +255,7 @@ class AsyncLambdaController:
         payload: Any,
         delay: Optional[int] = None,
         force_sync: bool = False,
+        lane: Optional[int] = None,
     ):
         """
         Invoke an 'async-lambda' task asynchronously utilizing it's SQS queue
@@ -202,16 +266,29 @@ class AsyncLambdaController:
                 payload=payload,
                 delay=delay,
                 force_sync=force_sync,
+                lane=lane,
             )
         if destination_task_id not in self.tasks:
             raise Exception(
                 f"No such task exists with the task_id {destination_task_id}"
             )
+
         destination_task = self.tasks[destination_task_id]
         if destination_task.trigger_type != TaskTriggerType.MANAGED_SQS:
             raise Exception(
                 f"Unable to invoke task '{destination_task_id}' because it is a {destination_task.trigger_type} task"
             )
+
+        if lane is None and destination_task.should_propagate_lane_assignment():
+            lane = self.get_current_lane()
+        if lane is None:
+            lane = 0
+
+        if lane < 0 or lane >= destination_task.get_lane_count():
+            raise Exception(
+                f"Unable to invoke task {destination_task_id} in lane {lane} because it is not a valid lane for the task."
+            )
+
         if self.current_invocation_id is None:
             invocation_id = str(uuid4())
         else:
@@ -225,14 +302,22 @@ class AsyncLambdaController:
         )
 
         logger.info(
-            f"Invoking task '{destination_task.task_id}' - invocation_id '{invocation_id}' - delay {delay or 0} - size ({payload_size})"
+            f"Invoking task '{destination_task.task_id}' - invocation_id '{invocation_id}' - delay {delay or 0} - size ({payload_size}) - lane {lane}"
         )
         if force_sync or env.get_force_sync_mode():
             if delay:
                 time.sleep(delay)
             # Sync invocation with mock event/context
             current_task_id = self.current_task_id
-            mock_event = MockSQSLambdaEvent(sqs_body)
+            queue_arn = None
+            if current_task_id is not None:
+                current_task = self.get_task(current_task_id)
+                if (
+                    current_task is not None
+                    and current_task.trigger_type == TaskTriggerType.MANAGED_SQS
+                ):
+                    queue_arn = current_task.get_managed_queue_arn(lane=lane)
+            mock_event = MockSQSLambdaEvent(sqs_body, source_queue_arn=queue_arn)
             mock_context = MockLambdaContext(destination_task.task_id)
             response = self.handle_invocation(
                 mock_event, mock_context, task_id=destination_task_id
@@ -241,7 +326,7 @@ class AsyncLambdaController:
             return response
         else:
             get_sqs_client().send_message(
-                QueueUrl=destination_task.get_managed_queue_url(),
+                QueueUrl=destination_task.get_managed_queue_url(lane=lane),
                 MessageBody=sqs_body,
                 DelaySeconds=delay,
             )
@@ -289,6 +374,7 @@ class AsyncLambdaController:
         payload: Any,
         delay: Optional[int] = 0,
         force_sync: bool = False,
+        lane: Optional[int] = None,
     ):
         """
         Invoke an Async-Lambda task.
@@ -298,6 +384,7 @@ class AsyncLambdaController:
             payload=payload,
             delay=delay,
             force_sync=force_sync,
+            lane=lane,
         )
 
     def async_lambda_handler(self, event, context):
@@ -312,6 +399,8 @@ class AsyncLambdaController:
         max_receive_count: int = 1,
         dlq_task_id: Optional[str] = None,
         is_dlq_task: bool = False,
+        lane_count: Optional[int] = None,
+        propagate_lane_assignment: Optional[bool] = None,
         **kwargs,
     ):
         """
@@ -337,6 +426,8 @@ class AsyncLambdaController:
                         "max_receive_count": max_receive_count,
                         "dlq_task_id": dlq_task_id,
                         "is_dlq_task": is_dlq_task,
+                        "lane_count": lane_count,
+                        "propagate_lane_assignment": propagate_lane_assignment,
                     },
                     **kwargs,
                 )

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda/models/mock/mock_event.py

@@ -3,8 +3,10 @@ from typing import List, Optional, Tuple
 from urllib.parse import parse_qs, parse_qsl
 
 
-def MockSQSLambdaEvent(body: str) -> dict:
+def MockSQSLambdaEvent(body: str, source_queue_arn: Optional[str] = None) -> dict:
     now_timestamp = int(time())
+    if not source_queue_arn:
+        source_queue_arn = "arn:aws:sqs:us-east-1:123456789012:my-queue"
     return {
         "Records": [
             {
@@ -20,7 +22,7 @@ def MockSQSLambdaEvent(body: str) -> dict:
                 "messageAttributes": {},
                 "md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
                 "eventSource": "aws:sqs",
-                "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:my-queue",
+                "eventSourceARN": source_queue_arn,
                 "awsRegion": "us-east-1",
             }
         ]

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda/models/task.py

@@ -3,6 +3,8 @@ import re
 from enum import Enum
 from typing import TYPE_CHECKING, Any, Callable, Generic, List, Optional, TypeVar, Union
 
+from async_lambda.util import make_cf_tags
+
 from .. import env
 from ..build_config import get_build_config_for_task
 from ..config import config
@@ -36,7 +38,7 @@ class AsyncLambdaTask(Generic[EventType]):
     timeout: int
     memory: int
     ephemeral_storage: int
-    maximum_concurrency: Optional[int]
+    maximum_concurrency: Optional[Union[int, List[int]]]
 
     executable: Callable[[EventType], Any]
 
@@ -50,7 +52,7 @@ class AsyncLambdaTask(Generic[EventType]):
         timeout: int = 60,
         memory: int = 128,
         ephemeral_storage: int = 512,
-        maximum_concurrency: Optional[int] = None,
+        maximum_concurrency: Optional[Union[int, List[int]]] = None,
     ):
         AsyncLambdaTask.validate_task_id(task_id)
         self.controller = controller
@@ -86,61 +88,119 @@ class AsyncLambdaTask(Generic[EventType]):
         if len(task_id) > 32:
             raise ValueError("Task ID must be less than 32 characters long.")
 
-    def get_managed_queue_name(self):
+    def get_lane_count(self) -> int:
+        if self.trigger_type != TaskTriggerType.MANAGED_SQS:
+            raise Exception(f"The task {self.task_id} is not a managed queue task.")
+        if "lane_count" in self.trigger_config and isinstance(
+            self.trigger_config["lane_count"], int
+        ):
+            return self.trigger_config["lane_count"]
+        return self.controller.get_lane_count()
+
+    def should_propagate_lane_assignment(self) -> bool:
+        if self.trigger_type != TaskTriggerType.MANAGED_SQS:
+            raise Exception(f"The task {self.task_id} is not a managed queue task.")
+        if "propagate_lane_assignment" in self.trigger_config and isinstance(
+            self.trigger_config["propagate_lane_assignment"], bool
+        ):
+            return self.trigger_config["propagate_lane_assignment"]
+        return self.controller.should_propagate_lane_assignment()
+
+    def get_managed_queue_name(self, lane: int = 0):
         """
         Returns the managed queue's name for this task.
         """
         if self.trigger_type != TaskTriggerType.MANAGED_SQS:
             raise Exception(f"The task {self.task_id} is not a managed queue task.")
-        return f"{config.name}-{self.task_id}"
+        if lane == 0:
+            return f"{config.name}-{self.task_id}"
+        return f"{config.name}-{self.task_id}-L{lane}"
 
     def get_function_name(self):
         return f"{config.name}-{self.task_id}"
 
-    def get_managed_queue_arn(self):
+    def get_managed_queue_arn(self, lane: int = 0):
         if self.trigger_type != TaskTriggerType.MANAGED_SQS:
             raise Exception(f"The task {self.task_id} is not a managed queue task.")
-        return f"arn:aws:sqs:{env.get_aws_region()}:{env.get_aws_account_id()}:{self.get_managed_queue_name()}"
+        return f"arn:aws:sqs:{env.get_aws_region()}:{env.get_aws_account_id()}:{self.get_managed_queue_name(lane=lane)}"
 
-    def get_managed_queue_url(self):
+    def get_managed_queue_url(self, lane: int = 0):
         if self.trigger_type != TaskTriggerType.MANAGED_SQS:
             raise Exception(f"The task {self.task_id} is not a managed queue task.")
-        return f"https://sqs.{env.get_aws_region()}.amazonaws.com/{env.get_aws_account_id()}/{self.get_managed_queue_name()}"
+        return f"https://sqs.{env.get_aws_region()}.amazonaws.com/{env.get_aws_account_id()}/{self.get_managed_queue_name(lane=lane)}"
 
     def get_function_logical_id(self):
         return f"{self.task_id}ALFunc"
 
-    def get_managed_queue_logical_id(self):
+    def get_managed_queue_logical_id(self, lane: int = 0):
         if self.trigger_type != TaskTriggerType.MANAGED_SQS:
             raise Exception(f"The task {self.task_id} is not a managed queue task.")
-        return f"{self.task_id}ALQueue"
+        if lane == 0:
+            return f"{self.task_id}ALQueue"
+        return f"{self.task_id}ALQueueL{lane}"
 
-    def get_managed_queue_extra_logical_id(self, index: int):
-        return f"{self.get_function_logical_id()}Extra{index}"
+    def get_managed_queue_extra_logical_id(self, index: int, lane: int = 0):
+        if self.trigger_type != TaskTriggerType.MANAGED_SQS:
+            raise Exception(f"The task {self.task_id} is not a managed queue task.")
+        if lane == 0:
+            return f"{self.get_function_logical_id()}Extra{index}"
+        return f"{self.get_function_logical_id()}Extra{index}L{lane}"
 
-    def get_template_events(self):
+    def get_managed_queue_event_logical_id(self, lane: int = 0):
+        if self.trigger_type != TaskTriggerType.MANAGED_SQS:
+            raise Exception(f"The task {self.task_id} is not a managed queue task.")
+        if lane == 0:
+            return "ManagedSQS"
+        return f"ManagedSQSL{lane}"
+
+    def get_template_events(self) -> dict:
         sqs_properties = {}
+        if (
+            isinstance(self.maximum_concurrency, list)
+            and self.trigger_type != TaskTriggerType.MANAGED_SQS
+        ):
+            raise Exception(
+                f"Invalid maximum concurrency configuration for task {self.task_id}. Must be an int, not a list of ints. Lanes are only supported for ManagedSQS tasks."
+            )
+        if (
+            isinstance(self.maximum_concurrency, list)
+            and self.trigger_type == TaskTriggerType.MANAGED_SQS
+            and len(self.maximum_concurrency) != self.get_lane_count()
+        ):
+            raise Exception(
+                f"Invalid maximum concurrency configuration for task {self.task_id}. The list of maximum concurrency must be equal to the # of lanes for the task."
+            )
         if self.maximum_concurrency is not None:
             sqs_properties["ScalingConfig"] = {
                 "MaximumConcurrency": self.maximum_concurrency
             }
         if self.trigger_type == TaskTriggerType.MANAGED_SQS:
-            return {
-                "ManagedSQS": {
+            events = {}
+            for lane_index in range(self.get_lane_count()):
+                sqs_properties = {}
+                if isinstance(self.maximum_concurrency, list):
+                    sqs_properties["ScalingConfig"] = {
+                        "MaximumConcurrency": self.maximum_concurrency[lane_index]
+                    }
+                elif self.maximum_concurrency is not None:
+                    sqs_properties["ScalingConfig"] = {
+                        "MaximumConcurrency": self.maximum_concurrency
+                    }
+                events[self.get_managed_queue_event_logical_id(lane=lane_index)] = {
                     "Type": "SQS",
                     "Properties": {
                         "BatchSize": 1,
                         "Enabled": True,
                         "Queue": {
                             "Fn::GetAtt": [
-                                self.get_managed_queue_logical_id(),
+                                self.get_managed_queue_logical_id(lane=lane_index),
                                 "Arn",
                             ]
                         },
                         **sqs_properties,
                     },
                 }
-            }
+            return events
         elif self.trigger_type == TaskTriggerType.UNMANAGED_SQS:
             return {
                 "UnmanagedSQS": {
@@ -178,15 +238,16 @@ class AsyncLambdaTask(Generic[EventType]):
             }
         raise NotImplementedError()
 
-    def get_policy_sqs_resources(self):
+    def get_policy_sqs_resources(self) -> List[dict]:
         if self.trigger_type == TaskTriggerType.MANAGED_SQS:
             return [
                 {
                     "Fn::GetAtt": [
-                        self.get_managed_queue_logical_id(),
+                        self.get_managed_queue_logical_id(lane=lane_index),
                         "Arn",
                     ]
                 }
+                for lane_index in range(self.get_lane_count())
             ]
         elif self.trigger_type == TaskTriggerType.UNMANAGED_SQS:
             return [self.trigger_config["queue_arn"]]
@@ -223,24 +284,20 @@ class AsyncLambdaTask(Generic[EventType]):
                 },
             },
         ]
-        managed_tasks_logical_ids = [
-            _task.get_managed_queue_logical_id()
-            for _task in tasks
-            if _task.trigger_type == TaskTriggerType.MANAGED_SQS
+        managed_tasks_resources = [
+            resource
+            for task in tasks
+            if task.trigger_type == TaskTriggerType.MANAGED_SQS
+            for resource in task.get_policy_sqs_resources()
         ]
-        if len(managed_tasks_logical_ids) > 0:
+        if len(managed_tasks_resources) > 0:
             policy_statements.append(
                 {
                     "Effect": "Allow",
                     "Action": ["sqs:SendMessage"],
                     "Resource": [
-                        {
-                            "Fn::GetAtt": [
-                                queue_logical_id,
-                                "Arn",
-                            ]
-                        }
-                        for queue_logical_id in managed_tasks_logical_ids
+                        managed_tasks_resource
+                        for managed_tasks_resource in managed_tasks_resources
                     ],
                 },
             )
@@ -260,15 +317,15 @@ class AsyncLambdaTask(Generic[EventType]):
             )
         function_properties = {}
         if len(build_config.layers) > 0:
-            function_properties["Layers"] = list(build_config.layers)
+            function_properties["Layers"] = sorted(build_config.layers)
         if len(build_config.security_group_ids) > 0 or len(build_config.subnet_ids) > 0:
             function_properties["VpcConfig"] = {}
             if len(build_config.security_group_ids) > 0:
-                function_properties["VpcConfig"]["SecurityGroupIds"] = list(
+                function_properties["VpcConfig"]["SecurityGroupIds"] = sorted(
                     build_config.security_group_ids
                 )
             if len(build_config.subnet_ids) > 0:
-                function_properties["VpcConfig"]["SubnetIds"] = list(
+                function_properties["VpcConfig"]["SubnetIds"] = sorted(
                     build_config.subnet_ids
                 )
 
@@ -276,6 +333,7 @@ class AsyncLambdaTask(Generic[EventType]):
             self.get_function_logical_id(): {
                 "Type": "AWS::Serverless::Function",
                 "Properties": {
+                    "Tags": build_config.tags,
                     "Handler": f"{module}.lambda_handler",
                     "Runtime": config.runtime,
                     "Environment": {
@@ -319,33 +377,50 @@ class AsyncLambdaTask(Generic[EventType]):
                         "Arn",
                     ]
                 }
-            template[self.get_managed_queue_logical_id()] = {
-                "Type": "AWS::SQS::Queue",
-                "Properties": {
-                    "QueueName": self.get_managed_queue_name(),
-                    "RedrivePolicy": {
-                        "deadLetterTargetArn": dead_letter_target_arn,
-                        "maxReceiveCount": self.trigger_config["max_receive_count"],
+            for lane_index in range(self.get_lane_count()):
+                _extra_tags = {
+                    "async-lambda-lane": str(lane_index),
+                    "async-lambda-queue-type": "managed",
+                }
+                if (
+                    self.trigger_type == TaskTriggerType.MANAGED_SQS
+                    and self.trigger_config["is_dlq_task"]
+                ):
+                    _extra_tags["async-lambda-queue-type"] = "dlq-task"
+                template[self.get_managed_queue_logical_id(lane=lane_index)] = {
+                    "Type": "AWS::SQS::Queue",
+                    "Properties": {
+                        "Tags": make_cf_tags({**build_config.tags, **_extra_tags}),
+                        "QueueName": self.get_managed_queue_name(lane=lane_index),
+                        "RedrivePolicy": {
+                            "deadLetterTargetArn": dead_letter_target_arn,
+                            "maxReceiveCount": self.trigger_config["max_receive_count"],
+                        },
+                        "VisibilityTimeout": self.timeout,
                     },
-                    "VisibilityTimeout": self.timeout,
-                },
-            }
-            for extra_index, extra in enumerate(build_config.managed_queue_extras):
-                template[
-                    self.get_managed_queue_extra_logical_id(extra_index)
-                ] = self._managed_queue_extras_replace_references(extra)
+                }
+                for extra_index, extra in enumerate(build_config.managed_queue_extras):
+                    template[
+                        self.get_managed_queue_extra_logical_id(
+                            extra_index, lane=lane_index
+                        )
+                    ] = self._managed_queue_extras_replace_references(
+                        extra, lane=lane_index
+                    )
 
         return template
 
-    def _managed_queue_extras_replace_references(self, extra: dict) -> dict:
+    def _managed_queue_extras_replace_references(self, extra: dict, lane: int) -> dict:
         stringified_extra = json.dumps(extra)
         stringified_extra = re.sub(
             r"\$EXTRA(?P<index>[0-9]+)",
-            lambda m: self.get_managed_queue_extra_logical_id(int(m.group("index"))),
+            lambda m: self.get_managed_queue_extra_logical_id(
+                int(m.group("index")), lane=lane
+            ),
             stringified_extra,
         )
         stringified_extra = stringified_extra.replace(
-            "$QUEUEID", self.get_managed_queue_logical_id()
+            "$QUEUEID", self.get_managed_queue_logical_id(lane=lane)
         )
 
         return json.loads(stringified_extra)

async-lambda-unstable-0.3.5/async_lambda/util.py

@@ -0,0 +1,8 @@
+from typing import Dict, List
+
+
+def make_cf_tags(tags: Dict[str, str]) -> List[Dict[str, str]]:
+    _tags = []
+    for key, value in tags.items():
+        _tags.append({"Key": key, "Value": value})
+    return _tags

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda_unstable.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: async-lambda-unstable
-Version: 0.3.2
+Version: 0.3.5
 Summary: A framework for creating AWS Lambda Async Workflows. - Unstable Branch
 Author-email: "Nuclei, Inc" <engineering@nuclei.ai>
 Description-Content-Type: text/markdown
@@ -47,7 +47,7 @@ All task decorators share common arguments for configuring the underlying lambda
 - `memory: int = 128` Sets the memory allocation for the function.
 - `timeout: int = 60` Sets the timeout for the function (max 900 seconds).
 - `ephemeral_storage: int = 512` Sets the ephemeral storage allocation for the function.
-- `maximum_concurrency: Optional[int] = None` Sets the maximum concurrency value for the SQS trigger for the function. (only applies to `async_task` and `sqs_task` tasks.)
+- `maximum_concurrency: Optional[int | List[int]] = None` Sets the maximum concurrency value for the SQS trigger for the function. (only applies to `async_task` and `sqs_task` tasks.) When using the `lanes` feature, this can be a list of maximum concurrency for each lane. The length of the list must equal the # of lanes.
 
 ## Async Task
 
@@ -82,6 +82,34 @@ def task_1(event: ManagedSQSEvent):
     app.async_invoke("Task1", {})
 ```
 
+### Lanes
+
+Sometimes you may want multiple "lanes" for events to travel through, especially when you have constrained throughput with `maximum_concurrency`. Utilize the `lanes` feature to open up multiple paths to an `async-task`. This can be useful if you have a large backlog of messages you need to process, but you don't want to interrupt the normal message flow.
+
+The # of lanes can be controlled at the controller, sub-controller, and/or task level. With the configuration propagating down the tree, but it can be overridden at any of the levels. The # of lanes can be set with the `lane_count` parameter.
+
+By default all usages of `async_invoke` will place the message in the default lane (`0`). To change this specify `lane=` in the `async_invoke` call. By default, any further calls of `async_invoke` down the call stack will continue to put the messages into the same lane if it is available. You can turn of this behavior by setting `propagate_lane_assignment=False` at the controller level.
+
+For example, we will use a payload field to determine which lane processing should occur in. We will set the maximum concurrency for the default lane at 10, and for the other lane at `2`.
+
+```python
+app = AsyncLambdaController(lane_count=2)
+
+@app.async_task("SwitchBoard")
+def switch_board(event: ManagedSQSEvent):
+    value = event.payload['value']
+    lane = 0
+    if value > 50_000:
+        lane = 1
+    app.async_invoke("ProcessingTask", event.payload, lane=lane)
+
+@app.async_task("ProcessingTask", maximum_concurrency=[10, 2])
+def processing_task(event: ManagedSQSEvent):
+    ...
+```
+
+`async-lambda` creates `n` queues and lambda triggers per `async-task` where `n = lane_count`. All of the `n` queues are still consumed by a single lambda function.
+
 ## Unmanaged SQS Task
 
 Unmanaged SQS tasks consume from any arbitrary SQS queue (1 message per invocation).
@@ -243,6 +271,20 @@ so that you can reference the extras and the associated managed sqs resource by
 - `$QUEUEID"` will be replaced with the `LogicalId` of the associated Managed SQS queue.
 - `$EXTRA<index>` will be replaced with the `LogicalId` of the extra at the specified index.
 
+## `method_settings`
+
+**This config value can only be set at the app or stage level.**
+
+```
+[
+    {...}
+]
+```
+
+If your `async-lambda` app contains any `api_task` tasks, then a `AWS::Serverless::Api` resource is created.
+
+The value is passed into the [`MethodSettings`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-methodsettings) property of the `AWS::Serverless::Api`. The spec for `MethodSetting` can be found [here](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-apigateway-stage-methodsetting.html).
+
 ## `domain_name`
 
 **This config value can only be set at the app or stage level.**
@@ -281,6 +323,22 @@ If your `async-lambda` app contains any `api_task` tasks, then a `AWS::Serverles
 
 This config value will set the [`CertificateArn`](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-api-domainconfiguration.html) field of the [`Domain` property](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-api.html#sam-api-domain)
 
+## `tags`
+
+```
+{
+    "TAG_NAME": "TAG_VALUE"
+}
+```
+
+This config value will set the `Tags` field of all resources created by async-lambda. This will not set the field on `managed_queue_extras` resources.
+
+The keys `framework` and `framework-version` will always be set and the system values will override any values set by the user.
+
+For managed queues the tags `async-lambda-queue-type` will be set to `dlq`, `dlq-task`, or `managed` depending on the queue type.
+
+For `async_task` queues (non dlq-task) the `async-lambda-lane` will be set.
+
 # Building an `async-lambda` app
 
 **When the app is packaged for lambda, only the main module, and the `vendor` and `src` directories are included.**

{async-lambda-unstable-0.3.2 → async-lambda-unstable-0.3.5}/async_lambda_unstable.egg-info/SOURCES.txt

@@ -8,6 +8,7 @@ async_lambda/config.py
 async_lambda/controller.py
 async_lambda/env.py
 async_lambda/py.typed
+async_lambda/util.py
 async_lambda/models/__init__.py
 async_lambda/models/task.py
 async_lambda/models/events/__init__.py