async-lambda-unstable 0.4.1__tar.gz → 0.4.3__tar.gz

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: async-lambda-unstable
-Version: 0.4.1
+Version: 0.4.3
 Summary: A framework for creating AWS Lambda Async Workflows. - Unstable Branch
 Author-email: "Nuclei, Inc" <engineering@nuclei.ai>
 Description-Content-Type: text/markdown
@@ -191,24 +191,57 @@ def api_task(event: APIEvent):
 
 # Middleware
 
-Middleware functions can be registered with controllers which will execute before the task code is run.
-These functions can be configured to trigger on specific types of tasks and can modify the `event`.
+Middleware functions can be registered with controllers which will wrap the execution of tasks.
+These functions can be configured to trigger on specific types of tasks and can trigger
+side effects and modify the `event` or `response` objects.
 
-Middleware functions must have the signature `Callable[[BaseEvent], BaseEvent]`.
-Middleware functions are run in the order which they were registered and parent controller middleware will be run first.
-Middleware functions which are registered more than once will only be run once.
+Middleware functions must have the signature `Callable[[BaseEvent, Callable[[BaseEvent], T]], T]`.
+The first argument is the `event`, and the second argument (`call_next`) is a function which will propagate the
+calls down the middleware/task stack. The `call_next` function must be called, and its result in most cases be returned.
+If this is not done then tasks will not run as expected.
+
+**Extreme care should be taken with middleware as a simple mistake can have catastrophic effects.**
+
+- Middleware functions are run in the order which they were registered and parent controller middleware will be run first.
+
+- Middleware functions which are registered more than once will only be run once.
 
 Registration can be done when the `AsyncLambdaController` is initialized with the parameter `middleware` or by using the `add_middleware` method.
 
+Middleware functions have three sections:
+
+1. Pre task
+2. Task execution
+3. Post task
+
+```python
+def async_lambda_middleware(event: BaseEvent, call_next):
+    # pre task
+    result = call_next(event) # task execution
+    # post task
+    return result
+```
+
+If there are multiple middleware functions then `call_next` will actually be calling the next middleware function in the stack.
+
+For example if there is middleware functions `A` and `B` registered in that order.
+Then the execution order would go:
+
+`A(Pre)` -> `B(Pre)` -> `Task` -> `B(Post)` -> `A(Post)`
+
 EX:
 
 ```python
-def async_task_only_middleware(event: ManagedSQSEvent):
+def async_task_only_middleware(event: ManagedSQSEvent, call_next):
     print(f"Invocation Payload: {event}")
-    return event
-
-def all_task_types_middleware(event: BaseEvent):
-    print(f"This task is of the type {type(event)}")
+    result = call_next(event)
+    print(f"Invocation Result: {result}")
+    return result
+
+def all_task_types_middleware(event: BaseEvent, call_next):
+    print(f"This event is of the type {type(event)}")
+    result = call_next(event)
+    print(f"The result is of the type {type(result)}")
     return event
 
 controller = AsyncLambdaController(middleware=[([BaseEvent], all_task_types_middleware)])

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/README.md

@@ -183,24 +183,57 @@ def api_task(event: APIEvent):
 
 # Middleware
 
-Middleware functions can be registered with controllers which will execute before the task code is run.
-These functions can be configured to trigger on specific types of tasks and can modify the `event`.
+Middleware functions can be registered with controllers which will wrap the execution of tasks.
+These functions can be configured to trigger on specific types of tasks and can trigger
+side effects and modify the `event` or `response` objects.
 
-Middleware functions must have the signature `Callable[[BaseEvent], BaseEvent]`.
-Middleware functions are run in the order which they were registered and parent controller middleware will be run first.
-Middleware functions which are registered more than once will only be run once.
+Middleware functions must have the signature `Callable[[BaseEvent, Callable[[BaseEvent], T]], T]`.
+The first argument is the `event`, and the second argument (`call_next`) is a function which will propagate the
+calls down the middleware/task stack. The `call_next` function must be called, and its result in most cases be returned.
+If this is not done then tasks will not run as expected.
+
+**Extreme care should be taken with middleware as a simple mistake can have catastrophic effects.**
+
+- Middleware functions are run in the order which they were registered and parent controller middleware will be run first.
+
+- Middleware functions which are registered more than once will only be run once.
 
 Registration can be done when the `AsyncLambdaController` is initialized with the parameter `middleware` or by using the `add_middleware` method.
 
+Middleware functions have three sections:
+
+1. Pre task
+2. Task execution
+3. Post task
+
+```python
+def async_lambda_middleware(event: BaseEvent, call_next):
+    # pre task
+    result = call_next(event) # task execution
+    # post task
+    return result
+```
+
+If there are multiple middleware functions then `call_next` will actually be calling the next middleware function in the stack.
+
+For example if there is middleware functions `A` and `B` registered in that order.
+Then the execution order would go:
+
+`A(Pre)` -> `B(Pre)` -> `Task` -> `B(Post)` -> `A(Post)`
+
 EX:
 
 ```python
-def async_task_only_middleware(event: ManagedSQSEvent):
+def async_task_only_middleware(event: ManagedSQSEvent, call_next):
     print(f"Invocation Payload: {event}")
-    return event
-
-def all_task_types_middleware(event: BaseEvent):
-    print(f"This task is of the type {type(event)}")
+    result = call_next(event)
+    print(f"Invocation Result: {result}")
+    return result
+
+def all_task_types_middleware(event: BaseEvent, call_next):
+    print(f"This event is of the type {type(event)}")
+    result = call_next(event)
+    print(f"The result is of the type {type(result)}")
     return event
 
 controller = AsyncLambdaController(middleware=[([BaseEvent], all_task_types_middleware)])

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/async_lambda/__init__.py

@@ -13,4 +13,4 @@ from .models.events.managed_sqs_event import ManagedSQSEvent as ManagedSQSEvent
 from .models.events.scheduled_event import ScheduledEvent as ScheduledEvent
 from .models.events.unmanaged_sqs_event import UnmanagedSQSEvent as UnmanagedSQSEvent
 
-__version__ = "0.4.1"
+__version__ = "0.4.3"

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/async_lambda/controller.py

@@ -4,12 +4,13 @@ import logging
 import re
 import time
 from datetime import datetime, timezone
-from typing import Any, Callable, Dict, List, Optional, Tuple, Type, TypeVar
+from typing import Any, Callable, Dict, List, Optional, Tuple, Type
 from uuid import uuid4
 
 from . import env
 from .build_config import get_build_config_for_stage
 from .client import get_s3_client, get_sqs_client
+from .middleware import MET, RT, MiddlewareFunction, MiddlewareRegistration
 from .models.events.api_event import APIEvent
 from .models.events.base_event import BaseEvent
 from .models.events.dynamodb_event import DynamoDBEvent
@@ -23,11 +24,6 @@ from .util import make_cf_tags
 
 logger = logging.getLogger(__name__)
 
-MET = TypeVar("MET", bound=BaseEvent)
-
-MiddlewareFunction = Callable[[MET], MET]
-MiddlewareRegistration = Tuple[List[Type[MET]], MiddlewareFunction[MET]]
-
 
 class AsyncLambdaController:
     is_sub: bool
@@ -39,6 +35,7 @@ class AsyncLambdaController:
     current_invocation_id: Optional[str] = None
     parent_controller: Optional["AsyncLambdaController"] = None
     middleware: List[MiddlewareRegistration]
+
     dlq_task_id: Optional[str] = None
 
     def __init__(
@@ -47,32 +44,33 @@ class AsyncLambdaController:
         lane_count: Optional[int] = None,
         propagate_lane_assignment: Optional[bool] = None,
         middleware: Optional[List[MiddlewareRegistration]] = None,
+        exception_handler: Optional[Callable[[Exception], None]] = None,
     ):
         self.tasks = dict()
         self.is_sub = is_sub
         self.lane_count = lane_count
         self.propagate_lane_assignment = propagate_lane_assignment
         self.middleware = middleware or list()
+        self.exception_handler = exception_handler
 
     def add_middleware(
-        self, event_types: List[Type[BaseEvent]], func: MiddlewareFunction[MET]
+        self, event_types: List[Type[BaseEvent]], func: MiddlewareFunction[MET, RT]
     ):
         self.middleware.append((event_types, func))
 
-    def execute_middleware(
-        self, event: MET, _ran_fns: Optional[List[Callable]] = None
-    ) -> Tuple[MET, List[Callable]]:
-        if _ran_fns is None:
-            _ran_fns = list()
+    def get_middleware_for_event(self, event: MET) -> List[MiddlewareFunction[MET, RT]]:
         if self.parent_controller is not None:
-            event, _ran_fns = self.parent_controller.execute_middleware(event, _ran_fns)
+            _middleware_functions = self.parent_controller.get_middleware_for_event(
+                event
+            )
+        else:
+            _middleware_functions = list()
+
         for event_types, func in self.middleware:
-            if func in _ran_fns:
-                continue
             if any(isinstance(event, event_type) for event_type in event_types):
-                event = func(event)
-                _ran_fns.append(func)
-        return event, _ran_fns
+                _middleware_functions.append(func)
+
+        return _middleware_functions
 
     def add_task(self, task: AsyncLambdaTask):
         """
@@ -411,7 +409,7 @@ class AsyncLambdaController:
         """
         Invoke an Async-Lambda task.
         """
-        self._async_invoke(
+        return self._async_invoke(
             destination_task_id=task_id,
             payload=payload,
             delay=delay,

async-lambda-unstable-0.4.3/async_lambda/middleware.py

@@ -0,0 +1,30 @@
+from typing import Callable, Generic, List, Tuple, Type, TypeVar
+
+from .models.events.base_event import BaseEvent
+
+MET = TypeVar("MET", bound=BaseEvent)
+RT = TypeVar("RT")
+
+MiddlewareFunction = Callable[[MET, Callable[[MET], RT]], RT]
+MiddlewareRegistration = Tuple[List[Type[MET]], MiddlewareFunction[MET, RT]]
+
+
+class MiddlewareStackExecutor(Generic[MET, RT]):
+    def __init__(
+        self,
+        middleware: List[MiddlewareFunction],
+        final: Callable[[MET], RT],
+    ):
+        self.middleware = middleware.copy()
+        self.final = final
+        self._ran_fns = list()
+
+    def call_next(self, event: MET) -> RT:
+        while True:
+            if len(self.middleware) == 0:
+                return self.final(event)
+            next_fn = self.middleware.pop(0)
+            if next_fn in self._ran_fns:
+                continue
+            self._ran_fns.append(next_fn)
+            return next_fn(event, self.call_next)

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/async_lambda/models/task.py

@@ -13,6 +13,7 @@ from ..config import config
 if TYPE_CHECKING:
     from ..controller import AsyncLambdaController  # pragma: not covered
 
+from ..middleware import RT, MiddlewareStackExecutor
 from .events.dynamodb_event import DynamoDBEvent
 from .events.managed_sqs_event import ManagedSQSEvent
 from .events.scheduled_event import ScheduledEvent
@@ -33,7 +34,7 @@ EventType = TypeVar(
 )
 
 
-class AsyncLambdaTask(Generic[EventType]):
+class AsyncLambdaTask(Generic[EventType, RT]):
     controller: "AsyncLambdaController"
     task_id: str
     trigger_type: TaskTriggerType
@@ -46,12 +47,12 @@ class AsyncLambdaTask(Generic[EventType]):
     init_tasks: List[Union[Callable[[str], Any], Callable[[], Any]]]
     _has_run_init_tasks: bool
 
-    executable: Callable[[EventType], Any]
+    executable: Callable[[EventType], RT]
 
     def __init__(
         self,
         controller: "AsyncLambdaController",
-        executable: Callable[[EventType], Any],
+        executable: Callable[[EventType], RT],
         task_id: str,
         trigger_type: TaskTriggerType,
         trigger_config: Optional[dict] = None,
@@ -481,10 +482,13 @@ class AsyncLambdaTask(Generic[EventType]):
             return self.controller.get_task(self.trigger_config["dlq_task_id"])
         return self.controller.get_dlq_task()
 
-    def execute(self, event: EventType) -> Any:
+    def execute(self, event: EventType) -> RT:
         """
         Executes the tasks function
         """
         self._run_init_tasks()
-        self.controller.execute_middleware(event)
-        return self.executable(event)
+        middleware = self.controller.get_middleware_for_event(event)
+        middleware_stack_executor = MiddlewareStackExecutor[EventType, RT](
+            middleware=middleware, final=self.executable
+        )
+        return middleware_stack_executor.call_next(event)

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/async_lambda_unstable.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: async-lambda-unstable
-Version: 0.4.1
+Version: 0.4.3
 Summary: A framework for creating AWS Lambda Async Workflows. - Unstable Branch
 Author-email: "Nuclei, Inc" <engineering@nuclei.ai>
 Description-Content-Type: text/markdown
@@ -191,24 +191,57 @@ def api_task(event: APIEvent):
 
 # Middleware
 
-Middleware functions can be registered with controllers which will execute before the task code is run.
-These functions can be configured to trigger on specific types of tasks and can modify the `event`.
+Middleware functions can be registered with controllers which will wrap the execution of tasks.
+These functions can be configured to trigger on specific types of tasks and can trigger
+side effects and modify the `event` or `response` objects.
 
-Middleware functions must have the signature `Callable[[BaseEvent], BaseEvent]`.
-Middleware functions are run in the order which they were registered and parent controller middleware will be run first.
-Middleware functions which are registered more than once will only be run once.
+Middleware functions must have the signature `Callable[[BaseEvent, Callable[[BaseEvent], T]], T]`.
+The first argument is the `event`, and the second argument (`call_next`) is a function which will propagate the
+calls down the middleware/task stack. The `call_next` function must be called, and its result in most cases be returned.
+If this is not done then tasks will not run as expected.
+
+**Extreme care should be taken with middleware as a simple mistake can have catastrophic effects.**
+
+- Middleware functions are run in the order which they were registered and parent controller middleware will be run first.
+
+- Middleware functions which are registered more than once will only be run once.
 
 Registration can be done when the `AsyncLambdaController` is initialized with the parameter `middleware` or by using the `add_middleware` method.
 
+Middleware functions have three sections:
+
+1. Pre task
+2. Task execution
+3. Post task
+
+```python
+def async_lambda_middleware(event: BaseEvent, call_next):
+    # pre task
+    result = call_next(event) # task execution
+    # post task
+    return result
+```
+
+If there are multiple middleware functions then `call_next` will actually be calling the next middleware function in the stack.
+
+For example if there is middleware functions `A` and `B` registered in that order.
+Then the execution order would go:
+
+`A(Pre)` -> `B(Pre)` -> `Task` -> `B(Post)` -> `A(Post)`
+
 EX:
 
 ```python
-def async_task_only_middleware(event: ManagedSQSEvent):
+def async_task_only_middleware(event: ManagedSQSEvent, call_next):
     print(f"Invocation Payload: {event}")
-    return event
-
-def all_task_types_middleware(event: BaseEvent):
-    print(f"This task is of the type {type(event)}")
+    result = call_next(event)
+    print(f"Invocation Result: {result}")
+    return result
+
+def all_task_types_middleware(event: BaseEvent, call_next):
+    print(f"This event is of the type {type(event)}")
+    result = call_next(event)
+    print(f"The result is of the type {type(result)}")
     return event
 
 controller = AsyncLambdaController(middleware=[([BaseEvent], all_task_types_middleware)])

{async-lambda-unstable-0.4.1 → async-lambda-unstable-0.4.3}/async_lambda_unstable.egg-info/SOURCES.txt

@@ -8,6 +8,7 @@ async_lambda/config.py
 async_lambda/controller.py
 async_lambda/defer.py
 async_lambda/env.py
+async_lambda/middleware.py
 async_lambda/py.typed
 async_lambda/util.py
 async_lambda/models/__init__.py