jararaca 0.2.37a12__tar.gz → 0.3.0__tar.gz

{jararaca-0.2.37a12 → jararaca-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: jararaca
-Version: 0.2.37a12
+Version: 0.3.0
 Summary: A simple and fast API framework for Python
 Author: Lucas S
 Author-email: me@luscasleo.dev
@@ -12,6 +12,7 @@ Classifier: Programming Language :: Python :: 3.13
 Provides-Extra: docs
 Provides-Extra: http
 Provides-Extra: opentelemetry
+Provides-Extra: watch
 Requires-Dist: aio-pika (>=9.4.3,<10.0.0)
 Requires-Dist: croniter (>=3.0.3,<4.0.0)
 Requires-Dist: fastapi (>=0.113.0,<0.114.0)
@@ -27,6 +28,7 @@ Requires-Dist: types-croniter (>=3.0.3.20240731,<4.0.0.0)
 Requires-Dist: types-redis (>=4.6.0.20240903,<5.0.0.0)
 Requires-Dist: uvicorn (>=0.30.6,<0.31.0)
 Requires-Dist: uvloop (>=0.20.0,<0.21.0)
+Requires-Dist: watchdog (>=3.0.0,<4.0.0) ; extra == "watch"
 Requires-Dist: websockets (>=13.0.1,<14.0.0)
 Project-URL: Repository, https://github.com/LuscasLeo/jararaca
 Description-Content-Type: text/markdown

{jararaca-0.2.37a12 → jararaca-0.3.0}/docs/index.md

@@ -40,6 +40,19 @@ Starts a message bus worker that processes asynchronous messages from a message
 - `--queue`: Queue name (default: "jararaca_q")
 - `--prefetch-count`: Number of messages to prefetch (default: 1)
 
+### `worker_v2` - Enhanced Message Bus Worker
+
+```bash
+jararaca worker_v2 APP_PATH [OPTIONS]
+```
+
+Starts an enhanced version of the message bus worker with improved backend support.
+
+**Options:**
+
+- `--broker-url`: The URL for the message broker (required)
+- `--backend-url`: The URL for the message broker backend (required)
+
 ### `server` - HTTP Server
 
 ```bash
@@ -83,6 +96,11 @@ uvicorn app_module:asgi_app
 
 Starts a FastAPI HTTP server for your microservice.
 
+**Options:**
+
+- `--host`: Host to bind the server (default: "0.0.0.0")
+- `--port`: Port to bind the server (default: 8000)
+
 ### `scheduler` - Task Scheduler
 
 ```bash
@@ -95,6 +113,20 @@ Runs scheduled tasks defined in your application using cron expressions.
 
 - `--interval`: Polling interval in seconds (default: 1)
 
+### `scheduler_v2` - Enhanced Task Scheduler
+
+```bash
+jararaca scheduler_v2 APP_PATH [OPTIONS]
+```
+
+Runs an enhanced version of the task scheduler with support for message broker backend integration.
+
+**Options:**
+
+- `--interval`: Polling interval in seconds (default: 1, required)
+- `--broker-url`: The URL for the message broker (required)
+- `--backend-url`: The URL for the message broker backend (required)
+
 ### `gen-tsi` - Generate TypeScript Interfaces
 
 ```bash

jararaca-0.3.0/docs/scheduler.md

@@ -0,0 +1,216 @@
+# Jararaca Scheduler System
+
+The scheduler system in Jararaca provides robust task scheduling capabilities that allow you to run periodic tasks using cron expressions. This document explains how the scheduler works, its different implementations, and how to use it in your applications.
+
+## Overview
+
+The Jararaca scheduler allows you to:
+
+- Run background tasks at scheduled intervals
+- Use cron expressions for flexible scheduling
+- Control overlap behavior (whether to allow multiple instances of the same task)
+- Distribute scheduled tasks across multiple instances
+- Handle delayed message execution
+
+The scheduler has two implementations:
+1. **Basic Scheduler** - Simple scheduler for local execution
+2. **Enhanced Scheduler (V2)** - Distributed scheduler with improved backend support and message broker integration
+
+```mermaid
+graph TD
+    A[Microservice] --> B[Scheduler System]
+    B --> C[Basic Scheduler]
+    B --> D[Enhanced Scheduler V2]
+    
+    C --> E[Local Task Execution]
+    D --> F[Message Broker]
+    F --> G[Workers]
+    D --> H[Backend Store]
+    H --> I[Last Execution Time]
+    H --> J[Delayed Messages]
+```
+
+## Using the Scheduler
+
+### Defining Scheduled Tasks
+
+You can define scheduled tasks using the `@ScheduledAction` decorator:
+
+```python
+from jararaca import ScheduledAction
+
+class TasksController:
+    @ScheduledAction("*/5 * * * *")  # Run every 5 minutes
+    async def scheduled_task(self):
+        # Your task implementation
+        print("This runs every 5 minutes")
+        
+    @ScheduledAction("0 */2 * * *", allow_overlap=False, timeout=60)
+    async def heavy_task(self):
+        # A heavier task that shouldn't overlap
+        print("This runs every 2 hours without overlap")
+```
+
+### Scheduler Decorator Options
+
+The `@ScheduledAction` decorator accepts several parameters:
+
+- `cron`: A string representing the cron expression for the scheduled action
+- `allow_overlap`: A boolean indicating if new executions should start even if the previous one is still running (default: `False`)
+- `exclusive`: A boolean indicating if the scheduled action should be executed in only one instance of the application (requires a distributed backend, default: `True`)
+- `timeout`: An integer representing the timeout for the scheduled action in seconds (default: `None`)
+- `exception_handler`: A callable that will be called when an exception is raised during execution (default: `None`)
+
+### Cron Expressions
+
+Jararaca uses standard cron expressions for scheduling. Here are some examples:
+
+- `* * * * *` - Run every minute
+- `*/15 * * * *` - Run every 15 minutes
+- `0 * * * *` - Run at the beginning of every hour
+- `0 0 * * *` - Run at midnight every day
+- `0 0 * * 0` - Run at midnight every Sunday
+- `0 0 1 * *` - Run at midnight on the first day of every month
+
+## Basic Scheduler Implementation
+
+The basic scheduler is suitable for simpler applications where tasks run locally:
+
+```python
+from jararaca import Microservice, ScheduledAction
+
+app = Microservice(
+    # Your microservice configuration
+)
+
+# Run the scheduler
+from jararaca.scheduler.scheduler import Scheduler
+scheduler = Scheduler(app, interval=1)
+scheduler.run()
+```
+
+## Enhanced Scheduler V2 Implementation
+
+The V2 scheduler adds support for distributed execution and message broker integration, making it ideal for more complex applications:
+
+```python
+from jararaca import Microservice, ScheduledAction
+from jararaca.scheduler.scheduler_v2 import SchedulerV2
+
+app = Microservice(
+    # Your microservice configuration
+)
+
+# Run the enhanced scheduler
+scheduler = SchedulerV2(
+    app=app,
+    interval=1,
+    broker_url="amqp://guest:guest@localhost:5672/?exchange=jararaca_ex",
+    backend_url="redis://localhost:6379",
+)
+scheduler.run()
+```
+
+### Message Broker Integration
+
+The V2 scheduler uses a message broker (currently supporting RabbitMQ) to distribute tasks:
+
+1. The scheduler determines when a task should run based on its cron expression
+2. Instead of executing the task directly, it sends a message to the message broker
+3. A worker picks up the message and executes the task
+4. The backend store (Redis) tracks execution state to prevent overlap when configured
+
+This architecture allows for better scalability and reliability:
+
+```mermaid
+sequenceDiagram
+    participant S as SchedulerV2
+    participant B as Message Broker
+    participant R as Redis Backend
+    participant W as Worker
+    
+    S->>R: Check last execution time
+    R-->>S: Return last execution time
+    S->>S: Determine if task should run
+    S->>B: Publish task message
+    S->>R: Update last execution time
+    B-->>W: Deliver task message
+    W->>R: Mark task as running
+    W->>W: Execute task
+    W->>R: Mark task as completed
+```
+
+### Delayed Message Queue
+
+The V2 scheduler also supports delayed messages:
+
+```python
+from jararaca import use_publisher
+from jararaca.scheduler.types import DelayedMessageData
+
+# Schedule a message to be published at a future time
+async def schedule_reminder():
+    message = ReminderMessage(
+        user_id="123",
+        message="Don't forget your appointment!"
+    )
+    
+    # Current time + 1 hour in seconds
+    dispatch_time = int(time.time()) + 3600
+    
+    # Get publisher
+    publisher = use_publisher()
+    
+    # Schedule delayed message
+    await publisher.publish_delayed(
+        message,
+        dispatch_time=dispatch_time
+    )
+```
+
+## Redis Backend Implementation
+
+The Redis backend implementation provides:
+
+1. **Distributed Locking** - Ensures tasks only run on one instance when exclusivity is required
+2. **Execution Tracking** - Tracks the running state of tasks to prevent overlap
+3. **Delayed Message Queue** - Manages messages scheduled for future delivery
+
+The implementation uses Redis data structures:
+- Keys for last execution time and dispatch time
+- Sorted sets for delayed message queue
+- Hash sets for execution indicators
+
+## Running the Scheduler
+
+### CLI Command for Basic Scheduler
+
+```bash
+jararaca scheduler APP_PATH [OPTIONS]
+```
+
+**Options:**
+- `--interval`: Polling interval in seconds (default: 1)
+
+### CLI Command for Enhanced Scheduler (V2)
+
+```bash
+jararaca scheduler_v2 APP_PATH [OPTIONS]
+```
+
+**Options:**
+- `--interval`: Polling interval in seconds (default: 1, required)
+- `--broker-url`: The URL for the message broker (required)
+- `--backend-url`: The URL for the message broker backend (required)
+
+## Best Practices
+
+1. **Task Duration** - Be mindful of task duration, especially for frequent tasks
+2. **Error Handling** - Implement proper error handling in your tasks
+3. **Overlap Control** - Use `allow_overlap=False` for resource-intensive tasks
+4. **Timeouts** - Set appropriate timeouts to prevent stuck tasks
+5. **Monitoring** - Log task execution for monitoring purposes
+
+## Conclusion
+
+The Jararaca scheduler system provides a powerful, flexible way to implement periodic tasks in your applications. With two implementations to choose from, you can select the one that best fits your application's requirements, from simple local scheduling to complex distributed task execution.

{jararaca-0.2.37a12 → jararaca-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "jararaca"
-version = "0.2.37a12"
+version = "0.3.0"
 description = "A simple and fast API framework for Python"
 authors = ["Lucas S <me@luscasleo.dev>"]
 readme = "README.md"
@@ -27,6 +27,7 @@ opentelemetry-exporter-otlp = { version = "^1.27.0", optional = true }
 types-croniter = "^3.0.3.20240731"
 types-redis = "^4.6.0.20240903"
 mako = "^1.3.5"
+watchdog = { version = "^3.0.0", optional = true }
 
 
 [tool.poetry.extras]
@@ -39,6 +40,7 @@ opentelemetry = [
 ]
 http = ["httptools", "httpx"]
 docs = ["mkdocs-material"]
+watch = ["watchdog"]
 
 [tool.poetry.group.lint.dependencies]
 mypy = "^1.11.2"

{jararaca-0.2.37a12 → jararaca-0.3.0}/src/jararaca/__init__.py

@@ -2,6 +2,7 @@ from importlib import import_module
 from typing import TYPE_CHECKING
 
 if TYPE_CHECKING:
+    from jararaca.broker_backend.redis_broker_backend import RedisMessageBrokerBackend
     from jararaca.messagebus.bus_message_controller import (
         ack,
         nack,
@@ -60,10 +61,12 @@ if TYPE_CHECKING:
     from .messagebus.interceptors.aiopika_publisher_interceptor import (
         AIOPikaConnectionFactory,
         GenericPoolConfig,
+    )
+    from .messagebus.interceptors.publisher_interceptor import (
         MessageBusPublisherInterceptor,
     )
+    from .messagebus.message import Message, MessageOf
     from .messagebus.publisher import use_publisher
-    from .messagebus.types import Message, MessageOf
     from .messagebus.worker import MessageBusWorker
     from .microservice import Microservice, use_app_context, use_current_container
     from .persistence.base import T_BASEMODEL, BaseEntity
@@ -115,6 +118,7 @@ if TYPE_CHECKING:
     from .tools.app_config.interceptor import AppConfigurationInterceptor
 
     __all__ = [
+        "RedisMessageBrokerBackend",
         "FilterRuleApplier",
         "SortRuleApplier",
         "use_bus_message_controller",
@@ -216,6 +220,11 @@ if TYPE_CHECKING:
 __SPEC_PARENT__: str = __spec__.parent  # type: ignore
 # A mapping of {<member name>: (package, <module name>)} defining dynamic imports
 _dynamic_imports: "dict[str, tuple[str, str, str | None]]" = {
+    "RedisMessageBrokerBackend": (
+        __SPEC_PARENT__,
+        "broker_backend.redis_broker_backend",
+        None,
+    ),
     "FilterRuleApplier": (__SPEC_PARENT__, "persistence.sort_filter", None),
     "SortRuleApplier": (__SPEC_PARENT__, "persistence.sort_filter", None),
     "use_bus_message_controller": (
@@ -286,8 +295,8 @@ _dynamic_imports: "dict[str, tuple[str, str, str | None]]" = {
     ),
     "Identifiable": (__SPEC_PARENT__, "persistence.utilities", None),
     "IdentifiableEntity": (__SPEC_PARENT__, "persistence.utilities", None),
-    "MessageOf": (__SPEC_PARENT__, "messagebus.types", None),
-    "Message": (__SPEC_PARENT__, "messagebus.types", None),
+    "MessageOf": (__SPEC_PARENT__, "messagebus.message", None),
+    "Message": (__SPEC_PARENT__, "messagebus.message", None),
     "StringCriteria": (__SPEC_PARENT__, "persistence.utilities", None),
     "DateCriteria": (__SPEC_PARENT__, "persistence.utilities", None),
     "DateOrderedFilter": (__SPEC_PARENT__, "persistence.utilities", None),
@@ -339,7 +348,7 @@ _dynamic_imports: "dict[str, tuple[str, str, str | None]]" = {
     ),
     "MessageBusPublisherInterceptor": (
         __SPEC_PARENT__,
-        "messagebus.interceptors.aiopika_publisher_interceptor",
+        "messagebus.interceptors.publisher_interceptor",
         None,
     ),
     "RedisWebSocketConnectionBackend": (

jararaca-0.3.0/src/jararaca/broker_backend/__init__.py

@@ -0,0 +1,102 @@
+from abc import ABC
+from contextlib import asynccontextmanager
+from typing import AsyncContextManager, AsyncGenerator, Iterable
+
+from jararaca.scheduler.types import DelayedMessageData
+
+
+class MessageBrokerBackend(ABC):
+
+    def lock(self) -> AsyncContextManager[None]:
+        """
+        Acquire a lock for the message broker backend.
+        This is used to ensure that only one instance of the scheduler is running at a time.
+        """
+        raise NotImplementedError(f"lock() is not implemented by {self.__class__}.")
+
+    async def get_last_dispatch_time(self, action_name: str) -> int | None:
+        """
+        Get the last dispatch time of the scheduled action.
+        This is used to determine if the scheduled action should be executed again
+        or if it should be skipped.
+        """
+        raise NotImplementedError(
+            f"get_last_dispatch_time() is not implemented by {self.__class__}."
+        )
+
+    async def set_last_dispatch_time(self, action_name: str, timestamp: int) -> None:
+        """
+        Set the last dispatch time of the scheduled action.
+        This is used to determine if the scheduled action should be executed again
+        or if it should be skipped.
+        """
+        raise NotImplementedError(
+            f"set_last_dispatch_time() is not implemented by {self.__class__}."
+        )
+
+    async def get_in_execution_count(self, action_name: str) -> int:
+        """
+        Get the number of scheduled actions in execution.
+        This is used to determine if the scheduled action should be executed again
+        or if it should be skipped.
+        """
+        raise NotImplementedError(
+            f"get_in_execution_count() is not implemented by {self.__class__}."
+        )
+
+    def in_execution(self, action_name: str) -> AsyncContextManager[None]:
+        """
+        Acquire a lock for the scheduled action.
+        This is used to ensure that only one instance of the scheduled action is running at a time.
+        """
+        raise NotImplementedError(
+            f"in_execution() is not implemented by {self.__class__}."
+        )
+
+    async def dequeue_next_delayed_messages(
+        self, start_timestamp: int
+    ) -> Iterable[DelayedMessageData]:
+        """
+        Dequeue the next delayed messages from the message broker.
+        This is used to trigger the scheduled action.
+        """
+        raise NotImplementedError(
+            f"dequeue_next_delayed_messages() is not implemented by {self.__class__}."
+        )
+
+    async def enqueue_delayed_message(
+        self, delayed_message: DelayedMessageData
+    ) -> None:
+        """
+        Enqueue a delayed message to the message broker.
+        This is used to trigger the scheduled action.
+        """
+        raise NotImplementedError(
+            f"enqueue_delayed_message() is not implemented by {self.__class__}."
+        )
+
+    async def dispose(self) -> None:
+        """
+        Dispose of the message broker backend.
+        This is used to clean up resources used by the message broker backend.
+        """
+
+
+class NullBackend(MessageBrokerBackend):
+    """
+    A null backend that does nothing.
+    This is used for testing purposes.
+    """
+
+    @asynccontextmanager
+    async def lock(self) -> AsyncGenerator[None, None]:
+        yield
+
+    async def get_last_dispatch_time(self, action_name: str) -> int:
+        return 0
+
+    async def set_last_dispatch_time(self, action_name: str, timestamp: int) -> None:
+        pass
+
+    async def dispose(self) -> None:
+        pass

jararaca-0.3.0/src/jararaca/broker_backend/mapper.py

@@ -0,0 +1,21 @@
+from jararaca.broker_backend import MessageBrokerBackend
+
+
+def get_message_broker_backend_from_url(url: str) -> MessageBrokerBackend:
+    """
+    Factory function to create a message broker backend instance from a URL.
+    Currently, only Redis is supported.
+    """
+    if (
+        url.startswith("redis://")
+        or url.startswith("rediss://")
+        or url.startswith("redis-socket://")
+        or url.startswith("rediss+socket://")
+    ):
+        from jararaca.broker_backend.redis_broker_backend import (
+            RedisMessageBrokerBackend,
+        )
+
+        return RedisMessageBrokerBackend(url)
+    else:
+        raise ValueError(f"Unsupported message broker backend URL: {url}")

jararaca-0.3.0/src/jararaca/broker_backend/redis_broker_backend.py

@@ -0,0 +1,162 @@
+import logging
+import time
+from contextlib import asynccontextmanager
+from typing import AsyncGenerator, Iterable
+from uuid import uuid4
+
+import redis.asyncio
+
+from jararaca.broker_backend import MessageBrokerBackend
+from jararaca.scheduler.types import DelayedMessageData
+
+logger = logging.getLogger(__name__)
+
+
+class RedisMessageBrokerBackend(MessageBrokerBackend):
+    def __init__(self, url: str) -> None:
+        self.redis = redis.asyncio.Redis.from_url(url)
+        self.last_dispatch_time_key = "last_dispatch_time:{action_name}"
+        self.last_execution_time_key = "last_execution_time:{action_name}"
+        self.execution_indicator_key = "in_execution:{action_name}:{timestamp}"
+        self.execution_indicator_expiration = 60 * 5
+        self.delayed_messages_key = "delayed_messages"
+        self.delayed_messages_metadata_key = "delayed_messages_metadata:{task_id}"
+
+    @asynccontextmanager
+    async def lock(self) -> AsyncGenerator[None, None]:
+        yield
+
+    async def get_last_dispatch_time(self, action_name: str) -> int | None:
+
+        key = self.last_dispatch_time_key.format(action_name=action_name)
+        last_execution_time = await self.redis.get(key)
+        if last_execution_time is None:
+            return None
+        return int(last_execution_time)
+
+    async def set_last_dispatch_time(self, action_name: str, timestamp: int) -> None:
+        key = self.last_dispatch_time_key.format(action_name=action_name)
+        await self.redis.set(key, timestamp)
+
+    async def get_last_execution_time(self, action_name: str) -> int | None:
+        key = self.last_execution_time_key.format(action_name=action_name)
+        last_execution_time = await self.redis.get(key)
+        if last_execution_time is None:
+            return None
+        return int(last_execution_time)
+
+    async def set_last_execution_time(self, action_name: str, timestamp: int) -> None:
+        key = self.last_execution_time_key.format(action_name=action_name)
+        await self.redis.set(key, timestamp)
+
+    async def get_in_execution_count(self, action_name: str) -> int:
+        key = self.execution_indicator_key.format(
+            action_name=action_name, timestamp="*"
+        )
+        in_execution_count = await self.redis.keys(key)
+        if in_execution_count is None:
+            return 0
+
+        return len(in_execution_count)
+
+    @asynccontextmanager
+    async def in_execution(self, action_name: str) -> AsyncGenerator[None, None]:
+        """
+        Acquire a lock for the scheduled action.
+        This is used to ensure that only one instance of the scheduled action is running at a time.
+        """
+        key = self.execution_indicator_key.format(
+            action_name=action_name, timestamp=int(time.time())
+        )
+        await self.redis.set(key, 1, ex=self.execution_indicator_expiration)
+        try:
+            yield
+        finally:
+            await self.redis.delete(key)
+
+    async def enqueue_delayed_message(
+        self, delayed_message: DelayedMessageData
+    ) -> None:
+        """
+        Enqueue a delayed message to the message broker.
+        This is used to trigger the scheduled action.
+        """
+        task_id = str(uuid4())
+        async with self.redis.pipeline() as pipe:
+            pipe.set(
+                self.delayed_messages_metadata_key.format(task_id=task_id),
+                delayed_message.model_dump_json().encode(),
+            )
+            pipe.zadd(
+                self.delayed_messages_key,
+                {task_id: delayed_message.dispatch_time},
+                nx=True,
+            )
+            await pipe.execute()
+
+    async def dequeue_next_delayed_messages(
+        self, start_timestamp: int
+    ) -> Iterable[DelayedMessageData]:
+        """
+        Dequeue the next delayed messages from the message broker.
+        This is used to trigger the scheduled action.
+        """
+        tasks_ids = await self.redis.zrangebyscore(
+            name=self.delayed_messages_key,
+            max=start_timestamp,
+            min="-inf",
+            withscores=False,
+        )
+
+        if not tasks_ids:
+            return []
+
+        tasks_bytes_data: list[bytes] = []
+
+        for task_id_bytes in tasks_ids:
+            metadata = await self.redis.get(
+                self.delayed_messages_metadata_key.format(
+                    task_id=task_id_bytes.decode()
+                )
+            )
+            if metadata is None:
+                logger.warning(
+                    f"Delayed message metadata not found for task_id: {task_id_bytes.decode()}"
+                )
+
+                continue
+
+            tasks_bytes_data.append(metadata)
+
+        async with self.redis.pipeline() as pipe:
+            for task_id_bytes in tasks_ids:
+                pipe.zrem(self.delayed_messages_key, task_id_bytes.decode())
+                pipe.delete(
+                    self.delayed_messages_metadata_key.format(
+                        task_id=task_id_bytes.decode()
+                    )
+                )
+            await pipe.execute()
+
+        delayed_messages: list[DelayedMessageData] = []
+
+        for task_bytes_data in tasks_bytes_data:
+            try:
+                delayed_message = DelayedMessageData.model_validate_json(
+                    task_bytes_data.decode()
+                )
+                delayed_messages.append(delayed_message)
+            except Exception:
+                logger.error(
+                    f"Error parsing delayed message: {task_bytes_data.decode()}"
+                )
+                continue
+
+        return delayed_messages
+
+    async def dispose(self) -> None:
+        """
+        Dispose of the message broker backend.
+        This is used to close the connection to the message broker.
+        """
+        await self.redis.close()