ltq 0.3.1__tar.gz → 0.4.0__tar.gz

ltq-0.4.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.3
+Name: ltq
+Version: 0.4.0
+Summary: Add your description here
+Author: Tom Clesius
+Author-email: Tom Clesius <tomclesius@gmail.com>
+Requires-Dist: redis>=7.1.0
+Requires-Dist: croniter>=6.0.0 ; extra == 'scheduler'
+Requires-Dist: sentry-sdk>=2.0.0 ; extra == 'sentry'
+Requires-Python: >=3.13
+Provides-Extra: scheduler
+Provides-Extra: sentry
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/tclesius/ltq/refs/heads/main/assets/logo.png" alt="LTQ" width="400">
+</p>
+
+<p align="center">
+  A lightweight, Async-first task queue built on Redis.
+</p>
+
+## Installation
+
+```bash
+pip install ltq
+# or
+uv add ltq
+```
+
+## Broker Backends
+
+LTQ supports multiple broker backends:
+
+- **Redis** (default): `broker_url="redis://localhost:6379"`
+- **Memory**: `broker_url="memory://"` (useful for testing)
+
+All workers and schedulers accept a `broker_url` parameter.
+
+## Quick Start
+
+```python
+import asyncio
+import ltq
+
+worker = ltq.Worker("emails", broker_url="redis://localhost:6379")
+
+@worker.task()
+async def send_email(to: str, subject: str, body: str) -> None:
+    # your async code here
+    pass
+
+async def main():
+    # Enqueue a task
+    await send_email.send("user@example.com", "Hello", "World")
+
+    # Or enqueue multiple tasks
+    for email in ["a@example.com", "b@example.com"]:
+        await send_email.send(email, "Hi", "Message")
+
+asyncio.run(main())
+```
+
+Each worker has a namespace (e.g., `"emails"`), and tasks are automatically namespaced as `{namespace}:{function_name}`.
+
+## Running Workers
+
+```bash
+# Run a single worker
+ltq run myapp:worker
+
+# With options
+ltq run myapp:worker --concurrency 100 --log-level DEBUG
+```
+
+## Running an App
+
+Register multiple workers into an `App` to run them together:
+
+```python
+import ltq
+
+app = ltq.App()
+app.register_worker(emails_worker)
+app.register_worker(notifications_worker)
+```
+
+```bash
+ltq run --app myapp:app
+```
+
+### App Middleware
+
+Apply middleware globally to all workers in an app:
+
+```python
+from ltq.middleware import Sentry
+
+app = ltq.App(middlewares=[Sentry(dsn="https://...")])
+
+# Or register after creation
+app.register_middleware(Sentry(dsn="https://..."))
+app.register_middleware(MyMiddleware(), pos=0)
+
+# When workers are registered, app middlewares are prepended to each worker's stack
+app.register_worker(emails_worker)
+```
+
+### Threading Model
+
+By default, `App` runs each worker in its own thread with a separate event loop. This provides isolation between workers while keeping them in the same process. Workers won't block each other since each has its own async event loop.
+
+**For maximum isolation** (separate memory, crash protection), run each worker in its own process:
+
+```bash
+# Terminal 1
+ltq run myapp:emails_worker
+
+# Terminal 2
+ltq run myapp:notifications_worker
+```
+
+This gives you full process isolation at the cost of more overhead.
+
+## Queue Management
+
+Manage queues using the CLI:
+
+```bash
+# Clear a task queue
+ltq clear emails:send_email
+
+# Get queue size
+ltq size emails:send_email
+
+# With custom Redis URL
+ltq clear emails:send_email --redis-url redis://localhost:6380
+ltq size emails:send_email --redis-url redis://localhost:6380
+```
+
+Queue names are automatically namespaced as `{worker_name}:{function_name}`.
+
+## Scheduler
+
+Run tasks on a cron schedule (requires `ltq[scheduler]`):
+
+```python
+import ltq
+
+scheduler = ltq.Scheduler()
+scheduler.cron("*/5 * * * *", send_email.message("admin@example.com", "Report", "..."))
+scheduler.start()  # Runs scheduler in blocking mode with asyncio.run()
+```
+
+## Task Options
+
+Configure task behavior with options:
+
+```python
+from datetime import timedelta
+
+@worker.task(max_tries=3, max_age=timedelta(hours=1), max_rate="10/s")
+async def send_email(to: str, subject: str, body: str) -> None:
+    # your async code here
+    pass
+```
+
+**Available options:**
+
+- `max_tries` (int): Maximum retry attempts before rejecting the message
+- `max_age` (timedelta): Maximum message age before rejection
+- `max_rate` (str): Rate limit in format `"N/s"`, `"N/m"`, or `"N/h"` (requests per second/minute/hour)
+
+## Middleware
+
+Middleware are async context managers that wrap task execution. The default stack is `[MaxTries(), MaxAge(), MaxRate()]`, so you only need to specify middlewares if you want to customize or add additional ones:
+
+```python
+from ltq.middleware import MaxTries, MaxAge, MaxRate, Sentry
+
+worker = ltq.Worker(
+    "emails",
+    broker_url="redis://localhost:6379",
+    middlewares=[
+        MaxTries(),
+        MaxAge(),
+        MaxRate(),
+        Sentry(dsn="https://..."),
+    ],
+)
+```
+
+**Built-in:** `MaxTries`, `MaxAge`, `MaxRate`, `Sentry` (requires `ltq[sentry]`)
+
+You can also register middleware after creating the worker:
+
+```python
+worker.register_middleware(Sentry(dsn="https://..."))
+
+# Insert at specific position (default is -1 for append)
+worker.register_middleware(MyMiddleware(), pos=0)
+```
+
+**Custom middleware:**
+
+```python
+from contextlib import asynccontextmanager
+from ltq.middleware import Middleware
+from ltq.message import Message
+from ltq.task import Task
+
+class Logger(Middleware):
+    @asynccontextmanager
+    async def __call__(self, message: Message, task: Task):
+        print(f"Processing {message.task_name}")
+        yield
+        print(f"Completed {message.task_name}")
+```

ltq-0.4.0/README.md

@@ -0,0 +1,204 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/tclesius/ltq/refs/heads/main/assets/logo.png" alt="LTQ" width="400">
+</p>
+
+<p align="center">
+  A lightweight, Async-first task queue built on Redis.
+</p>
+
+## Installation
+
+```bash
+pip install ltq
+# or
+uv add ltq
+```
+
+## Broker Backends
+
+LTQ supports multiple broker backends:
+
+- **Redis** (default): `broker_url="redis://localhost:6379"`
+- **Memory**: `broker_url="memory://"` (useful for testing)
+
+All workers and schedulers accept a `broker_url` parameter.
+
+## Quick Start
+
+```python
+import asyncio
+import ltq
+
+worker = ltq.Worker("emails", broker_url="redis://localhost:6379")
+
+@worker.task()
+async def send_email(to: str, subject: str, body: str) -> None:
+    # your async code here
+    pass
+
+async def main():
+    # Enqueue a task
+    await send_email.send("user@example.com", "Hello", "World")
+
+    # Or enqueue multiple tasks
+    for email in ["a@example.com", "b@example.com"]:
+        await send_email.send(email, "Hi", "Message")
+
+asyncio.run(main())
+```
+
+Each worker has a namespace (e.g., `"emails"`), and tasks are automatically namespaced as `{namespace}:{function_name}`.
+
+## Running Workers
+
+```bash
+# Run a single worker
+ltq run myapp:worker
+
+# With options
+ltq run myapp:worker --concurrency 100 --log-level DEBUG
+```
+
+## Running an App
+
+Register multiple workers into an `App` to run them together:
+
+```python
+import ltq
+
+app = ltq.App()
+app.register_worker(emails_worker)
+app.register_worker(notifications_worker)
+```
+
+```bash
+ltq run --app myapp:app
+```
+
+### App Middleware
+
+Apply middleware globally to all workers in an app:
+
+```python
+from ltq.middleware import Sentry
+
+app = ltq.App(middlewares=[Sentry(dsn="https://...")])
+
+# Or register after creation
+app.register_middleware(Sentry(dsn="https://..."))
+app.register_middleware(MyMiddleware(), pos=0)
+
+# When workers are registered, app middlewares are prepended to each worker's stack
+app.register_worker(emails_worker)
+```
+
+### Threading Model
+
+By default, `App` runs each worker in its own thread with a separate event loop. This provides isolation between workers while keeping them in the same process. Workers won't block each other since each has its own async event loop.
+
+**For maximum isolation** (separate memory, crash protection), run each worker in its own process:
+
+```bash
+# Terminal 1
+ltq run myapp:emails_worker
+
+# Terminal 2
+ltq run myapp:notifications_worker
+```
+
+This gives you full process isolation at the cost of more overhead.
+
+## Queue Management
+
+Manage queues using the CLI:
+
+```bash
+# Clear a task queue
+ltq clear emails:send_email
+
+# Get queue size
+ltq size emails:send_email
+
+# With custom Redis URL
+ltq clear emails:send_email --redis-url redis://localhost:6380
+ltq size emails:send_email --redis-url redis://localhost:6380
+```
+
+Queue names are automatically namespaced as `{worker_name}:{function_name}`.
+
+## Scheduler
+
+Run tasks on a cron schedule (requires `ltq[scheduler]`):
+
+```python
+import ltq
+
+scheduler = ltq.Scheduler()
+scheduler.cron("*/5 * * * *", send_email.message("admin@example.com", "Report", "..."))
+scheduler.start()  # Runs scheduler in blocking mode with asyncio.run()
+```
+
+## Task Options
+
+Configure task behavior with options:
+
+```python
+from datetime import timedelta
+
+@worker.task(max_tries=3, max_age=timedelta(hours=1), max_rate="10/s")
+async def send_email(to: str, subject: str, body: str) -> None:
+    # your async code here
+    pass
+```
+
+**Available options:**
+
+- `max_tries` (int): Maximum retry attempts before rejecting the message
+- `max_age` (timedelta): Maximum message age before rejection
+- `max_rate` (str): Rate limit in format `"N/s"`, `"N/m"`, or `"N/h"` (requests per second/minute/hour)
+
+## Middleware
+
+Middleware are async context managers that wrap task execution. The default stack is `[MaxTries(), MaxAge(), MaxRate()]`, so you only need to specify middlewares if you want to customize or add additional ones:
+
+```python
+from ltq.middleware import MaxTries, MaxAge, MaxRate, Sentry
+
+worker = ltq.Worker(
+    "emails",
+    broker_url="redis://localhost:6379",
+    middlewares=[
+        MaxTries(),
+        MaxAge(),
+        MaxRate(),
+        Sentry(dsn="https://..."),
+    ],
+)
+```
+
+**Built-in:** `MaxTries`, `MaxAge`, `MaxRate`, `Sentry` (requires `ltq[sentry]`)
+
+You can also register middleware after creating the worker:
+
+```python
+worker.register_middleware(Sentry(dsn="https://..."))
+
+# Insert at specific position (default is -1 for append)
+worker.register_middleware(MyMiddleware(), pos=0)
+```
+
+**Custom middleware:**
+
+```python
+from contextlib import asynccontextmanager
+from ltq.middleware import Middleware
+from ltq.message import Message
+from ltq.task import Task
+
+class Logger(Middleware):
+    @asynccontextmanager
+    async def __call__(self, message: Message, task: Task):
+        print(f"Processing {message.task_name}")
+        yield
+        print(f"Completed {message.task_name}")
+```

{ltq-0.3.1 → ltq-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "ltq"
-version = "0.3.1"
+version = "0.4.0"
 description = "Add your description here"
 readme = "README.md"
 authors = [{ name = "Tom Clesius", email = "tomclesius@gmail.com" }]
@@ -19,7 +19,7 @@ requires = ["uv_build>=0.9.26,<0.10.0"]
 build-backend = "uv_build"
 
 [tool.bumpversion]
-current_version = "0.3.1"
+current_version = "0.4.0"
 commit = true
 tag = true
 message = "v{new_version}"

ltq-0.4.0/src/ltq/__init__.py

@@ -0,0 +1,24 @@
+from .app import App
+from .broker import Broker
+from .task import Task
+from .worker import Worker
+from .scheduler import Scheduler
+from .logger import get_logger
+from .errors import RejectError, RetryError
+from .middleware import Middleware, MaxTries, MaxAge, MaxRate, Sentry
+
+__all__ = [
+    "App",
+    "Broker",
+    "Worker",
+    "Scheduler",
+    "Task",
+    "get_logger",
+    "RejectError",
+    "RetryError",
+    "Middleware",
+    "MaxTries",
+    "MaxAge",
+    "MaxRate",
+    "Sentry",
+]

ltq-0.4.0/src/ltq/app.py

@@ -0,0 +1,41 @@
+import asyncio
+import threading
+
+from .middleware import Middleware
+from .worker import Worker
+
+
+class App:
+    def __init__(self, middlewares: list[Middleware] | None = None) -> None:
+        self.workers: dict[str, Worker] = dict()
+        self.middlewares: list[Middleware] = middlewares or []
+
+    def register_middleware(self, middleware: Middleware, pos: int = -1) -> None:
+        if pos == -1:
+            self.middlewares.append(middleware)
+        else:
+            self.middlewares.insert(pos, middleware)
+
+    def register_worker(self, worker: Worker) -> None:
+        if worker.name in self.workers:
+            raise RuntimeError(f"Worker '{worker.name}' is already registered")
+        worker.middlewares = list(self.middlewares) + worker.middlewares
+        self.workers[worker.name] = worker
+
+    @staticmethod
+    def _run_worker(worker: Worker) -> None:
+        asyncio.run(worker.run())
+
+    async def run(self) -> None:
+        threads: list[threading.Thread] = []
+        for worker in self.workers.values():
+            t = threading.Thread(target=self._run_worker, args=(worker,), daemon=True)
+            t.start()
+            threads.append(t)
+
+        try:
+            while any(t.is_alive() for t in threads):
+                await asyncio.sleep(0.2)
+        except asyncio.CancelledError:
+            # Allow graceful shutdown when the run coroutine is cancelled.
+            pass

ltq-0.4.0/src/ltq/broker.py

@@ -0,0 +1,140 @@
+from __future__ import annotations
+
+import asyncio
+import time
+from urllib.parse import urlparse
+import uuid
+from collections import defaultdict
+
+import redis.asyncio as aioredis
+
+from .message import Message
+
+
+class Broker:
+    @staticmethod
+    def from_url(url: str) -> Broker:
+        urlp = urlparse(url)
+        if urlp.scheme == "memory":
+            return MemoryBroker()
+        elif urlp.scheme == "redis":
+            return RedisBroker(url)
+        else:
+            raise RuntimeError(f"Unknown scheme: {urlp.scheme}")
+
+    async def close(self) -> None: ...
+    async def publish(self, message: Message, delay: float = 0) -> None: ...
+    async def consume(self, queue: str) -> Message: ...
+    async def ack(self, message: Message) -> None: ...
+    async def nack(
+        self,
+        message: Message,
+        delay: float = 0,
+        drop: bool = False,
+    ) -> None: ...
+    async def len(self, queue: str) -> int: ...
+    async def clear(self, queue: str) -> None: ...
+
+
+class RedisBroker(Broker):
+    def __init__(self, url: str) -> None:
+        self.url = url
+        self._client = aioredis.from_url(url)
+        self._id = uuid.uuid4().hex[:8]
+        self._consume = self._client.register_script("""
+            local ready = redis.call('zrangebyscore', KEYS[1], 0, ARGV[1], 'LIMIT', 0, 1)
+            if #ready == 0 then return nil end
+            local msg = ready[1]
+            redis.call('zadd', KEYS[2], ARGV[1], msg)
+            redis.call('zrem', KEYS[1], msg)
+            return msg
+        """)
+
+    async def close(self) -> None:
+        await self._client.aclose()
+
+    async def publish(
+        self,
+        message: Message,
+        delay: float = 0,
+    ) -> None:
+        score = time.time() + delay
+        await self._client.zadd(
+            f"queue:{message.task_name}",
+            {
+                message.to_json(): score,
+            },
+        )  # type: ignore
+
+    async def consume(self, queue: str) -> Message:
+        while True:
+            msg = await self._consume(
+                keys=[f"queue:{queue}", f"processing:{queue}:{self._id}"],
+                args=[time.time()],
+            )
+            if msg:
+                return Message.from_json(msg)
+            await asyncio.sleep(0.1)
+
+    async def ack(self, message: Message) -> None:
+        key = f"processing:{message.task_name}:{self._id}"
+        await self._client.zrem(key, message.to_json())  # type: ignore
+
+    async def nack(
+        self,
+        message: Message,
+        delay: float = 0,
+        drop: bool = False,
+    ) -> None:
+        key = f"processing:{message.task_name}:{self._id}"
+        await self._client.zrem(key, message.to_json())  # type: ignore
+        if not drop:
+            await self.publish(message, delay=delay)
+
+    async def len(self, queue: str) -> int:
+        return await self._client.zcard(f"queue:{queue}") or 0  # type: ignore
+
+    async def clear(self, queue: str) -> None:
+        await self._client.delete(f"queue:{queue}", f"processing:{queue}:{self._id}")  # type: ignore
+
+
+class MemoryBroker(Broker):
+    def __init__(self) -> None:
+        self._queues: defaultdict[str, dict[str, float]] = defaultdict(dict)
+
+    async def close(self) -> None:
+        pass
+
+    async def publish(
+        self,
+        message: Message,
+        delay: float = 0,
+    ) -> None:
+        self._queues[message.task_name][message.to_json()] = time.time() + delay
+
+    async def consume(self, queue: str) -> Message:
+        while True:
+            now = time.time()
+            for msg, score in list(self._queues[queue].items()):
+                if score <= now:
+                    del self._queues[queue][msg]
+                    return Message.from_json(msg)
+            await asyncio.sleep(0.1)
+
+    async def ack(self, message: Message) -> None:
+        pass
+
+    async def nack(
+        self,
+        message: Message,
+        delay: float = 0,
+        drop: bool = False,
+    ) -> None:
+        if not drop:
+            await self.publish(message, delay=delay)
+
+    async def len(self, queue: str) -> int:
+        return len(self._queues[queue])
+
+    async def clear(self, queue: str) -> None:
+        self._queues.pop(queue, None)