kickdown 0.4.0__tar.gz

kickdown-0.4.0/.claude/settings.local.json

@@ -0,0 +1,7 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(uv run *)"
+    ]
+  }
+}

kickdown-0.4.0/.github/CODEOWNERS

@@ -0,0 +1 @@
+* @vklokov

kickdown-0.4.0/.github/workflows/ci.yml

@@ -0,0 +1,40 @@
+name: PR Verify
+on:
+  pull_request:
+    types:
+      - opened
+      - reopened
+      - synchronize
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - name: Install uv
+        uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
+        with:
+          enable-cache: true
+
+      - name: Run Ruff
+        uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 #v3.6.1
+
+      - name: Check formatting
+        uses: astral-sh/ruff-action@4919ec5cf1f49eff0871dbcea0da843445b837e6 #v3.6.1
+        with:
+          args: "format --check"
+
+      - name: Run ty
+        run: uv run ty check
+
+
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - name: Install uv
+        uses: astral-sh/setup-uv@cec208311dfd045dd5311c1add060b2062131d57 # v8.0.0
+        with:
+          enable-cache: true
+      - name: Run Pytest
+        run: uv run pytest -q

kickdown-0.4.0/.gitignore

@@ -0,0 +1,19 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+
+tmp/
+
+_client.py
+_server.py
+
+.mypy_cache
+.pytest_cache
+.ruff_cache

kickdown-0.4.0/.python-version

@@ -0,0 +1 @@
+3.13

kickdown-0.4.0/.vscode/settings.json

@@ -0,0 +1,11 @@
+{
+  "editor.tabSize": 2,
+  "editor.insertSpaces": true,
+  "editor.detectIndentation": true,
+  "editor.renderWhitespace": "all",
+  "editor.formatOnSave": true,
+  "python.defaultInterpreterPath": "./.venv/bin/python",
+  "[python]": {
+    "editor.defaultFormatter": "charliermarsh.ruff",
+  },
+}

kickdown-0.4.0/CHANGELOG.md

@@ -0,0 +1,152 @@
+# Changelog
+
+## 0.4.0 - 2026-09-17
+
+### Added
+- Tasks are no longer lost when a worker process dies mid-task: claiming a task atomically moves it from its queue into an in-flight list private to that consumer (`kickdown:inflight:{consumer_id}`), and it is only removed once the task finishes
+- Every server keeps a heartbeat key (`kickdown:beat:{consumer_id}`) alive and registers itself in `kickdown:consumers`; a `Reaper` loop returns the in-flight tasks of any consumer whose heartbeat expired, guarded by a lock so servers do not race
+- A clean shutdown returns the server's own unfinished tasks to their queues instead of leaving them for the reaper
+- The admin dashboard's `In-flight` column now shows real per-queue counts
+
+### Changed
+- Delivery is now at-least-once, so workers must be idempotent: a task interrupted by a crash runs again
+- Queues are polled by a Lua claim script instead of `BLPOP`; polling is now interval-based (100ms) because no Redis primitive can block on a move across several keys at once
+- Workers can be registered as classes with `perform` as a `classmethod`, not just as instances — `add_workers` accepts both, typed as the new `Worker` alias (`type[Performable] | Performable`); passing a class did not type-check before
+- `Task` field defaults are declared so that type checkers see them — `Task(queue=..., operation=..., params=...)` now type-checks for users, which it did not before despite the package shipping `py.typed`
+
+## 0.4.0a1 - 2026-09-17
+
+### Changed
+- Rewritten around an arbitrary number of queues instead of a single configured queue
+  - Workers (`Performable`) now declare their own `queue` and `operation`; the server derives the set of polled queues from the registered workers automatically
+  - Queues are polled in round-robin order with equal frequency — no priority between queues at this stage
+  - Workers are keyed by `(queue, operation)`, so the same `operation` name can be reused safely across different queues
+- `Client.enqueue` now takes a single `Task` (carrying `queue`, `operation`, `params`, `jid`, `retry_count`) instead of a worker class and payload dict
+- `Client.pending(queue)` / `Client.stats(queue)` moved onto the queue handle: `client.queue(name).pending()` / `.stats()`
+- `Client` is now fully async (`enqueue`, `pending`, `close`), and supports `async with` instead of a sync context manager
+- `Server` no longer takes a `Config` object; it's constructed directly with `redis_url` and `concurrency`
+- Retries use a hardcoded exponential backoff (`1s * 1.5 ** attempt`) instead of a per-task `backoff_coefficient` — a failed task is retried with `retry_count` decremented and `attempt` incremented until `retry_count` reaches `0`
+- Retries are no longer awaited in memory: the failed task is stored in a scheduled sorted set in Redis and picked up when due, so it survives a process restart and does not hold a concurrency slot for the duration of the delay
+- On shutdown, the server now waits for in-flight tasks (including ones mid-retry) to finish before closing the Redis connection
+
+### Added
+- A default logger (plain `logging.Logger`, text output to stdout) on `Client` and `Server` — no custom logger interface/protocol; swap it by assigning `server.logger = ...` / `client.logger = ...`
+  - `Client` logs when a task is accepted (enqueued)
+  - `Server` logs process start/shutdown (with the polled queues and concurrency), each task's start/completion, retries, permanent failures, and unrecognized operations — all messages include the task's `jid`
+- `Server` always starts a small HTTP server (`web_port`, default `3030`) with `/live` and `/ready` for liveness/readiness probes; `/ready` checks Redis connectivity
+- Per-queue `processed`/`failed` counters (`Stats` model) via `client.queue(name).stats()`; the consumer increments them on task completion, permanent failure (retries exhausted), and unrecognized operations
+- `GET /admin` renders an HTML dashboard: total processed/failed/scheduled counters, and a per-queue table of pending/scheduled/in-flight counts (in-flight is a placeholder until tasks are tracked while they run); optionally gated behind HTTP Basic Auth via `Server(admin_username=..., admin_password=...)` (no auth if either is left unset)
+- `Queue.purge()` drops every pending task in a queue and returns how many were dropped (scheduled tasks and counters are untouched); not exposed over HTTP
+- `Server.enqueue(task)` pushes a task using the server's own Redis connection, so workers/hooks can schedule tasks without a separate `Client`
+- A `Scheduler` loop runs inside every `Server`, moving due tasks from each queue's `kickdown:scheduled:{name}` set back into the queue (atomically, via a Lua script); a server only sweeps the queues it has workers for
+- `Queue`, a handle bound to one queue name, carrying every per-queue operation (`push`, `pending`, `length`, `scheduled`, `stats`, `purge`, counters); obtained via `Client.queue(name)` / `Server.queue(name)`, and `Server.queues` now returns `Queue` objects instead of names
+
+### Removed
+- The old `Loggable` protocol and the custom `Logger` wrapper class (replaced by the stdlib logger above)
+- `Server.uptime()` / `Status` and the `humanize` dependency
+
+---
+
+## 0.3.3 - 2026-06-19
+
+### Changed
+- Pinned `redis` to the 7.x line (`~=7.4`) to avoid an unintended upgrade to `redis` 8.x; previously `>=7.4.0` allowed any newer major version
+
+---
+
+## 0.3.2 - 2026-06-03
+
+### Added
+- Job execution duration is now tracked and logged (in seconds) on job completion
+
+---
+
+## 0.3.1 - 2026-05-26
+
+### Added
+- Lifecycle hooks: `Server.on_startup` and `Server.on_shutdown` register async callbacks for server startup and shutdown
+  - Usable as decorators or called directly with a callable argument
+  - Startup hooks run after the Redis connection is verified, before the consumer starts
+  - Shutdown hooks run after the consumer is stopped, before the Redis connection is closed
+  - Exceptions in hooks are logged and do not crash the server
+
+---
+
+## 0.3.0 - 2026-05-21
+
+### Added
+- Retry mechanism for failed jobs
+  - `Client.enqueue` accepts `retry_count` (default: `1`) and `backoff_coefficient` (default: `1.5`)
+  - Both values are stored on the `Job` model and travel with the job through the queue
+  - On failure, the consumer re-enqueues the job with an exponential backoff delay (`backoff_coefficient ** attempt` seconds) until retries are exhausted
+  - Once all retries are exhausted the job is marked failed as before
+  - Jobs log a warning on each retry attempt and an error only when permanently failed
+
+---
+
+## 0.2.8 - 2026-05-20
+
+### Changed
+- `Server.add_worker` now accepts a worker **instance** (`Performable`) — the same instance is reused across all jobs of that type
+- `Client.enqueue` accepts a worker **class** (`type[Performable]`) — no instantiation cost at enqueue time, only the class name is needed to build the job
+- Healthcheck rewritten with FastAPI + uvicorn, replacing the manual raw TCP/HTTP server
+  - `GET /live` — always returns 200, used as a liveness probe
+  - `GET /ready` — checks consumer heartbeat and Redis connectivity (`store.ping`); returns 503 with reason on failure
+- Job IDs switched from `uuid4` (stdlib) to `uuid7` (`uuid_extensions`), providing time-sortable identifiers
+
+### Fixed
+- `config.queue` was logged as a bound method object; now called correctly as `config.queue()`
+
+### Dependencies
+- Added `fastapi>=0.136.1`
+- Added `uvicorn>=0.34.0`
+
+---
+
+## 0.2.6 - 2026-05-20
+
+### Changed
+- All Redis operations extracted into a new internal `Store` class (`kickdown/store.py`)
+- Redis key names (`kickdown:queue:*`, `kickdown:stats:*`) are now centralized in `Store`
+- `Consumer` and `Client` no longer interact with Redis directly — all calls go through `Store`
+- `asyncio.to_thread` wrapping for blocking Redis calls moved from `Consumer` into `Store`
+
+---
+
+## 0.2.5 - 2026-05-20
+
+### Changed
+- Workers are now registered on `Server` via `add_worker()` instead of being passed to `Config`
+- `Server` lazily initializes the consumer on `start()`, decoupling construction from Redis connection
+- `Consumer` receives workers directly rather than reading them from `Config`
+
+### Added
+- `Server.uptime()` returns a human-readable duration (e.g. `"1 day, 4 hours"`) via the `humanize` library
+- `Status` response includes an `uptime` field
+- `Server.start()` raises `RuntimeError` if no workers have been registered
+
+### Fixed
+- `Consumer.is_ok` converted from a method to a property, fixing incorrect truthy evaluation in `status()`
+
+---
+
+## 0.2.4 — 2026-05-19
+
+### Changed
+- Queue names are now raw identifiers (e.g. `"default"`, `"emails"`); the full Redis key is constructed internally as `kickdown:queue:{name}`
+- Stats counters moved from a Redis hash to separate keys (`kickdown:stats:processed`, `kickdown:stats:failed`), incremented with `INCR`
+
+---
+
+## 0.2.3 — 2026-05-19
+
+### Added
+- `Client.stats()` returns a `Stats` model with cumulative `processed` and `failed` job counts
+- Consumer increments `kickdown.stats.processed` / `kickdown.stats.failed` in Redis after each job completes or raises
+
+---
+
+## 0.2.2 — 2026-05-19
+
+### Added
+- `Client.pending()` returns a list of jobs currently waiting in the queue (non-destructive)

kickdown-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Vladimir Klokov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

kickdown-0.4.0/PKG-INFO

@@ -0,0 +1,314 @@
+Metadata-Version: 2.5
+Name: kickdown
+Version: 0.4.0
+Summary: A Redis-backed background job queue for Python
+Project-URL: Homepage, https://github.com/vklokov/kickdown
+Project-URL: Repository, https://github.com/vklokov/kickdown
+Project-URL: Issues, https://github.com/vklokov/kickdown/issues
+Project-URL: Changelog, https://github.com/vklokov/kickdown/blob/main/CHANGELOG.md
+Author-email: Vladimir Klokov <klokov.dev@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: asyncio,background,jobs,queue,redis,tasks,worker
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Distributed Computing
+Classifier: Typing :: Typed
+Requires-Python: >=3.13
+Requires-Dist: fastapi>=0.136.1
+Requires-Dist: pydantic>=2.13.4
+Requires-Dist: redis~=7.4
+Requires-Dist: uuid7>=0.1.0
+Requires-Dist: uvicorn>=0.34.0
+Description-Content-Type: text/markdown
+
+# kickdown
+
+A Redis-backed background job queue for Python.
+
+Requires Python 3.13+ and **Redis 6.2 or newer** (the queue relies on `LMOVE`
+and on server-side Lua scripts introduced in that release).
+
+## Installation
+
+```sh
+uv add kickdown
+```
+
+Pre-releases are not picked up by default, so ask for one explicitly:
+
+```sh
+uv add kickdown --prerelease=allow
+```
+
+## Usage
+
+### Defining a worker
+
+A worker declares which queue it consumes from, which `operation` name
+identifies it, and an async `perform`:
+
+```python
+class SendEmailWorker:
+    queue = "emails"
+    operation = "send_email"
+
+    @classmethod
+    async def perform(cls, payload: dict) -> None:
+        recipient = payload["to"]
+        # ... send email
+```
+
+Workers are registered as classes, so nothing has to be instantiated to run a
+task. If a worker needs per-instance state, register an instance instead — with
+`perform` as a regular `async def perform(self, payload)`; both forms are
+accepted, and `Worker` is the type covering them.
+
+Any number of queues is supported — a worker's `queue` attribute is what
+determines which Redis list it consumes from. The server automatically polls
+every queue that has at least one registered worker.
+
+### Running the server
+
+```python
+import asyncio
+from kickdown import Server
+from your_workers import SendEmailWorker, ExportReportWorker
+
+server = Server(
+    redis_url="redis://localhost:6379",
+    concurrency=5,  # optional, default: 1 - max tasks processed concurrently
+)
+server.add_workers(SendEmailWorker, ExportReportWorker)
+
+asyncio.run(server.run())
+```
+
+`add_workers` accepts any number of workers, as classes or as instances.
+Workers are resolved by their `(queue, operation)` pair, so the same
+`operation` name can be reused safely across different queues.
+
+Queues are polled with equal frequency in round-robin order — there is
+currently no notion of priority between queues.
+
+### Enqueueing jobs
+
+```python
+from kickdown import Client, Task
+
+client = Client(redis_url="redis://localhost:6379")
+
+task = Task(
+    queue="emails",
+    operation="send_email",
+    params={"to": "user@example.com"},
+)
+jid = await client.enqueue(task)
+```
+
+`enqueue` returns the job ID (`jid`) that can be used for tracing. It is
+generated automatically (a time-sortable `uuid7`) if not set explicitly on
+the `Task`.
+
+`Client` can also be used as an async context manager, which closes the
+underlying Redis connection on exit:
+
+```python
+async with Client(redis_url="redis://localhost:6379") as client:
+    await client.enqueue(task)
+```
+
+`Server` can enqueue tasks too, using the same Redis connection — handy for
+a worker that needs to schedule a follow-up task, or for enqueueing from a
+startup hook, without opening a separate `Client`:
+
+```python
+jid = await server.enqueue(task)
+```
+
+#### Retries
+
+`retry_count` (default `1`) on `Task` sets how many times a failed task is
+retried before being dropped. On failure, the consumer re-enqueues the task
+with `retry_count` decremented by one and `attempt` incremented by one, after
+an exponentially growing delay. Once `retry_count` reaches `0` the task is
+dropped.
+
+The delay is `1s * 1.5 ** attempt` — both the base delay and the backoff
+coefficient are fixed in the library and cannot be configured per task:
+
+| Retry | Delay |
+| ----- | ----- |
+| 1st   | 1.0s  |
+| 2nd   | 1.5s  |
+| 3rd   | 2.3s  |
+
+A retried task is not held in memory while it waits: it goes into the queue's
+scheduled sorted set in Redis (`kickdown:scheduled:{name}`, scored by its due
+timestamp), and a scheduler loop running inside every server moves due tasks
+back into the queue. So a retry survives a process restart, and the
+concurrency slot is freed immediately instead of being blocked for the whole
+delay.
+
+A server only sweeps the queues it has workers for, which is also the only
+place its own retries can land.
+
+#### Crash recovery
+
+A task is never held only in the worker process's memory. Claiming one moves
+it, in a single Redis operation, from its queue into an in-flight list private
+to that consumer (`kickdown:inflight:{consumer_id}`), where it stays until the
+worker finishes. Each server keeps a heartbeat key alive while it runs, and a
+reaper loop inside every server watches for consumers whose heartbeat has
+expired: whatever is left in a dead consumer's list is pushed back into the
+queue it came from. On a clean shutdown a server returns its own unfinished
+tasks immediately instead of waiting to be reaped.
+
+That makes delivery **at-least-once**: a task interrupted by a crash runs
+again, and a task that crashed the process *after* its side effects completed
+runs those side effects twice. Workers must be idempotent.
+
+Two consequences worth knowing:
+
+- A task that reliably kills its process (an OOM, say) will be requeued and
+  kill it again. There is no poison-pill limit yet.
+- A reaped task keeps its `retry_count`: being interrupted is not counted as a
+  failed attempt.
+
+Queue names are raw identifiers (e.g. `"default"`, `"emails"`). The client
+constructs the full Redis key internally as `kickdown:queue:{name}`.
+
+### Inspecting a queue
+
+`client.queue(name)` returns a `Queue` handle — every per-queue operation
+lives on it, so the queue name is given once instead of on every call:
+
+```python
+emails = client.queue("emails")
+
+# Tasks waiting to be processed (non-destructive)
+tasks = await emails.pending()
+count = await emails.length()
+
+# Tasks waiting for their retry delay to elapse
+retries = await emails.scheduled()
+
+# Cumulative processed/failed counters
+stats = await emails.stats()
+print(stats.processed, stats.failed)
+```
+
+Enqueueing stays on the client (`client.enqueue(task)`): a `Task` already
+carries its own `queue`, and that field remains the single source of truth
+for routing.
+
+`purge()` drops every task waiting in the queue and returns how many were
+dropped. It does not touch scheduled tasks or the counters, and there is no
+undo:
+
+```python
+dropped = await emails.purge()
+```
+
+`processed` counts tasks whose worker completed successfully; `failed`
+counts tasks that were permanently dropped (retries exhausted, or no
+worker registered for the task's `operation`).
+
+### Lifecycle hooks
+
+Register async callbacks to run on server startup and shutdown — useful for
+initialising shared resources like database pools.
+
+```python
+server = Server(redis_url=...)
+
+
+@server.on_startup
+async def init_db():
+    app.db = await asyncpg.create_pool(DATABASE_URL)
+
+
+@server.on_shutdown
+async def close_db():
+    await app.db.close()
+```
+
+Both methods can also be called directly instead of used as decorators:
+
+```python
+server.on_startup(init_db)
+server.on_shutdown(close_db)
+```
+
+Startup hooks run after the Redis connection is verified, before the
+consumer starts. Shutdown hooks run after the consumer stops (whether by
+`SIGTERM`/`SIGINT` or an unexpected error), before the Redis connection is
+closed.
+
+### Logging
+
+`Client` and `Server` each expose a plain `logging.Logger` as `.logger`,
+pre-configured to write text to stdout — there's no custom logger
+interface to implement. To use your own logger (different format, handler,
+sink, etc.), just assign it:
+
+```python
+import logging
+
+server.logger = logging.getLogger("myapp.kickdown")
+client.logger = logging.getLogger("myapp.kickdown")
+```
+
+`Client` logs when a task is accepted. `Server` logs process start/shutdown
+(with the polled queues and concurrency), and each task's start,
+completion, retries and permanent failures — every task-related message
+includes the task's `jid`.
+
+### Healthcheck
+
+`Server` always starts a small HTTP server for liveness/readiness probes,
+on the port given by `web_port` (default `3030`):
+
+```python
+server = Server(redis_url=..., web_port=3030)
+```
+
+```
+GET /live   -> 200 {"status": "ok"}                         # process is up
+GET /ready  -> 200 {"status": "ok"}                          # Redis reachable
+            -> 503 {"status": "redis unavailable"}           # Redis unreachable
+```
+
+### Admin page
+
+`GET /admin` renders an HTML dashboard: a summary block with total
+processed/failed counters (aggregated across all queues, from the same
+counters as `client.stats()`), and a table of every polled queue with its
+current pending count (how many tasks are physically waiting in it).
+
+By default it's open to anyone who can reach the port. To require HTTP
+Basic Auth, set both `admin_username` and `admin_password`:
+
+```python
+server = Server(
+    redis_url=...,
+    admin_username="alice",
+    admin_password="secret",
+)
+```
+
+If either is left unset, `/admin` requires no credentials.
+
+## Scaling
+
+`Server.run()` uses a single asyncio event loop with an `asyncio.Semaphore`
+to cap concurrent task execution within the process — this is a good fit
+since `Performable.perform` is a coroutine. To scale across CPU cores or
+machines, run multiple `Server` processes against the same Redis instance;
+each process independently pops from the shared queues, so Redis balances
+the work between them without any extra coordination.