localqueue 0.1.1__tar.gz → 0.3.0__tar.gz

{localqueue-0.1.1 → localqueue-0.3.0}/CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## Unreleased
+
+## 0.3.0
+
+- Add a concurrent SQLite stress test for multiple producers and consumers.
+- Add `examples/sqlite_concurrency_benchmark.py` for measuring local queue throughput.
+- Make the SQLite benchmark reset its store and retry transient lock contention.
+- Add `examples/sqlite_process_harness.py` for process-level throughput and crash-recovery checks.
+- Add process-based stress tests for SQLite producer and consumer coordination.
+- Add dead-letter filters and summaries for `queue dead`.
+- Add enqueue deduplication with `--dedupe-key` and `dedupe_key=`.
+- Add queue-level retry defaults that workers can inherit from `PersistentQueue`.
+- Add worker rate limiting and circuit-breaker controls.
+- Add usage docs for rate limiting, circuit breaker, and queue positioning.
+- Add a short-term maturity note focused on performance and guarantees.
+
+## 0.2.0
+
+- Add `queue stats --watch` for monitoring queue counts while workers run.
+- Add `queue dead --watch` for repeatedly listing dead letters.
+- Add `queue requeue-dead --all` for bulk recovery after a fix.
+- Add structured `command not found` handling for `queue exec`.
+- Add `examples/process_webhook.sh` as a shell/curl worker example.
+
 ## 0.1.1
 
 - Add project URLs for PyPI metadata.

localqueue-0.3.0/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: localqueue
+Version: 0.3.0
+Summary: Durable local queues for Python, with persistent retry state powered by Tenacity.
+Project-URL: Homepage, https://brunoportis.github.io/localqueue/
+Project-URL: Documentation, https://brunoportis.github.io/localqueue/
+Project-URL: Repository, https://github.com/brunoportis/localqueue
+Project-URL: Issues, https://github.com/brunoportis/localqueue/issues
+Author: Bruno Portis
+License-Expression: MIT
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.11
+Requires-Dist: tenacity>=9.1.4
+Provides-Extra: all
+Requires-Dist: lmdb>=2.2.0; extra == 'all'
+Requires-Dist: pyyaml>=6.0.3; extra == 'all'
+Requires-Dist: rich>=15.0.0; extra == 'all'
+Requires-Dist: typer>=0.16.0; extra == 'all'
+Provides-Extra: cli
+Requires-Dist: pyyaml>=6.0.3; extra == 'cli'
+Requires-Dist: rich>=15.0.0; extra == 'cli'
+Requires-Dist: typer>=0.16.0; extra == 'cli'
+Provides-Extra: lmdb
+Requires-Dist: lmdb>=2.2.0; extra == 'lmdb'
+Provides-Extra: sqlite
+Description-Content-Type: text/markdown
+
+# `localqueue`
+
+[![Tests](https://github.com/brunoportis/localqueue/actions/workflows/tests.yml/badge.svg)](https://github.com/brunoportis/localqueue/actions/workflows/tests.yml)
+[![CodeQL](https://github.com/brunoportis/localqueue/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/brunoportis/localqueue/actions/workflows/github-code-scanning/codeql)
+![Coverage](https://img.shields.io/badge/coverage-%E2%89%A595%25-brightgreen)
+
+`localqueue` is a small durable queue for one machine. It stores work on the local filesystem by default and keeps retry state with Tenacity.
+
+Use it for scripts, CLI tools, cron jobs, and small Python workers that share one local store. Use `localqueue.retry` when another system already delivers work and you only need retry state that survives restarts.
+
+```bash
+localqueue queue exec emails -- python scripts/send_email.py
+```
+
+```python
+from localqueue import PersistentQueue, persistent_worker
+
+queue = PersistentQueue("emails")
+queue.put({"to": "user@example.com"})
+
+@persistent_worker(queue)
+def send_email(job: dict[str, str]) -> None:
+    deliver(job["to"])
+```
+
+Full docs live at [brunoportis.github.io/localqueue](https://brunoportis.github.io/localqueue/). The source docs are in [`docs/`](docs/).
+
+## Install
+
+```bash
+pip install localqueue
+```
+
+For the CLI:
+
+```bash
+pip install "localqueue[cli]"
+```
+
+For the optional LMDB backend:
+
+```bash
+pip install "localqueue[lmdb]"
+```
+
+## When not to use
+
+`localqueue` is not a distributed broker. If you need multi-host coordination, high write concurrency, managed retention, or strict cross-service ordering, use a system designed for that operating model.

localqueue-0.3.0/README.md

@@ -0,0 +1,48 @@
+# `localqueue`
+
+[![Tests](https://github.com/brunoportis/localqueue/actions/workflows/tests.yml/badge.svg)](https://github.com/brunoportis/localqueue/actions/workflows/tests.yml)
+[![CodeQL](https://github.com/brunoportis/localqueue/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/brunoportis/localqueue/actions/workflows/github-code-scanning/codeql)
+![Coverage](https://img.shields.io/badge/coverage-%E2%89%A595%25-brightgreen)
+
+`localqueue` is a small durable queue for one machine. It stores work on the local filesystem by default and keeps retry state with Tenacity.
+
+Use it for scripts, CLI tools, cron jobs, and small Python workers that share one local store. Use `localqueue.retry` when another system already delivers work and you only need retry state that survives restarts.
+
+```bash
+localqueue queue exec emails -- python scripts/send_email.py
+```
+
+```python
+from localqueue import PersistentQueue, persistent_worker
+
+queue = PersistentQueue("emails")
+queue.put({"to": "user@example.com"})
+
+@persistent_worker(queue)
+def send_email(job: dict[str, str]) -> None:
+    deliver(job["to"])
+```
+
+Full docs live at [brunoportis.github.io/localqueue](https://brunoportis.github.io/localqueue/). The source docs are in [`docs/`](docs/).
+
+## Install
+
+```bash
+pip install localqueue
+```
+
+For the CLI:
+
+```bash
+pip install "localqueue[cli]"
+```
+
+For the optional LMDB backend:
+
+```bash
+pip install "localqueue[lmdb]"
+```
+
+## When not to use
+
+`localqueue` is not a distributed broker. If you need multi-host coordination, high write concurrency, managed retention, or strict cross-service ordering, use a system designed for that operating model.

{localqueue-0.1.1 → localqueue-0.3.0}/docs/api.md

@@ -24,6 +24,7 @@ Constructor options:
 | `store_path` | path for the default SQLite queue store |
 | `lease_timeout` | seconds before an inflight message is redelivered |
 | `maxsize` | maximum number of ready messages; `0` means unbounded |
+| `retry_defaults` | Tenacity retry keyword defaults inherited by workers |
 
 Core methods:
 
@@ -44,6 +45,8 @@ Core methods:
 | `stats()` | count ready, delayed, inflight, dead, and total messages |
 | `dead_letters(limit=None)` | list dead-letter messages |
 | `requeue_dead(message, delay=0.0)` | return a dead-letter message to ready delivery |
+| `prune_dead_letters(older_than)` | remove dead letters older than the given age |
+| `count_dead_letters_older_than(older_than)` | count dead letters older than the given age without removing them |
 | `empty()` | whether there are no ready messages |
 | `full()` | whether ready capacity is reached |
 | `purge()` | remove all queue records |
@@ -74,8 +77,21 @@ Constructor options:
 | `dead_letter_on_failure` | dead-letter final handler failures when `True` |
 | `dead_letter_on_exhaustion` | compatibility alias for `dead_letter_on_failure` |
 | `release_delay` | delay used when releasing failed messages |
+| `min_interval` | minimum seconds to wait between worker message starts |
+| `circuit_breaker_failures` | consecutive recoverable failures before pausing the worker |
+| `circuit_breaker_cooldown` | pause duration after the breaker opens |
 | `**retry_kwargs` | forwarded to `PersistentRetrying` |
 
+Queue-level retry defaults can also be attached to `PersistentQueue` and are
+merged into worker retry kwargs before explicit worker overrides. That keeps
+shared queue policies close to the queue definition while still letting a
+worker override a specific retry parameter when needed.
+
+`min_interval` is a per-worker rate limit. Set it when a handler talks to an
+external service that should not be hit back-to-back. The circuit-breaker pair
+(`circuit_breaker_failures` + `circuit_breaker_cooldown`) opens after repeated
+recoverable failures and pauses the worker before it fetches the next message.
+
 #### `QueueMessage`
 
 Dataclass returned by `put()` and `get_message()`.
@@ -91,12 +107,17 @@ Dataclass returned by `put()` and `get_message()`.
 | `available_at` | earliest delivery timestamp |
 | `leased_until` | lease expiration timestamp, if inflight |
 | `leased_by` | optional worker id that currently owns the lease, if inflight |
+| `dedupe_key` | optional idempotency key used to reuse the same stored message |
+| `attempt_history` | list of lease and outcome events recorded for this message |
 | `last_error` | structured error from the most recent failed processing attempt, if recorded |
 | `failed_at` | timestamp for `last_error`, if recorded |
 
 For `localqueue queue exec` failures, `last_error` also includes `command`,
 `exit_code`, `stdout`, and `stderr` fields so command workers can be inspected
-from `queue inspect` and `queue dead`.
+from `queue inspect` and `queue dead`. `dedupe_key` is returned on inspection
+and lets repeated enqueues reuse the same message until it is acknowledged or
+cleaned up. `attempt_history` shows the lease and terminal events that led to
+the current state.
 
 #### `QueueStats`
 
@@ -109,6 +130,12 @@ Dataclass returned by `stats()`.
 | `inflight` | leased messages not yet acknowledged, released, or dead-lettered |
 | `dead` | dead-letter messages hidden from normal delivery |
 | `total` | all messages still stored for the queue |
+| `by_worker_id` | current inflight counts grouped by `leased_by` |
+| `leases_by_worker_id` | historical lease counts grouped by `leased_by`, a coarse throughput proxy |
+| `last_seen_by_worker_id` | most recent heartbeat timestamp for each recorded worker id |
+| `oldest_ready_age_seconds` | age of the oldest ready message currently waiting |
+| `oldest_inflight_age_seconds` | age of the oldest current inflight lease |
+| `average_inflight_age_seconds` | average age across current inflight leases |
 
 ### Worker decorators
 
@@ -155,6 +182,10 @@ Protocol for custom queue stores.
 SQLite-backed queue store. This is the default backend. Records are serialized
 as versioned JSON; values must be JSON-serializable.
 
+The SQLite store tracks its on-disk schema version with `PRAGMA user_version`.
+Current releases accept older compatible versions and reject future versions
+until a migration is defined.
+
 #### `LMDBQueueStore`
 
 LMDB-backed queue store. Records are serialized as versioned JSON; values must be
@@ -168,6 +199,9 @@ Thread-safe in-memory queue store for tests.
 
 Raised when LMDB reports that the queue store is locked by another process.
 
+Install `localqueue[cli]` when you want the CLI entry points, and
+`localqueue[lmdb]` when you want the LMDB queue store backend.
+
 ## localqueue.retry
 
 ### Retry decorators
@@ -316,6 +350,7 @@ class AttemptStore:
     def load(self, key: str) -> RetryRecord | None: ...
     def save(self, key: str, record: RetryRecord) -> None: ...
     def delete(self, key: str) -> None: ...
+    def prune_exhausted(self, *, older_than: float, now: float) -> int: ...
 ```
 
 #### `LMDBAttemptStore`

localqueue-0.3.0/docs/compare.md

@@ -0,0 +1,50 @@
+---
+icon: lucide/inbox
+---
+
+# Compare
+
+`localqueue` is for local, durable queues on one machine. If your jobs need a broker, a cluster, or managed retention, pick a tool that is built for that shape.
+
+## Use `localqueue` when
+
+- you want a local filesystem-backed queue
+- jobs are handled by scripts, CLIs, or small Python workers
+- you care more about operational simplicity than distributed scheduling
+- at-least-once delivery is acceptable
+- you want to inspect and requeue failed work from the terminal
+- you want worker-side controls like `min_interval`, queue-level retry defaults,
+  and a simple circuit breaker without introducing a broker
+
+## Reach for something else when
+
+- producers and consumers must run on different machines
+- you need broker-managed delivery guarantees
+- write contention is high
+- retention, metrics, sharding, or fanout are first-class requirements
+- you want a queue that is already part of a larger distributed platform
+- you need broker-level rate limiting or cross-host worker coordination
+
+## Quick comparison
+
+| System | Best fit | Why choose it instead |
+| --- | --- | --- |
+| `localqueue` | single-host workers, scripts, local ops | simplest path when the queue can live on one filesystem |
+| Celery | Python task orchestration with a broker | richer distributed worker ecosystem |
+| RQ | Redis-backed Python jobs | simpler Redis-based queueing for server-side apps |
+| Dramatiq | Python workers with broker support | clean task API for distributed workers |
+| SQS | managed cloud queueing | external durability and multi-host coordination |
+| RabbitMQ | brokered messaging | mature routing and delivery primitives |
+| Redis Streams | stream processing and consumer groups | shared stream coordination with Redis |
+| Kafka | high-throughput event pipelines | log-based distributed ingestion and replay |
+
+## Rule of thumb
+
+If the queue can stay local and the value is in simple recovery, `localqueue` fits.
+If the queue itself is part of a distributed system boundary, use a broker or stream platform.
+
+See also:
+
+- [Operational maturity](operational-maturity.md)
+- [Persistent queues](queues.md)
+- [Persistent retries](retries.md)

localqueue-0.3.0/docs/index.md

@@ -0,0 +1,74 @@
+---
+icon: lucide/inbox
+---
+
+# Overview
+
+`localqueue` is a small durable queue for one machine. It stores work on the local filesystem by default and keeps retry state with Tenacity.
+
+It fits scripts, CLI tools, cron jobs, development helpers, and small workers that can safely share one local store. It is not a distributed broker and does not provide multi-host coordination.
+
+!!! note
+    The default model is local-file storage, at-least-once delivery, and best-effort ordering under concurrency.
+
+## CLI
+
+Use the CLI when you want to enqueue, inspect, watch, and recover jobs from the terminal.
+
+```bash
+localqueue queue add emails --value '{"to":"user@example.com"}'
+localqueue queue exec emails -- python scripts/send_email.py
+localqueue queue stats emails --watch --interval 1
+localqueue queue dead emails --summary
+localqueue queue health emails
+```
+
+`queue exec` runs an external command per message. The message value is written to stdin as JSON, and exit code `0` acks the message.
+
+For more command examples, see [Persistent queues](queues.md).
+
+## Library
+
+Use the Python API when the worker lives in your codebase.
+
+```python
+from localqueue import PersistentQueue, persistent_worker
+
+queue = PersistentQueue("emails")
+queue.put({"to": "user@example.com"})
+
+@persistent_worker(queue)
+def send_email(job: dict[str, str]) -> None:
+    deliver(job["to"])
+```
+
+`localqueue.retry` stays available when another system already delivers work and you only need retry state.
+
+See [Persistent retries](retries.md) for the retry API and store options.
+
+## When to use
+
+- small Python workers on one machine
+- scripts that need durable work after restarts
+- terminal-driven queues for jobs you want to inspect and requeue
+- retry state that must survive process restarts
+
+## When not to use
+
+- multi-host scheduling
+- high write concurrency
+- managed retention or broker-level metrics
+- strict global ordering
+- distributed coordination
+
+See [Compare](compare.md) for a short decision guide.
+
+## Read more
+
+- [Persistent queues](queues.md)
+- [Persistent retries](retries.md)
+- [Stability](stability.md)
+- [Compare](compare.md)
+- [API reference](api.md)
+- [Operational maturity](operational-maturity.md)
+- [Release checklist](release.md)

localqueue-0.3.0/docs/operational-maturity.md

@@ -0,0 +1,137 @@
+---
+icon: lucide/list-checks
+---
+
+# Operational Maturity Checklist
+
+This project is intentionally small: durable local queues for Python workers,
+plus persistent retry state powered by Tenacity. The checklist below tracks the
+work needed before describing it as a mature production queue system instead of
+a local worker library.
+
+## Near-term focus
+
+The next release should improve two things that matter for real local workers:
+
+- performance, especially under multiple producers and consumers sharing one
+  SQLite file
+- guarantees, especially where the project is intentionally at-least-once and
+  best-effort rather than distributed or strongly ordered
+
+That means measuring the current ceiling before promising more:
+
+- benchmark WAL contention and queue throughput with concurrent processes
+- use `examples/sqlite_concurrency_benchmark.py` to get a repeatable local
+  throughput snapshot
+- a quick local run on the development machine stayed in the low thousands of
+  messages per second and handled a few producers and consumers without failing
+  the queue, which is useful as a local ceiling check but not a production
+  guarantee
+- exercise abrupt consumer exit and lease recovery with
+  `examples/sqlite_process_harness.py --mode crash-recovery`
+- document the practical limits for a single host and a single SQLite file
+- keep ordering language explicit: best effort under concurrency, not strict
+  global ordering
+- keep delivery language explicit: at least once, not exactly once
+- keep recovery language explicit: inspection and requeue are local operations,
+  not cross-cluster coordination
+
+## Concurrency and locking
+
+- [x] Document practical SQLite concurrency limits for producers and workers.
+- [x] Add stress tests for multiple producer and consumer processes sharing one
+  SQLite queue file.
+- [x] Measure throughput and lock contention under WAL mode.
+- [ ] Define guidance for when users should move to Postgres, Redis, SQS,
+  RabbitMQ, or another external broker.
+
+## Distributed operation
+
+- [x] Decide whether multi-host operation is in scope.
+- [x] Add explicit documentation that the default model is local-file storage,
+  not distributed coordination.
+- [x] Evaluate worker heartbeats beyond lease expiration.
+- [x] Evaluate worker identity, liveness inspection, and stale-worker reporting.
+- [x] Evaluate sharding or namespacing guidance for independent queues.
+
+`localqueue` does not try to coordinate work across multiple machines. The
+supported model is a single host, one local store, and local recovery through
+inspection, retry, and requeue. Heartbeats and stale-worker reporting are
+inspection tools for that local model, not a distributed liveness protocol.
+
+## Delivery semantics
+
+- [x] Document at-least-once delivery prominently in the README and queue docs.
+- [x] Add idempotency guidance for handlers.
+- [x] Provide examples for idempotent job keys and external side effects.
+- [x] Evaluate deduplication support for enqueue operations.
+- [x] Evaluate stronger ordering guarantees, or document why ordering is best
+  effort under concurrency.
+- [x] Document the failure window between a successful handler side effect and
+  `ack()`.
+
+## Retry and worker policies
+
+- [x] Add examples for different policies by exception type.
+- [x] Evaluate built-in permanent-failure classification.
+- [x] Evaluate queue-level retry defaults that can be reused by many workers.
+- [x] Evaluate rate limiting for handlers that call external services.
+- [x] Evaluate circuit-breaker integration or documentation.
+- [x] Add clearer guidance for `release_delay`, Tenacity `wait`, and
+  `lease_timeout` interactions.
+- [x] Reject ambiguous worker-loop combinations such as `--forever` with
+  `--max-jobs`.
+
+## Observability
+
+- [ ] Add optional Prometheus or OpenTelemetry metrics.
+- [x] Track per-message attempt history, not only the latest error.
+- [x] Track queue latency, handler duration, and time spent inflight.
+- [x] Track throughput by queue and worker id.
+- [x] Add aggregate failure summaries for dead-letter inspection.
+- [x] Evaluate structured event logging for enqueue, lease, ack, release, and
+  dead-letter transitions.
+
+## Storage operations
+
+- [x] Define a schema migration strategy for SQLite stores.
+- [x] Document backup and restore expectations.
+- [x] Add maintenance guidance for `VACUUM`, WAL files, and long-lived stores.
+- [x] Add retention or cleanup controls for dead-letter records.
+- [x] Add cleanup controls for old retry records.
+- [x] Add preview mode before cleanup removes records.
+- [x] Add configurable retention defaults for dead letters and retry records.
+- [x] Add a one-shot queue health summary for day-to-day checks.
+- [x] Document recovery behavior for corrupted or partially written store files.
+- [x] Add compatibility tests for SQLite store files created by older package
+  versions.
+
+The SQLite store keeps a schema version in `PRAGMA user_version`. Current
+releases accept older compatible versions and reject future versions they do
+not know how to migrate yet. That keeps upgrades explicit and leaves a clear
+path for future schema changes.
+
+## API stability
+
+- [x] Decide which APIs are stable before a `1.0` release.
+- [x] Document compatibility aliases and their planned lifetime.
+- [x] Add deprecation policy.
+- [ ] Increase coverage margin above the current 95% gate.
+- [x] Keep docs aligned with defaults for SQLite, optional LMDB, and CLI extras.
+
+## Guardrails against misuse
+
+- [x] Document recommended payload size limits.
+- [x] Add clearer lease-timeout guidance for slow jobs.
+- [x] Add examples for safe shutdown and long-running workers.
+- [x] Add health-check examples for worker processes.
+- [x] Add configuration validation for risky combinations where possible.
+- [x] Add warnings or docs for unbounded queue growth.
+
+## Positioning
+
+- [x] Keep public wording focused on durable local queues for scripts and small
+  workers.
+- [x] Avoid presenting the project as a replacement for distributed brokers.
+- [x] Add production-readiness notes that distinguish supported use cases from
+  future work.