localqueue 0.3.2__tar.gz → 0.3.4__tar.gz

{localqueue-0.3.2 → localqueue-0.3.4}/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## 0.3.4
+
+- Keep `queue exec` and `queue process` alive in `--forever --block` mode when the queue starts empty.
+- Preserve the batch empty-queue exit behavior for non-forever workers.
+
+## 0.3.3
+
+- Split queue and retry store implementations into dedicated modules.
+- Refactor the CLI worker option wiring and centralize worker helpers.
+- Add a SonarCloud workflow and stabilize the SQLite concurrency queue test.
+
 ## 0.3.2
 
 - Add a CLI Docker image and publish it to GHCR on version tags.

{localqueue-0.3.2 → localqueue-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: localqueue
-Version: 0.3.2
+Version: 0.3.4
 Summary: Durable local queues for Python, with persistent retry state powered by Tenacity.
 Project-URL: Homepage, https://brunoportis.github.io/localqueue/
 Project-URL: Documentation, https://brunoportis.github.io/localqueue/
@@ -48,12 +48,6 @@ Use it for scripts, CLI tools, cron jobs, and small Python workers that share on
 localqueue queue exec emails -- python scripts/send_email.py
 ```
 
-To run the CLI in a clean container:
-
-```bash
-docker run --rm ghcr.io/brunoportis/localqueue:latest --help
-```
-
 ```python
 from localqueue import PersistentQueue, persistent_worker
 
@@ -85,6 +79,15 @@ For the optional LMDB backend:
 pip install "localqueue[lmdb]"
 ```
 
+## Running with Docker
+
+The image is also published on GitHub Container Registry:
+
+```bash
+docker pull ghcr.io/brunoportis/localqueue:latest
+docker run --rm ghcr.io/brunoportis/localqueue:latest --help
+```
+
 ## When not to use
 
 `localqueue` is not a distributed broker. If you need multi-host coordination, high write concurrency, managed retention, or strict cross-service ordering, use a system designed for that operating model.

{localqueue-0.3.2 → localqueue-0.3.4}/README.md

@@ -13,12 +13,6 @@ Use it for scripts, CLI tools, cron jobs, and small Python workers that share on
 localqueue queue exec emails -- python scripts/send_email.py
 ```
 
-To run the CLI in a clean container:
-
-```bash
-docker run --rm ghcr.io/brunoportis/localqueue:latest --help
-```
-
 ```python
 from localqueue import PersistentQueue, persistent_worker
 
@@ -50,6 +44,15 @@ For the optional LMDB backend:
 pip install "localqueue[lmdb]"
 ```
 
+## Running with Docker
+
+The image is also published on GitHub Container Registry:
+
+```bash
+docker pull ghcr.io/brunoportis/localqueue:latest
+docker run --rm ghcr.io/brunoportis/localqueue:latest --help
+```
+
 ## When not to use
 
 `localqueue` is not a distributed broker. If you need multi-host coordination, high write concurrency, managed retention, or strict cross-service ordering, use a system designed for that operating model.

{localqueue-0.3.2 → localqueue-0.3.4}/docs/release.md

@@ -27,48 +27,49 @@ Before `1.0.0`, use this convention:
 ## Checklist
 
 1. Update `version` in `pyproject.toml`.
-2. Update `CHANGELOG.md`.
-3. Confirm `requires-python`, license metadata, and PyPI classifiers still match
+2. Run `uv lock` and commit the updated `uv.lock`.
+3. Update `CHANGELOG.md`.
+4. Confirm `requires-python`, license metadata, and PyPI classifiers still match
    supported Python versions and the intended release policy.
-4. Run the test suite:
+5. Run the test suite:
 
    ```bash
    uv run pytest
    ```
 
-5. Run lint and typecheck:
+6. Run lint and typecheck:
 
    ```bash
    uv run ruff check .
    uv run basedpyright
    ```
 
-6. Build the docs:
+7. Build the docs:
 
    ```bash
    uv run zensical build --clean
    ```
 
-7. Confirm the lockfile is current:
+8. Confirm the lockfile is current:
 
    ```bash
    uv lock --check
    ```
 
-8. Build the release artifacts:
+9. Build the release artifacts:
 
    ```bash
    uv build
    ```
 
-9. Inspect the artifacts:
+10. Inspect the artifacts:
 
    ```bash
    tar -tzf dist/localqueue-<version>.tar.gz | less
    python -m zipfile -l dist/localqueue-<version>-py3-none-any.whl
    ```
 
-10. Install the wheel in a clean virtual environment:
+11. Install the wheel in a clean virtual environment:
 
    ```bash
    uv venv /tmp/localqueue-release-venv
@@ -77,7 +78,7 @@ Before `1.0.0`, use this convention:
    /tmp/localqueue-release-venv/bin/localqueue --help
    ```
 
-11. Run a local smoke test:
+12. Run a local smoke test:
 
     ```bash
     uv run python examples/enqueue_email.py user@example.com \
@@ -90,18 +91,21 @@ Before `1.0.0`, use this convention:
       --max-tries 3
     ```
 
-12. Create the release tag:
+13. Create the release tag:
 
     ```bash
     git tag v<version>
     ```
 
-13. Publish the artifacts.
+14. Publish the artifacts.
 
-14. Confirm the docs deployment finished successfully in the `Documentation`
+    This release workflow also publishes the CLI image to GitHub Container
+    Registry as `ghcr.io/brunoportis/localqueue:<tag>` and `:latest`.
+
+15. Confirm the docs deployment finished successfully in the `Documentation`
     workflow. GitHub Pages must be configured to deploy from GitHub Actions in
     the repository settings.
 
 ## Current Release
 
-The current planned release is `0.3.2`.
+The current planned release is `0.3.4`.

{localqueue-0.3.2 → localqueue-0.3.4}/examples/retry_demo.py

@@ -1,7 +1,7 @@
 import argparse
-import random
 import time
 from queue import Empty
+from secrets import SystemRandom
 from typing import Callable
 from typing import Any
 
@@ -22,6 +22,7 @@ WorkerJobFn = Callable[[str, dict[str, str]], None]
 DEFAULT_EMAIL_FAILURE_RATE = 0.8
 DEFAULT_VIDEO_FAILURE_RATE = 0.7
 DEFAULT_WORKER_FAILURE_RATE = 0.65
+DEMO_RANDOM = SystemRandom()
 
 
 def build_fast_examples(
@@ -38,7 +39,7 @@ def build_fast_examples(
     )
     def send_email_fast(task_id: str, email_address: str) -> None:
         print(f"[send_email_fast] task={task_id} email={email_address}")
-        if random.random() < email_failure_rate:
+        if DEMO_RANDOM.random() < email_failure_rate:
             raise ConnectionError("mail server down")
         print(f"[send_email_fast] task={task_id} sent")
 
@@ -50,7 +51,7 @@ def build_fast_examples(
     )
     def process_video_fast(task_id: str, file_path: str) -> None:
         print(f"[process_video_fast] task={task_id} file={file_path}")
-        if random.random() < video_failure_rate:
+        if DEMO_RANDOM.random() < video_failure_rate:
             raise ValueError("transcoder busy")
         print(f"[process_video_fast] task={task_id} processed")
 
@@ -71,7 +72,7 @@ def build_slow_examples(
     )
     def send_email_slow(task_id: str, email_address: str) -> None:
         print(f"[send_email_slow] task={task_id} email={email_address}")
-        if random.random() < email_failure_rate:
+        if DEMO_RANDOM.random() < email_failure_rate:
             raise ConnectionError("mail server down")
         print(f"[send_email_slow] task={task_id} sent")
 
@@ -83,7 +84,7 @@ def build_slow_examples(
     )
     def process_video_slow(task_id: str, file_path: str) -> None:
         print(f"[process_video_slow] task={task_id} file={file_path}")
-        if random.random() < video_failure_rate:
+        if DEMO_RANDOM.random() < video_failure_rate:
             raise ValueError("transcoder busy")
         print(f"[process_video_slow] task={task_id} processed")
 
@@ -103,7 +104,7 @@ def build_worker_example(
         print(
             f"[worker] job={task_id} kind={payload['kind']} target={payload['target']}"
         )
-        if random.random() < worker_failure_rate:
+        if DEMO_RANDOM.random() < worker_failure_rate:
             raise RuntimeError("transient worker failure")
         print(f"[worker] job={task_id} completed")
 
@@ -183,40 +184,49 @@ def run_worker_demo(
             f"[worker] message={task_id} kind={payload['kind']} "
             + f"target={payload['target']}"
         )
-        if random.random() < worker_failure_rate:
+        if DEMO_RANDOM.random() < worker_failure_rate:
             raise RuntimeError("transient worker failure")
         print(f"[worker] message={task_id} completed")
 
     for cycle in range(1, 8):
-        print(f"\n[worker] poll cycle {cycle}")
-        processed = 0
-
-        while True:
-            try:
-                message = queue.get_message(block=False)
-            except Empty:
-                break
-
-            payload = message.value
-            try:
-                _ = retryer(process_worker_job, message.id, payload)
-            except PersistentRetryExhausted as exc:
-                _ = queue.dead_letter(message)
-                print(
-                    f"[queue] message={exc.key} moved to dead letter "
-                    + f"after {exc.attempts} attempts"
-                )
-            except Exception as exc:
-                _ = queue.release(message, delay=1)
-                print(f"[queue] message={message.id} released for retry: {exc}")
-            else:
-                _ = queue.ack(message)
-                print(f"[queue] message={message.id} acked")
-            processed += 1
-
-        if processed == 0:
-            print("[worker] no tasks available")
-        time.sleep(1)
+        _run_worker_cycle(queue, retryer, cycle, process_worker_job)
+
+
+def _run_worker_cycle(
+    queue: PersistentQueue[dict[str, Any]],
+    retryer: PersistentRetrying,
+    cycle: int,
+    process_worker_job: Any,
+) -> None:
+    print(f"\n[worker] poll cycle {cycle}")
+    processed = 0
+
+    while True:
+        try:
+            message = queue.get_message(block=False)
+        except Empty:
+            break
+
+        payload = message.value
+        try:
+            _ = retryer(process_worker_job, message.id, payload)
+        except PersistentRetryExhausted as exc:
+            _ = queue.dead_letter(message)
+            print(
+                f"[queue] message={exc.key} moved to dead letter "
+                + f"after {exc.attempts} attempts"
+            )
+        except Exception as exc:
+            _ = queue.release(message, delay=1)
+            print(f"[queue] message={message.id} released for retry: {exc}")
+        else:
+            _ = queue.ack(message)
+            print(f"[queue] message={message.id} acked")
+        processed += 1
+
+    if processed == 0:
+        print("[worker] no tasks available")
+    time.sleep(1)
 
 
 def parse_args() -> argparse.Namespace:

localqueue-0.3.4/examples/sqlite_concurrency_benchmark.py

@@ -0,0 +1,313 @@
+#!/usr/bin/env python3
+from __future__ import annotations
+
+import argparse
+import json
+import sqlite3
+import threading
+import time
+from collections import Counter
+from collections.abc import Callable
+from pathlib import Path
+from queue import Empty
+from typing import Any
+
+from localqueue import PersistentQueue
+
+
+def main() -> int:
+    args = _parse_args()
+    summary = _run_benchmark(
+        queue_name=args.queue,
+        store_path=args.store_path,
+        producers=args.producers,
+        consumers=args.consumers,
+        messages=args.messages,
+        lease_timeout=args.lease_timeout,
+    )
+    print(json.dumps(summary, indent=2, sort_keys=True))
+    return 0 if summary["errors"] == 0 else 1
+
+
+def _parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Measure concurrent SQLite queue throughput locally."
+    )
+    parser.add_argument("--queue", default="jobs")
+    parser.add_argument(
+        "--store-path",
+        default="localqueue_queue_benchmark.sqlite3",
+        help="SQLite queue file to use for the benchmark.",
+    )
+    parser.add_argument("--messages", type=int, default=200)
+    parser.add_argument("--producers", type=int, default=4)
+    parser.add_argument("--consumers", type=int, default=4)
+    parser.add_argument("--lease-timeout", type=float, default=0.2)
+    return parser.parse_args()
+
+
+def _run_benchmark(
+    *,
+    queue_name: str,
+    store_path: str,
+    producers: int,
+    consumers: int,
+    messages: int,
+    lease_timeout: float,
+) -> dict[str, Any]:
+    if producers <= 0 or consumers <= 0 or messages <= 0:
+        raise ValueError("producers, consumers, and messages must be greater than zero")
+
+    _remove_sqlite_files(store_path)
+
+    start = threading.Event()
+    producers_done = threading.Event()
+    consumed = Counter[str]()
+    consumed_lock = threading.Lock()
+    errors: list[BaseException] = []
+    errors_lock = threading.Lock()
+    lock_retries = Counter[str]()
+    per_producer, remainder = divmod(messages, producers)
+
+    producer_threads = [
+        threading.Thread(
+            target=_producer_thread,
+            args=(
+                index,
+                queue_name,
+                store_path,
+                lease_timeout,
+                start,
+                per_producer + (1 if index < remainder else 0),
+                lock_retries,
+                errors,
+                errors_lock,
+            ),
+        )
+        for index in range(producers)
+    ]
+    consumer_threads = [
+        threading.Thread(
+            target=_consumer_thread,
+            args=(
+                index,
+                queue_name,
+                store_path,
+                lease_timeout,
+                start,
+                producers_done,
+                consumed,
+                consumed_lock,
+                messages,
+                lock_retries,
+                errors,
+                errors_lock,
+            ),
+        )
+        for index in range(consumers)
+    ]
+    threads = producer_threads + consumer_threads
+
+    started_at = time.perf_counter()
+    _start_threads(threads)
+    start.set()
+
+    _join_benchmark_threads(
+        threads,
+        producer_count=producers,
+        producers_done=producers_done,
+        errors=errors,
+        errors_lock=errors_lock,
+    )
+    elapsed = time.perf_counter() - started_at
+    return _benchmark_summary(
+        queue_name=queue_name,
+        store_path=store_path,
+        lease_timeout=lease_timeout,
+        messages=messages,
+        producers=producers,
+        consumers=consumers,
+        consumed=consumed,
+        errors=errors,
+        lock_retries=lock_retries,
+        elapsed=elapsed,
+    )
+
+
+def _remove_sqlite_files(store_path: str) -> None:
+    store_file = Path(store_path)
+    for suffix in ("", "-wal", "-shm"):
+        candidate = Path(f"{store_file}{suffix}")
+        if candidate.exists():
+            candidate.unlink()
+
+
+def _record_error(
+    exc: BaseException, errors: list[BaseException], errors_lock: threading.Lock
+) -> None:
+    with errors_lock:
+        errors.append(exc)
+
+
+def _retry_sqlite_locked(
+    action: str, fn: Callable[[], Any], lock_retries: Counter[str]
+) -> Any:
+    attempts = 0
+    while True:
+        try:
+            return fn()
+        except sqlite3.OperationalError as exc:
+            if "database is locked" not in str(exc).lower():
+                raise
+            attempts += 1
+            lock_retries[action] += 1
+            time.sleep(min(0.001 * attempts, 0.05))
+
+
+def _producer_thread(
+    index: int,
+    queue_name: str,
+    store_path: str,
+    lease_timeout: float,
+    start: threading.Event,
+    message_count: int,
+    lock_retries: Counter[str],
+    errors: list[BaseException],
+    errors_lock: threading.Lock,
+) -> None:
+    try:
+        queue: PersistentQueue[dict[str, Any]] = PersistentQueue(
+            queue_name, store_path=store_path, lease_timeout=lease_timeout
+        )
+        start.wait()
+        for sequence in range(message_count):
+            _retry_sqlite_locked(
+                "put",
+                lambda sequence=sequence: queue.put(
+                    {"producer": index, "sequence": sequence}
+                ),
+                lock_retries,
+            )
+    except Exception as exc:  # pragma: no cover - defensive
+        _record_error(exc, errors, errors_lock)
+
+
+def _consumer_thread(
+    index: int,
+    queue_name: str,
+    store_path: str,
+    lease_timeout: float,
+    start: threading.Event,
+    producers_done: threading.Event,
+    consumed: Counter[str],
+    consumed_lock: threading.Lock,
+    messages: int,
+    lock_retries: Counter[str],
+    errors: list[BaseException],
+    errors_lock: threading.Lock,
+) -> None:
+    try:
+        queue: PersistentQueue[dict[str, Any]] = PersistentQueue(
+            queue_name, store_path=store_path, lease_timeout=lease_timeout
+        )
+        start.wait()
+        while not _all_messages_consumed(
+            producers_done, consumed, consumed_lock, messages
+        ):
+            try:
+                message = _retry_sqlite_locked(
+                    "get",
+                    lambda: queue.get_message(
+                        block=False, leased_by=f"consumer-{index}"
+                    ),
+                    lock_retries,
+                )
+            except Empty:
+                time.sleep(0.001)
+                continue
+            if not _retry_sqlite_locked(
+                "ack", lambda message=message: queue.ack(message), lock_retries
+            ):
+                _record_error(
+                    RuntimeError(f"ack failed for {message.id}"), errors, errors_lock
+                )
+                return
+            with consumed_lock:
+                consumed[message.id] += 1
+    except Exception as exc:  # pragma: no cover - defensive
+        _record_error(exc, errors, errors_lock)
+
+
+def _all_messages_consumed(
+    producers_done: threading.Event,
+    consumed: Counter[str],
+    consumed_lock: threading.Lock,
+    messages: int,
+) -> bool:
+    with consumed_lock:
+        return producers_done.is_set() and sum(consumed.values()) >= messages
+
+
+def _start_threads(threads: list[threading.Thread]) -> None:
+    for thread in threads:
+        thread.start()
+
+
+def _join_benchmark_threads(
+    threads: list[threading.Thread],
+    *,
+    producer_count: int,
+    producers_done: threading.Event,
+    errors: list[BaseException],
+    errors_lock: threading.Lock,
+) -> None:
+    for thread in threads[:producer_count]:
+        thread.join()
+    producers_done.set()
+
+    for thread in threads[producer_count:]:
+        thread.join(timeout=10.0)
+        if thread.is_alive():
+            _record_error(
+                TimeoutError("consumer thread did not finish"), errors, errors_lock
+            )
+
+
+def _benchmark_summary(
+    *,
+    queue_name: str,
+    store_path: str,
+    lease_timeout: float,
+    messages: int,
+    producers: int,
+    consumers: int,
+    consumed: Counter[str],
+    errors: list[BaseException],
+    lock_retries: Counter[str],
+    elapsed: float,
+) -> dict[str, Any]:
+    queue: PersistentQueue[dict[str, Any]] = PersistentQueue(
+        queue_name, store_path=store_path, lease_timeout=lease_timeout
+    )
+    stats = queue.stats().as_dict()
+    consumed_total = sum(consumed.values())
+    throughput = consumed_total / elapsed if elapsed > 0 else 0.0
+    return {
+        "queue": queue_name,
+        "store_path": str(Path(store_path).resolve()),
+        "messages": messages,
+        "producers": producers,
+        "consumers": consumers,
+        "consumed": consumed_total,
+        "unique_messages": len(consumed),
+        "errors": len(errors),
+        "error_messages": [f"{type(error).__name__}: {error}" for error in errors],
+        "lock_retries": dict(lock_retries),
+        "elapsed_seconds": round(elapsed, 6),
+        "throughput_messages_per_second": round(throughput, 2),
+        "stats": stats,
+    }
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())