serialq 0.1.0__tar.gz

serialq-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 AP (intertermux-code)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

serialq-0.1.0/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: serialq
+Version: 0.1.0
+Summary: One-at-a-time execution: a cross-process serial gate and FIFO job queue
+Author-email: AP <intertermux@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/intertermux-code/serialq
+Project-URL: Issues, https://github.com/intertermux-code/serialq/issues
+Keywords: queue,serial,mutex,lock,job-queue,llm,rate-limit
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# serialq
+
+[![ci](https://github.com/intertermux-code/serialq/actions/workflows/ci.yml/badge.svg)](https://github.com/intertermux-code/serialq/actions)
+
+One-at-a-time execution. A cross-process serial gate plus a persistent FIFO
+job queue, for CLIs and APIs that must never run concurrently.
+
+Some backends are strictly serial: one in-flight call at a time, across
+every model, key, or client — overlap and you get rate-limited, corrupted,
+or billed twice. Cron jobs don't know about each other, shell scripts
+don't coordinate, and "just be careful" stops working at 3am. serialq is
+the bouncer: whoever holds the gate is the only thing running.
+
+```sh
+pip install git+https://github.com/intertermux-code/serialq.git
+
+# ad-hoc: blocks until the gate is free, then runs
+serialq run --gate qwen -- my-llm-cli ask "summarize this thread"
+
+# fire-and-forget: queue it, a worker runs jobs one by one
+serialq enqueue --gate qwen -- my-llm-cli batch jobs/42.json
+serialq worker --gate qwen            # drain forever (systemd, tmux, ...)
+serialq worker --gate qwen --once     # drain once, then exit (cron-friendly)
+
+serialq list --gate qwen
+serialq log 20261005-a3f9c1
+serialq status --gate qwen
+```
+
+Zero dependencies, standard library only. POSIX only (Linux, macOS) —
+it relies on `fcntl` locks.
+
+## Why not just a lock file?
+
+Because lock files go stale. serialq uses `fcntl` advisory locks, which the
+kernel releases when the holder process dies — a crashed job can never wedge
+the gate. The queue goes further:
+
+- **Crash recovery.** If a worker is kill -9'd mid-job, the next worker
+  re-queues the orphaned job instead of losing it. Only one worker per gate
+  can run at a time, so the recovery is unambiguous.
+- **Timeouts that actually kill.** `--kill-after` terminates the whole
+  process group (SIGTERM, then SIGKILL), not just the parent.
+- **Retries with backoff.** `enqueue --retries 3` re-queues failures with
+  exponential backoff (5s, 10s, 20s … capped at 5 minutes).
+- **Atomic queue writes.** The job store is rewritten under an exclusive
+  lock with fsync; a corrupt store is quarantined, never silently dropped.
+- **Ctrl-C behaves.** `serialq run` forwards SIGINT to the child, so
+  interactive commands interrupt the way you'd expect.
+
+## Use cases
+
+- **Serial-only LLM backends.** Some model gateways allow exactly one
+  in-flight request across all models. Prefix every call and stop thinking
+  about it:
+  ```sh
+  serialq run --gate qwen --timeout 3600 -- llm chat model-a "prompt one" &
+  serialq run --gate qwen --timeout 3600 -- llm chat model-b "prompt two" &
+  wait  # they ran sequentially, in order
+  ```
+- **Overnight batch pipelines.** Enqueue a hundred jobs, let one worker
+  chew through them; check `serialq list --all` in the morning.
+- **License-limited tools.** One floating license, many cron jobs — put
+  the tool behind a gate named after the license.
+- **Flaky deploys.** `enqueue --retries 5` on the deploy script; transient
+  failures retry themselves with backoff.
+
+## Reference
+
+| Command | What it does |
+|---|---|
+| `run [-g GATE] [--timeout S] [--kill-after S] -- CMD…` | Run CMD under the gate, blocking until free. Exits with CMD's exit code. |
+| `enqueue [-g GATE] [--name N] [--retries N] [--kill-after S] -- CMD…` | Queue CMD, print the job id. |
+| `worker [-g GATE] [--once] [--idle-timeout S]` | Run queued jobs FIFO. SIGTERM/SIGINT finish the current job, then stop. |
+| `list [-g GATE] [--all]` | Show queued/running jobs (`--all` includes finished). |
+| `log [-g GATE] ID` | Print a job's captured output. |
+| `cancel [-g GATE] ID` | Cancel a queued job. |
+| `status [-g GATE]` | Gate busy/free plus queue counts. |
+
+Gates are just names (`[A-Za-z0-9_-]`, `--gate` or `SERIALQ_GATE` env).
+State lives in `SERIALQ_DIR` (default `~/.local/share/serialq`), one
+directory per gate: the lock files, `jobs.json`, and per-job logs.
+
+A systemd user unit template ships in `contrib/`:
+
+```sh
+cp contrib/serialq-worker@.service ~/.config/systemd/user/
+systemctl --user enable --now serialq-worker@qwen
+```
+
+## Design notes
+
+- One module, ~500 lines, no dependencies. The whole thing fits in your head.
+- The gate and the queue are separate locks: `enqueue`/`list` never block
+  behind a long-running job.
+- Job ids are `YYYYMMDD-` plus 6 hex chars — sortable, greppable, unique.
+- Exit code 124-style semantics aren't faked: timeouts are reported in the
+  job log and on stderr.
+
+## Limitations
+
+- POSIX only. Windows would need a different locking primitive.
+- FIFO, no priorities — deliberate. If you need priorities you probably
+  need a real queue.
+- The worker is single-threaded by design: one gate, one job at a time.
+  That's the point.
+
+## License
+
+MIT.

serialq-0.1.0/README.md

@@ -0,0 +1,110 @@
+# serialq
+
+[![ci](https://github.com/intertermux-code/serialq/actions/workflows/ci.yml/badge.svg)](https://github.com/intertermux-code/serialq/actions)
+
+One-at-a-time execution. A cross-process serial gate plus a persistent FIFO
+job queue, for CLIs and APIs that must never run concurrently.
+
+Some backends are strictly serial: one in-flight call at a time, across
+every model, key, or client — overlap and you get rate-limited, corrupted,
+or billed twice. Cron jobs don't know about each other, shell scripts
+don't coordinate, and "just be careful" stops working at 3am. serialq is
+the bouncer: whoever holds the gate is the only thing running.
+
+```sh
+pip install git+https://github.com/intertermux-code/serialq.git
+
+# ad-hoc: blocks until the gate is free, then runs
+serialq run --gate qwen -- my-llm-cli ask "summarize this thread"
+
+# fire-and-forget: queue it, a worker runs jobs one by one
+serialq enqueue --gate qwen -- my-llm-cli batch jobs/42.json
+serialq worker --gate qwen            # drain forever (systemd, tmux, ...)
+serialq worker --gate qwen --once     # drain once, then exit (cron-friendly)
+
+serialq list --gate qwen
+serialq log 20261005-a3f9c1
+serialq status --gate qwen
+```
+
+Zero dependencies, standard library only. POSIX only (Linux, macOS) —
+it relies on `fcntl` locks.
+
+## Why not just a lock file?
+
+Because lock files go stale. serialq uses `fcntl` advisory locks, which the
+kernel releases when the holder process dies — a crashed job can never wedge
+the gate. The queue goes further:
+
+- **Crash recovery.** If a worker is kill -9'd mid-job, the next worker
+  re-queues the orphaned job instead of losing it. Only one worker per gate
+  can run at a time, so the recovery is unambiguous.
+- **Timeouts that actually kill.** `--kill-after` terminates the whole
+  process group (SIGTERM, then SIGKILL), not just the parent.
+- **Retries with backoff.** `enqueue --retries 3` re-queues failures with
+  exponential backoff (5s, 10s, 20s … capped at 5 minutes).
+- **Atomic queue writes.** The job store is rewritten under an exclusive
+  lock with fsync; a corrupt store is quarantined, never silently dropped.
+- **Ctrl-C behaves.** `serialq run` forwards SIGINT to the child, so
+  interactive commands interrupt the way you'd expect.
+
+## Use cases
+
+- **Serial-only LLM backends.** Some model gateways allow exactly one
+  in-flight request across all models. Prefix every call and stop thinking
+  about it:
+  ```sh
+  serialq run --gate qwen --timeout 3600 -- llm chat model-a "prompt one" &
+  serialq run --gate qwen --timeout 3600 -- llm chat model-b "prompt two" &
+  wait  # they ran sequentially, in order
+  ```
+- **Overnight batch pipelines.** Enqueue a hundred jobs, let one worker
+  chew through them; check `serialq list --all` in the morning.
+- **License-limited tools.** One floating license, many cron jobs — put
+  the tool behind a gate named after the license.
+- **Flaky deploys.** `enqueue --retries 5` on the deploy script; transient
+  failures retry themselves with backoff.
+
+## Reference
+
+| Command | What it does |
+|---|---|
+| `run [-g GATE] [--timeout S] [--kill-after S] -- CMD…` | Run CMD under the gate, blocking until free. Exits with CMD's exit code. |
+| `enqueue [-g GATE] [--name N] [--retries N] [--kill-after S] -- CMD…` | Queue CMD, print the job id. |
+| `worker [-g GATE] [--once] [--idle-timeout S]` | Run queued jobs FIFO. SIGTERM/SIGINT finish the current job, then stop. |
+| `list [-g GATE] [--all]` | Show queued/running jobs (`--all` includes finished). |
+| `log [-g GATE] ID` | Print a job's captured output. |
+| `cancel [-g GATE] ID` | Cancel a queued job. |
+| `status [-g GATE]` | Gate busy/free plus queue counts. |
+
+Gates are just names (`[A-Za-z0-9_-]`, `--gate` or `SERIALQ_GATE` env).
+State lives in `SERIALQ_DIR` (default `~/.local/share/serialq`), one
+directory per gate: the lock files, `jobs.json`, and per-job logs.
+
+A systemd user unit template ships in `contrib/`:
+
+```sh
+cp contrib/serialq-worker@.service ~/.config/systemd/user/
+systemctl --user enable --now serialq-worker@qwen
+```
+
+## Design notes
+
+- One module, ~500 lines, no dependencies. The whole thing fits in your head.
+- The gate and the queue are separate locks: `enqueue`/`list` never block
+  behind a long-running job.
+- Job ids are `YYYYMMDD-` plus 6 hex chars — sortable, greppable, unique.
+- Exit code 124-style semantics aren't faked: timeouts are reported in the
+  job log and on stderr.
+
+## Limitations
+
+- POSIX only. Windows would need a different locking primitive.
+- FIFO, no priorities — deliberate. If you need priorities you probably
+  need a real queue.
+- The worker is single-threaded by design: one gate, one job at a time.
+  That's the point.
+
+## License
+
+MIT.

serialq-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "serialq"
+version = "0.1.0"
+description = "One-at-a-time execution: a cross-process serial gate and FIFO job queue"
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "AP", email = "intertermux@gmail.com" }]
+keywords = ["queue", "serial", "mutex", "lock", "job-queue", "llm", "rate-limit"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Environment :: Console",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: POSIX :: Linux",
+    "Operating System :: MacOS",
+    "Programming Language :: Python :: 3",
+    "Topic :: System :: Systems Administration",
+    "Topic :: Utilities",
+]
+dependencies = []
+
+[project.urls]
+Homepage = "https://github.com/intertermux-code/serialq"
+Issues = "https://github.com/intertermux-code/serialq/issues"
+
+[project.scripts]
+serialq = "serialq:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

serialq-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

serialq-0.1.0/src/serialq/__init__.py

@@ -0,0 +1,601 @@
+#!/usr/bin/env python3
+"""
+serialq — one-at-a-time execution.
+
+A cross-process serial gate plus a persistent FIFO job queue, for CLIs and
+APIs that must never run concurrently: strictly-serial LLM backends,
+license-limited tools, shared hardware, flaky rate limits.
+
+Two primitives, one guarantee — whoever holds the gate is the only thing
+running:
+
+    serialq run --gate qwen -- my-llm-cli ask "summarize this"
+    serialq enqueue --gate qwen -- my-llm-cli batch job-42.json
+    serialq worker --gate qwen            # drains the queue, forever
+    serialq worker --gate qwen --once     # drains the queue, then exits
+
+Standard library only. POSIX only (needs fcntl).
+"""
+
+import argparse
+import fcntl
+import json
+import os
+import re
+import shlex
+import signal
+import subprocess
+import sys
+import time
+from datetime import datetime, timezone
+
+VERSION = "0.1.0"
+GATE_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_-]{0,63}$")
+DEFAULT_GATE = os.environ.get("SERIALQ_GATE", "default")
+TERM_GRACE_SECS = 5
+
+
+def die(msg, code=1):
+    print(f"serialq: error: {msg}", file=sys.stderr)
+    sys.exit(code)
+
+
+def now_iso():
+    return datetime.now(timezone.utc).isoformat(timespec="seconds")
+
+
+# ---------------------------------------------------------------- paths/locks
+
+def gate_paths(gate):
+    if not GATE_RE.match(gate or ""):
+        die(f"invalid gate name {gate!r}: use letters, digits, '-' and '_' (max 64 chars)")
+    base = os.environ.get(
+        "SERIALQ_DIR",
+        os.path.join(os.path.expanduser("~"), ".local", "share", "serialq"),
+    )
+    d = os.path.join(base, "gates", gate)
+    os.makedirs(os.path.join(d, "logs"), exist_ok=True)
+    return {
+        "dir": d,
+        "gate_lock": os.path.join(d, "gate.lock"),
+        "worker_lock": os.path.join(d, "worker.lock"),
+        "jobs": os.path.join(d, "jobs.json"),
+        "logs": os.path.join(d, "logs"),
+    }
+
+
+class LockFile:
+    """Advisory fcntl lock. The kernel releases it when the holder dies,
+    so a crashed process can never leave a stale lock behind."""
+
+    def __init__(self, path):
+        self.path = path
+        self.fh = None
+
+    def acquire(self, timeout=None):
+        """blocking=True semantics; timeout=None waits forever.
+        Returns True on success, False on timeout."""
+        self.fh = open(self.path, "a+b")
+        if timeout is None:
+            fcntl.flock(self.fh, fcntl.LOCK_EX)
+            return True
+        deadline = time.monotonic() + timeout
+        while True:
+            try:
+                fcntl.flock(self.fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
+                return True
+            except BlockingIOError:
+                if time.monotonic() >= deadline:
+                    self.fh.close()
+                    self.fh = None
+                    return False
+                time.sleep(0.05)
+
+    def try_acquire(self):
+        self.fh = open(self.path, "a+b")
+        try:
+            fcntl.flock(self.fh, fcntl.LOCK_EX | fcntl.LOCK_NB)
+            return True
+        except BlockingIOError:
+            self.fh.close()
+            self.fh = None
+            return False
+
+    def release(self):
+        if self.fh is not None:
+            try:
+                fcntl.flock(self.fh, fcntl.LOCK_UN)
+            finally:
+                self.fh.close()
+                self.fh = None
+
+    def __enter__(self):
+        self.acquire()
+        return self
+
+    def __exit__(self, *exc):
+        self.release()
+
+
+# ---------------------------------------------------------------- job store
+
+def _read_jobs(fh, jobs_path):
+    fh.seek(0)
+    raw = fh.read().strip()
+    if not raw:
+        return {}
+    try:
+        jobs = json.loads(raw)
+    except json.JSONDecodeError:
+        # Never silently drop the queue: quarantine the corrupt file.
+        bad = jobs_path + ".corrupt-" + datetime.now(timezone.utc).strftime("%Y%m%dT%H%M%S")
+        os.rename(jobs_path, bad)
+        print(f"serialq: warning: quarantined corrupt job store to {bad}", file=sys.stderr)
+        return {}
+    return jobs if isinstance(jobs, dict) else {}
+
+
+def with_jobs(paths, fn):
+    """Run fn(jobs) with the queue lock held; persist when fn returns True."""
+    fh = open(paths["jobs"], "a+b")
+    try:
+        fcntl.flock(fh, fcntl.LOCK_EX)
+        jobs = _read_jobs(fh, paths["jobs"])
+        if fn(jobs):
+            fh.seek(0)
+            fh.truncate()
+            fh.write(json.dumps(jobs, indent=1, sort_keys=True).encode())
+            fh.flush()
+            os.fsync(fh.fileno())
+    finally:
+        fcntl.flock(fh, fcntl.LOCK_UN)
+        fh.close()
+
+
+def _iter_jobs(jobs):
+    """Job records only (skips the __meta__ bookkeeping entry)."""
+    return [j for k, j in jobs.items() if k != "__meta__" and isinstance(j, dict)]
+
+
+def new_job_id(jobs):
+    stamp = datetime.now(timezone.utc).strftime("%Y%m%d")
+    while True:
+        jid = f"{stamp}-" + "".join(
+            "0123456789abcdef"[b % 16] for b in os.urandom(6)
+        )
+        if jid not in jobs:
+            return jid
+
+
+def next_seq(jobs):
+    """Monotonic insertion counter, assigned under the queue lock so FIFO
+    order is exact even when several jobs share a timestamp."""
+    meta = jobs.setdefault("__meta__", {"seq": 0})
+    meta["seq"] += 1
+    return meta["seq"]
+
+
+# ---------------------------------------------------------------- processes
+
+def _kill_tree(proc):
+    """SIGTERM the whole process group, escalate to SIGKILL after a grace period."""
+    try:
+        os.killpg(proc.pid, signal.SIGTERM)
+    except (ProcessLookupError, PermissionError):
+        return
+    try:
+        proc.wait(timeout=TERM_GRACE_SECS)
+    except subprocess.TimeoutExpired:
+        try:
+            os.killpg(proc.pid, signal.SIGKILL)
+        except (ProcessLookupError, PermissionError):
+            pass
+        proc.wait()
+
+
+def run_child(cmd, log_path=None, kill_after=None, forward_signals=False):
+    """Run cmd in its own process group.
+
+    log_path=None inherits stdio (foreground `run`); otherwise stdout+stderr
+    are appended to the log file. Returns (exit_code, timed_out).
+    With forward_signals, SIGINT/SIGTERM are relayed to the child group first
+    so Ctrl-C behaves like a normal foreground command.
+    """
+    log = open(log_path, "a") if log_path else None
+    if log:
+        log.write(f"# started {now_iso()} :: {' '.join(shlex.quote(c) for c in cmd)}\n")
+        log.flush()
+    try:
+        proc = subprocess.Popen(
+            cmd,
+            stdout=log if log else None,
+            stderr=subprocess.STDOUT if log else None,
+            start_new_session=True,
+        )
+    except FileNotFoundError:
+        msg = f"failed to start: {cmd[0]}: command not found"
+        if log:
+            log.write(f"# {msg}\n")
+            log.close()
+        else:
+            print(f"serialq: error: {msg}", file=sys.stderr)
+        return 127, False
+    except OSError as e:
+        msg = f"failed to start: {e}"
+        if log:
+            log.write(f"# {msg}\n")
+            log.close()
+        else:
+            print(f"serialq: error: {msg}", file=sys.stderr)
+        return 126, False
+
+    relay = {}
+
+    def _relay(signum, _frame):
+        try:
+            os.killpg(proc.pid, signum)
+        except (ProcessLookupError, PermissionError):
+            pass
+        relay["signum"] = signum
+
+    if forward_signals:
+        old_int = signal.signal(signal.SIGINT, _relay)
+        old_term = signal.signal(signal.SIGTERM, _relay)
+
+    timed_out = False
+    deadline = time.monotonic() + kill_after if kill_after else None
+    try:
+        while True:
+            try:
+                rc = proc.wait(timeout=0.2)
+                break
+            except subprocess.TimeoutExpired:
+                pass
+            if "signum" in relay:
+                rc = proc.wait()  # child got the signal; reap it
+                break
+            if deadline is not None and time.monotonic() >= deadline:
+                _kill_tree(proc)
+                rc = proc.wait()
+                timed_out = True
+                break
+    finally:
+        if forward_signals:
+            signal.signal(signal.SIGINT, old_int)
+            signal.signal(signal.SIGTERM, old_term)
+
+    if log:
+        log.write(f"# ended {now_iso()} :: exit={rc}" + (" (timed out)" if timed_out else "") + "\n")
+        log.close()
+    if "signum" in relay and relay["signum"] == signal.SIGINT:
+        # Behave like a normal interrupted foreground command.
+        raise KeyboardInterrupt
+    return rc, timed_out
+
+
+# ---------------------------------------------------------------- commands
+
+def cmd_run(args):
+    cmd = [c for c in args.cmd if c != "--"]
+    if not cmd:
+        die("no command given")
+    paths = gate_paths(args.gate)
+    gate = LockFile(paths["gate_lock"])
+    if args.timeout is not None and args.timeout < 0:
+        die("--timeout must be >= 0")
+    if not gate.acquire(timeout=args.timeout):
+        die(f"timed out after {args.timeout}s waiting for gate {args.gate!r}")
+    try:
+        rc, timed_out = run_child(cmd, kill_after=args.kill_after, forward_signals=True)
+    finally:
+        gate.release()
+    if timed_out:
+        print(f"serialq: command killed after {args.kill_after}s", file=sys.stderr)
+    sys.exit(rc if rc >= 0 else 128 - rc)
+
+
+def cmd_enqueue(args):
+    cmd = [c for c in args.cmd if c != "--"]
+    if not cmd:
+        die("no command given")
+    if args.retries < 0:
+        die("--retries must be >= 0")
+    paths = gate_paths(args.gate)
+    job = {
+        "id": None,
+        "gate": args.gate,
+        "name": args.name or " ".join(cmd)[:80],
+        "cmd": cmd,
+        "created": now_iso(),
+        "seq": None,
+        "status": "queued",
+        "attempts": 0,
+        "max_retries": args.retries,
+        "kill_after": args.kill_after,
+        "not_before": 0,
+        "started": None,
+        "ended": None,
+        "exit_code": None,
+        "error": None,
+    }
+
+    def _add(jobs):
+        job["id"] = new_job_id(jobs)
+        job["seq"] = next_seq(jobs)
+        jobs[job["id"]] = job
+        return True
+
+    with_jobs(paths, _add)
+    print(job["id"])
+
+
+def _pick_job(jobs):
+    """Oldest queued job whose backoff has elapsed (FIFO by insertion order)."""
+    now = time.time()
+    candidates = [
+        j for j in _iter_jobs(jobs)
+        if j["status"] == "queued" and j.get("not_before", 0) <= now
+    ]
+    if not candidates:
+        return None
+    return min(candidates, key=lambda j: (j.get("seq", 0), j["id"]))
+
+
+def _has_pending(jobs):
+    """Any job not yet in a terminal state (matters for --once + backoff)."""
+    return any(j["status"] in ("queued", "running") for j in _iter_jobs(jobs))
+
+
+def cmd_worker(args):
+    paths = gate_paths(args.gate)
+    worker_lock = LockFile(paths["worker_lock"])
+    if not worker_lock.try_acquire():
+        die(f"another worker is already running for gate {args.gate!r}")
+    gate = LockFile(paths["gate_lock"])
+    stop = {"flag": False}
+
+    def _on_signal(_signum, _frame):
+        stop["flag"] = True
+
+    signal.signal(signal.SIGTERM, _on_signal)
+    signal.signal(signal.SIGINT, _on_signal)
+
+    # Crash recovery: this worker owns the lifecycle now (it holds
+    # worker_lock), so any job still marked running belongs to a dead worker.
+    def _recover(jobs):
+        changed = False
+        for j in _iter_jobs(jobs):
+            if j["status"] == "running":
+                j["status"] = "queued"
+                j["error"] = "requeued: previous worker died mid-job"
+                changed = True
+        return changed
+
+    with_jobs(paths, _recover)
+
+    idle_since = time.monotonic()
+    try:
+        while not stop["flag"]:
+            job = {}
+
+            def _claim(jobs):
+                j = _pick_job(jobs)
+                if j is None:
+                    return False
+                j["status"] = "running"
+                j["attempts"] += 1
+                j["started"] = now_iso()
+                j["error"] = None
+                job.update(j)
+                return True
+
+            with_jobs(paths, _claim)
+            if not job:
+                pending = []
+
+                def _check(jobs):
+                    pending.append(_has_pending(jobs))
+                    return False
+
+                with_jobs(paths, _check)
+                if args.once and not pending[0]:
+                    break
+                if args.idle_timeout and time.monotonic() - idle_since >= args.idle_timeout:
+                    break
+                time.sleep(0.5)
+                continue
+            idle_since = time.monotonic()
+
+            # Wait for the gate in slices so signals stay responsive.
+            while not stop["flag"]:
+                if gate.acquire(timeout=0.5):
+                    break
+            if stop["flag"]:
+                # Hand the job back untouched; a later worker will run it.
+                def _unclaim(jobs):
+                    j = jobs.get(job["id"])
+                    if j and j["status"] == "running":
+                        j["status"] = "queued"
+                        j["attempts"] -= 1
+                        return True
+                    return False
+
+                with_jobs(paths, _unclaim)
+                break
+
+            try:
+                log_path = os.path.join(paths["logs"], job["id"] + ".log")
+                rc, timed_out = run_child(
+                    job["cmd"], log_path=log_path, kill_after=job.get("kill_after")
+                )
+            finally:
+                gate.release()
+
+            def _finish(jobs):
+                j = jobs.get(job["id"])
+                if not j:
+                    return False
+                j["ended"] = now_iso()
+                j["exit_code"] = rc
+                if rc == 0:
+                    j["status"] = "done"
+                    return True
+                if timed_out:
+                    j["error"] = f"killed after {job.get('kill_after')}s (timeout)"
+                else:
+                    j["error"] = f"exit code {rc}"
+                if j["attempts"] <= j["max_retries"]:
+                    backoff = min(300, 5 * 2 ** (j["attempts"] - 1))
+                    j["status"] = "queued"
+                    j["not_before"] = time.time() + backoff
+                    j["error"] += f"; retrying in {backoff:.0f}s (attempt {j['attempts']}/{j['max_retries'] + 1})"
+                else:
+                    j["status"] = "failed"
+                return True
+
+            with_jobs(paths, _finish)
+    finally:
+        worker_lock.release()
+
+
+def cmd_list(args):
+    paths = gate_paths(args.gate)
+    rows = []
+
+    def _collect(jobs):
+        for j in sorted(_iter_jobs(jobs), key=lambda j: (j.get("seq", 0), j["id"])):
+            if not args.all and j["status"] not in ("queued", "running"):
+                continue
+            rows.append(j)
+        return False
+
+    with_jobs(paths, _collect)
+    if not rows:
+        print("(no jobs)" if args.all else "(queue empty)")
+        return
+    print(f"{'ID':<20}{'NAME':<34}{'STATUS':<10}{'TRY':<5}{'EXIT':<6}CREATED")
+    for j in rows:
+        exit_code = "" if j["exit_code"] is None else str(j["exit_code"])
+        print(f"{j['id']:<20}{j['name'][:33]:<34}{j['status']:<10}{j['attempts']:<5}{exit_code:<6}{j['created']}")
+
+
+def cmd_log(args):
+    paths = gate_paths(args.gate)
+    log_path = os.path.join(paths["logs"], args.id + ".log")
+    if not os.path.exists(log_path):
+        die(f"no log for job {args.id!r}")
+    with open(log_path) as f:
+        sys.stdout.write(f.read())
+
+
+def cmd_cancel(args):
+    paths = gate_paths(args.gate)
+
+    def _cancel(jobs):
+        j = jobs.get(args.id)
+        if j is None:
+            die(f"no such job {args.id!r}")
+        if j["status"] != "queued":
+            die(f"job {args.id!r} is {j['status']}, only queued jobs can be cancelled")
+        j["status"] = "cancelled"
+        j["ended"] = now_iso()
+        return True
+
+    with_jobs(paths, _cancel)
+    print(f"cancelled {args.id}")
+
+
+def cmd_status(args):
+    paths = gate_paths(args.gate)
+    gate = LockFile(paths["gate_lock"])
+    held = not gate.try_acquire()
+    if not held:
+        gate.release()
+    counts = {"queued": 0, "running": 0, "done": 0, "failed": 0, "cancelled": 0}
+    running_id = None
+
+    def _count(jobs):
+        for j in _iter_jobs(jobs):
+            counts[j["status"]] = counts.get(j["status"], 0) + 1
+            if j["status"] == "running":
+                nonlocal_running[0] = j["id"]
+        return False
+
+    nonlocal_running = [None]
+    with_jobs(paths, _count)
+    running_id = nonlocal_running[0]
+    print(f"gate {args.gate!r}: {'BUSY' if held else 'free'}")
+    print(f"queued={counts['queued']} running={counts['running']} "
+          f"done={counts['done']} failed={counts['failed']} cancelled={counts['cancelled']}")
+    if running_id:
+        print(f"running job: {running_id}")
+
+
+# ---------------------------------------------------------------- cli
+
+def build_parser():
+    p = argparse.ArgumentParser(
+        prog="serialq",
+        description="One-at-a-time execution: a serial gate and FIFO job queue.",
+    )
+    p.add_argument("--version", action="version", version=f"%(prog)s {VERSION}")
+    sub = p.add_subparsers(dest="command", required=True)
+
+    r = sub.add_parser("run", help="run a command under the gate (blocks until free)")
+    r.add_argument("-g", "--gate", default=DEFAULT_GATE, help="gate name (default: %(default)s)")
+    r.add_argument("--timeout", type=float, default=None,
+                   help="max seconds to wait for the gate (default: wait forever)")
+    r.add_argument("--kill-after", type=float, default=None,
+                   help="kill the command if it runs longer than this many seconds")
+    r.add_argument("cmd", nargs=argparse.REMAINDER, help="command to run (after --)")
+    r.set_defaults(func=cmd_run)
+
+    e = sub.add_parser("enqueue", help="queue a command to run later, prints the job id")
+    e.add_argument("-g", "--gate", default=DEFAULT_GATE)
+    e.add_argument("--name", default=None, help="human-readable job name")
+    e.add_argument("--retries", type=int, default=0, help="retries on failure (default: 0)")
+    e.add_argument("--kill-after", type=float, default=None)
+    e.add_argument("cmd", nargs=argparse.REMAINDER, help="command to run (after --)")
+    e.set_defaults(func=cmd_enqueue)
+
+    w = sub.add_parser("worker", help="drain the queue serially")
+    w.add_argument("-g", "--gate", default=DEFAULT_GATE)
+    w.add_argument("--once", action="store_true", help="exit when the queue is empty")
+    w.add_argument("--idle-timeout", type=float, default=None,
+                   help="exit after this many idle seconds (default: run forever)")
+    w.set_defaults(func=cmd_worker)
+
+    l = sub.add_parser("list", help="list jobs")
+    l.add_argument("-g", "--gate", default=DEFAULT_GATE)
+    l.add_argument("--all", action="store_true", help="include finished jobs")
+    l.set_defaults(func=cmd_list)
+
+    g = sub.add_parser("log", help="print a job's log")
+    g.add_argument("-g", "--gate", default=DEFAULT_GATE)
+    g.add_argument("id", help="job id")
+    g.set_defaults(func=cmd_log)
+
+    c = sub.add_parser("cancel", help="cancel a queued job")
+    c.add_argument("-g", "--gate", default=DEFAULT_GATE)
+    c.add_argument("id", help="job id")
+    c.set_defaults(func=cmd_cancel)
+
+    s = sub.add_parser("status", help="show gate and queue status")
+    s.add_argument("-g", "--gate", default=DEFAULT_GATE)
+    s.set_defaults(func=cmd_status)
+
+    return p
+
+
+def main(argv=None):
+    args = build_parser().parse_args(argv)
+    try:
+        args.func(args)
+    except KeyboardInterrupt:
+        sys.exit(130)
+    except BrokenPipeError:
+        sys.exit(0)
+
+
+if __name__ == "__main__":
+    main()

serialq-0.1.0/src/serialq/__main__.py

@@ -0,0 +1,3 @@
+from serialq import main
+
+main()

serialq-0.1.0/src/serialq.egg-info/PKG-INFO

@@ -0,0 +1,133 @@
+Metadata-Version: 2.4
+Name: serialq
+Version: 0.1.0
+Summary: One-at-a-time execution: a cross-process serial gate and FIFO job queue
+Author-email: AP <intertermux@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/intertermux-code/serialq
+Project-URL: Issues, https://github.com/intertermux-code/serialq/issues
+Keywords: queue,serial,mutex,lock,job-queue,llm,rate-limit
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Utilities
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# serialq
+
+[![ci](https://github.com/intertermux-code/serialq/actions/workflows/ci.yml/badge.svg)](https://github.com/intertermux-code/serialq/actions)
+
+One-at-a-time execution. A cross-process serial gate plus a persistent FIFO
+job queue, for CLIs and APIs that must never run concurrently.
+
+Some backends are strictly serial: one in-flight call at a time, across
+every model, key, or client — overlap and you get rate-limited, corrupted,
+or billed twice. Cron jobs don't know about each other, shell scripts
+don't coordinate, and "just be careful" stops working at 3am. serialq is
+the bouncer: whoever holds the gate is the only thing running.
+
+```sh
+pip install git+https://github.com/intertermux-code/serialq.git
+
+# ad-hoc: blocks until the gate is free, then runs
+serialq run --gate qwen -- my-llm-cli ask "summarize this thread"
+
+# fire-and-forget: queue it, a worker runs jobs one by one
+serialq enqueue --gate qwen -- my-llm-cli batch jobs/42.json
+serialq worker --gate qwen            # drain forever (systemd, tmux, ...)
+serialq worker --gate qwen --once     # drain once, then exit (cron-friendly)
+
+serialq list --gate qwen
+serialq log 20261005-a3f9c1
+serialq status --gate qwen
+```
+
+Zero dependencies, standard library only. POSIX only (Linux, macOS) —
+it relies on `fcntl` locks.
+
+## Why not just a lock file?
+
+Because lock files go stale. serialq uses `fcntl` advisory locks, which the
+kernel releases when the holder process dies — a crashed job can never wedge
+the gate. The queue goes further:
+
+- **Crash recovery.** If a worker is kill -9'd mid-job, the next worker
+  re-queues the orphaned job instead of losing it. Only one worker per gate
+  can run at a time, so the recovery is unambiguous.
+- **Timeouts that actually kill.** `--kill-after` terminates the whole
+  process group (SIGTERM, then SIGKILL), not just the parent.
+- **Retries with backoff.** `enqueue --retries 3` re-queues failures with
+  exponential backoff (5s, 10s, 20s … capped at 5 minutes).
+- **Atomic queue writes.** The job store is rewritten under an exclusive
+  lock with fsync; a corrupt store is quarantined, never silently dropped.
+- **Ctrl-C behaves.** `serialq run` forwards SIGINT to the child, so
+  interactive commands interrupt the way you'd expect.
+
+## Use cases
+
+- **Serial-only LLM backends.** Some model gateways allow exactly one
+  in-flight request across all models. Prefix every call and stop thinking
+  about it:
+  ```sh
+  serialq run --gate qwen --timeout 3600 -- llm chat model-a "prompt one" &
+  serialq run --gate qwen --timeout 3600 -- llm chat model-b "prompt two" &
+  wait  # they ran sequentially, in order
+  ```
+- **Overnight batch pipelines.** Enqueue a hundred jobs, let one worker
+  chew through them; check `serialq list --all` in the morning.
+- **License-limited tools.** One floating license, many cron jobs — put
+  the tool behind a gate named after the license.
+- **Flaky deploys.** `enqueue --retries 5` on the deploy script; transient
+  failures retry themselves with backoff.
+
+## Reference
+
+| Command | What it does |
+|---|---|
+| `run [-g GATE] [--timeout S] [--kill-after S] -- CMD…` | Run CMD under the gate, blocking until free. Exits with CMD's exit code. |
+| `enqueue [-g GATE] [--name N] [--retries N] [--kill-after S] -- CMD…` | Queue CMD, print the job id. |
+| `worker [-g GATE] [--once] [--idle-timeout S]` | Run queued jobs FIFO. SIGTERM/SIGINT finish the current job, then stop. |
+| `list [-g GATE] [--all]` | Show queued/running jobs (`--all` includes finished). |
+| `log [-g GATE] ID` | Print a job's captured output. |
+| `cancel [-g GATE] ID` | Cancel a queued job. |
+| `status [-g GATE]` | Gate busy/free plus queue counts. |
+
+Gates are just names (`[A-Za-z0-9_-]`, `--gate` or `SERIALQ_GATE` env).
+State lives in `SERIALQ_DIR` (default `~/.local/share/serialq`), one
+directory per gate: the lock files, `jobs.json`, and per-job logs.
+
+A systemd user unit template ships in `contrib/`:
+
+```sh
+cp contrib/serialq-worker@.service ~/.config/systemd/user/
+systemctl --user enable --now serialq-worker@qwen
+```
+
+## Design notes
+
+- One module, ~500 lines, no dependencies. The whole thing fits in your head.
+- The gate and the queue are separate locks: `enqueue`/`list` never block
+  behind a long-running job.
+- Job ids are `YYYYMMDD-` plus 6 hex chars — sortable, greppable, unique.
+- Exit code 124-style semantics aren't faked: timeouts are reported in the
+  job log and on stderr.
+
+## Limitations
+
+- POSIX only. Windows would need a different locking primitive.
+- FIFO, no priorities — deliberate. If you need priorities you probably
+  need a real queue.
+- The worker is single-threaded by design: one gate, one job at a time.
+  That's the point.
+
+## License
+
+MIT.

serialq-0.1.0/src/serialq.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+pyproject.toml
+src/serialq/__init__.py
+src/serialq/__main__.py
+src/serialq.egg-info/PKG-INFO
+src/serialq.egg-info/SOURCES.txt
+src/serialq.egg-info/dependency_links.txt
+src/serialq.egg-info/entry_points.txt
+src/serialq.egg-info/top_level.txt
+tests/test_serialq.py

serialq-0.1.0/src/serialq.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

serialq-0.1.0/src/serialq.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+serialq = serialq:main

serialq-0.1.0/src/serialq.egg-info/top_level.txt

@@ -0,0 +1 @@
+serialq

serialq-0.1.0/tests/test_serialq.py

@@ -0,0 +1,187 @@
+"""serialq tests. Run with: python -m pytest tests/ -x -q"""
+import json
+import os
+import signal
+import subprocess
+import sys
+import time
+
+import pytest
+
+BIN = [sys.executable, "-m", "serialq"]
+
+
+@pytest.fixture()
+def sq(tmp_path, monkeypatch):
+    monkeypatch.setenv("SERIALQ_DIR", str(tmp_path / "sq"))
+    monkeypatch.setenv("SERIALQ_GATE", "test")
+    env = dict(os.environ, SERIALQ_DIR=str(tmp_path / "sq"), SERIALQ_GATE="test")
+    def run(*args, **kwargs):
+        return subprocess.run(BIN + list(args), capture_output=True, text=True,
+                              env=env, timeout=kwargs.pop("timeout", 30), **kwargs)
+    run.env = env
+    run.dir = tmp_path / "sq" / "gates" / "test"
+    return run
+
+
+def jobs_of(sq):
+    with open(sq.dir / "jobs.json") as f:
+        return json.load(f)
+
+
+def test_run_passthrough(sq):
+    r = sq("run", "--", "echo", "hello")
+    assert r.returncode == 0 and r.stdout.strip() == "hello"
+
+
+def test_run_exit_code(sq):
+    r = sq("run", "--", "sh", "-c", "exit 3")
+    assert r.returncode == 3
+
+
+def test_run_missing_command(sq):
+    r = sq("run", "--", "definitely-not-a-real-binary-xyz")
+    assert r.returncode == 127
+    assert "command not found" in r.stderr
+
+
+def test_gate_serializes(sq):
+    # Two concurrent runs must never overlap: each appends start/end markers.
+    marker = sq.dir.parent.parent / "markers.txt"
+    marker.parent.mkdir(parents=True, exist_ok=True)
+    script = (
+        "import time,sys; "
+        f"open({str(marker)!r},'a').write('start\\n'); "
+        "time.sleep(0.6); "
+        f"open({str(marker)!r},'a').write('end\\n')"
+    )
+    p1 = subprocess.Popen(BIN + ["run", "--", sys.executable, "-c", script], env=sq.env)
+    time.sleep(0.2)  # let p1 take the gate
+    p2 = subprocess.Popen(BIN + ["run", "--", sys.executable, "-c", script], env=sq.env)
+    assert p1.wait(timeout=30) == 0
+    assert p2.wait(timeout=30) == 0
+    lines = marker.read_text().split()
+    assert lines == ["start", "end", "start", "end"], lines
+
+
+def test_gate_wait_timeout(sq):
+    holder = subprocess.Popen(
+        BIN + ["run", "--", sys.executable, "-c", "import time; time.sleep(5)"],
+        env=sq.env)
+    try:
+        time.sleep(0.3)
+        t0 = time.monotonic()
+        r = sq("run", "--timeout", "0.5", "--", "echo", "hi")
+        dt = time.monotonic() - t0
+        assert r.returncode != 0
+        assert "timed out" in r.stderr
+        assert dt < 4, "should not have waited for the holder"
+    finally:
+        holder.terminate()
+        holder.wait(timeout=10)
+
+
+def test_run_kill_after(sq):
+    t0 = time.monotonic()
+    r = sq("run", "--kill-after", "1", "--", "sleep", "30", timeout=30)
+    dt = time.monotonic() - t0
+    assert r.returncode != 0
+    assert "killed after" in r.stderr
+    assert dt < 10, f"took {dt:.1f}s, kill did not work"
+
+
+def test_invalid_gate_rejected(sq):
+    r = sq("run", "-g", "../evil", "--", "echo", "hi")
+    assert r.returncode != 0
+    assert "invalid gate" in r.stderr
+
+
+def test_enqueue_worker_once_fifo(sq):
+    out = sq.dir.parent.parent / "order.txt"
+    out.parent.mkdir(parents=True, exist_ok=True)
+    ids = []
+    for name in ("first", "second", "third"):
+        r = sq("enqueue", "--name", name, "--",
+               sys.executable, "-c",
+               f"open({str(out)!r},'a').write({name!r}+'\\n')")
+        assert r.returncode == 0
+        ids.append(r.stdout.strip())
+    assert len(set(ids)) == 3
+    r = sq("worker", "--once")
+    assert r.returncode == 0
+    assert out.read_text().split() == ["first", "second", "third"]
+    jobs = {k: v for k, v in jobs_of(sq).items() if k != "__meta__"}
+    assert all(j["status"] == "done" and j["exit_code"] == 0 for j in jobs.values())
+    # logs captured
+    for jid in ids:
+        assert (sq.dir / "logs" / (jid + ".log")).exists()
+
+
+def test_worker_retry_then_fail(sq):
+    r = sq("enqueue", "--retries", "2", "--", "sh", "-c", "exit 1")
+    jid = r.stdout.strip()
+    t0 = time.monotonic()
+    assert sq("worker", "--once", timeout=60).returncode == 0
+    # retries use backoff 5s, 10s -> ~15s total
+    assert time.monotonic() - t0 >= 14
+    job = jobs_of(sq)[jid]
+    assert job["status"] == "failed"
+    assert job["attempts"] == 3  # 1 initial + 2 backoff retries (elapsed >= 14s proves the waits)
+    assert job["error"] == "exit code 1"
+
+
+def test_crash_recovery(sq):
+    marker = sq.dir.parent.parent / "crash.txt"
+    marker.parent.mkdir(parents=True, exist_ok=True)
+    r = sq("enqueue", "--", sys.executable, "-c",
+           f"import time; open({str(marker)!r},'a').write('ran\\n'); time.sleep(3)")
+    jid = r.stdout.strip()
+    worker = subprocess.Popen(BIN + ["worker"], env=sq.env)
+    time.sleep(1.5)  # job is now running
+    assert jobs_of(sq)[jid]["status"] == "running"
+    worker.send_signal(signal.SIGKILL)  # simulate a crash; no cleanup runs
+    worker.wait(timeout=10)
+    assert sq("worker", "--once", timeout=30).returncode == 0
+    job = jobs_of(sq)[jid]
+    assert job["status"] == "done", job
+    assert job["attempts"] == 2, job  # ran once before the crash, once after
+    assert marker.read_text().split() == ["ran", "ran"]
+
+
+def test_second_worker_refused(sq):
+    w1 = subprocess.Popen(BIN + ["worker"], env=sq.env,
+                          stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
+    try:
+        time.sleep(0.8)
+        r = sq("worker", "--once")
+        assert r.returncode != 0
+        assert "already running" in r.stderr
+    finally:
+        w1.terminate()
+        w1.wait(timeout=10)
+
+
+def test_cancel(sq):
+    r = sq("enqueue", "--", "echo", "never")
+    jid = r.stdout.strip()
+    assert sq("cancel", jid).returncode == 0
+    assert jobs_of(sq)[jid]["status"] == "cancelled"
+    assert sq("worker", "--once").returncode == 0  # cancelled job never runs
+    assert "never" not in (sq.dir / "logs" / (jid + ".log")).read_text() \
+        if (sq.dir / "logs" / (jid + ".log")).exists() else True
+
+
+def test_list_and_status(sq):
+    r = sq("enqueue", "--name", "demo-job", "--", "echo", "x")
+    jid = r.stdout.strip()
+    out = sq("list").stdout
+    assert jid in out and "demo-job" in out and "queued" in out
+    st = sq("status").stdout
+    assert "free" in st and "queued=1" in st
+    assert "demo-job" not in sq("list", "--all").stdout or True  # --all works too
+    assert jid in sq("list", "--all").stdout
+
+
+def test_log_missing(sq):
+    r = sq("log", "nope-123")
+    assert r.returncode != 0