baqueue 1.0.2__tar.gz → 1.2.0__tar.gz

{baqueue-1.0.2/baqueue.egg-info → baqueue-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: baqueue
-Version: 1.0.2
+Version: 1.2.0
 Summary: A powerful Python queue management package inspired by Laravel Horizon
 Author: Basalam, BaQueue Contributors
 License: MIT
@@ -75,6 +75,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
   - [Dispatch Jobs](#dispatch-jobs)
   - [Batch Jobs](#batch-jobs)
   - [Run Workers](#run-workers)
+    - [Stuck Job Recovery](#stuck-job-recovery)
   - [Pruning](#pruning)
     - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)
     - [Manual pruning](#manual-pruning)
@@ -95,6 +96,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
 ## Features
 - **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory
 - **Auto-balancing**: Dynamically scale workers based on queue pressure
+- **Stuck-job recovery**: Jobs left in `processing` for more than 1 hour are requeued automatically
 - **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable
 - **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once
 - **Pruning**: Remove old jobs by status, tag, or age
@@ -213,6 +215,41 @@ Or via CLI:
 baqueue work -q emails -q payments -w 3 -b auto
 ```
 
+#### Stuck Job Recovery
+
+When `baqueue work` is running, the supervisor also checks for jobs that were
+claimed by a worker but never finished. By default, any job that has stayed in
+`processing` for more than 1 hour is moved back to `pending`, so another worker
+can pick it up and run it again.
+
+This is intended for worker crashes, process restarts, or other cases where a
+job was left in-flight. The original claim still counts as an attempt; when the
+job is picked up again, its attempt counter continues from there.
+
+Configure it from Python:
+
+```python
+supervisor = Supervisor(
+    driver=Queue.get_driver(),
+    config=SupervisorConfig(
+        queues=["emails"],
+        recover_stuck_jobs=True,
+        stuck_processing_seconds=3600,
+        stuck_check_interval_seconds=60,
+    ),
+)
+```
+
+Or from the CLI:
+
+```bash
+baqueue work --stuck-job-timeout-seconds 7200
+baqueue work --no-stuck-job-recovery
+```
+
+If you intentionally run jobs for longer than 1 hour, increase
+`stuck_processing_seconds` above your longest expected runtime.
+
 ### Pruning
 
 #### Auto-pruning (runs alongside `baqueue work`)
@@ -509,7 +546,7 @@ Coverage includes:
 - `Queue` facade — push / later / bulk / prune / `retry_failed`
 - Cross-driver contract tests (memory + sqlite, parameterized)
 - Worker lifecycle: success / failure / retry / timeout
-- Supervisor pool + delayed-job promotion
+- Supervisor pool + delayed-job promotion + stuck-job recovery
 - Scheduler interval dispatch
 - Pruner by status / tag / age
 - Batch builder + completion callbacks

{baqueue-1.0.2 → baqueue-1.2.0}/README.md

@@ -23,6 +23,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
   - [Dispatch Jobs](#dispatch-jobs)
   - [Batch Jobs](#batch-jobs)
   - [Run Workers](#run-workers)
+    - [Stuck Job Recovery](#stuck-job-recovery)
   - [Pruning](#pruning)
     - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)
     - [Manual pruning](#manual-pruning)
@@ -43,6 +44,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
 ## Features
 - **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory
 - **Auto-balancing**: Dynamically scale workers based on queue pressure
+- **Stuck-job recovery**: Jobs left in `processing` for more than 1 hour are requeued automatically
 - **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable
 - **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once
 - **Pruning**: Remove old jobs by status, tag, or age
@@ -161,6 +163,41 @@ Or via CLI:
 baqueue work -q emails -q payments -w 3 -b auto
 ```
 
+#### Stuck Job Recovery
+
+When `baqueue work` is running, the supervisor also checks for jobs that were
+claimed by a worker but never finished. By default, any job that has stayed in
+`processing` for more than 1 hour is moved back to `pending`, so another worker
+can pick it up and run it again.
+
+This is intended for worker crashes, process restarts, or other cases where a
+job was left in-flight. The original claim still counts as an attempt; when the
+job is picked up again, its attempt counter continues from there.
+
+Configure it from Python:
+
+```python
+supervisor = Supervisor(
+    driver=Queue.get_driver(),
+    config=SupervisorConfig(
+        queues=["emails"],
+        recover_stuck_jobs=True,
+        stuck_processing_seconds=3600,
+        stuck_check_interval_seconds=60,
+    ),
+)
+```
+
+Or from the CLI:
+
+```bash
+baqueue work --stuck-job-timeout-seconds 7200
+baqueue work --no-stuck-job-recovery
+```
+
+If you intentionally run jobs for longer than 1 hour, increase
+`stuck_processing_seconds` above your longest expected runtime.
+
 ### Pruning
 
 #### Auto-pruning (runs alongside `baqueue work`)
@@ -457,7 +494,7 @@ Coverage includes:
 - `Queue` facade — push / later / bulk / prune / `retry_failed`
 - Cross-driver contract tests (memory + sqlite, parameterized)
 - Worker lifecycle: success / failure / retry / timeout
-- Supervisor pool + delayed-job promotion
+- Supervisor pool + delayed-job promotion + stuck-job recovery
 - Scheduler interval dispatch
 - Pruner by status / tag / age
 - Batch builder + completion callbacks

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/__init__.py

@@ -7,7 +7,7 @@ from baqueue.batch import Batch
 from baqueue.events import EventBus
 from baqueue.retry import BackoffStrategy
 
-__version__ = "1.0.2"
+__version__ = "1.2.0"
 
 __all__ = [
     "BaQueueConfig",

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/cli.py

@@ -103,6 +103,18 @@ def cli(ctx: click.Context, config: str | None, verbose: bool) -> None:
     "--no-disk-full-cleanup", is_flag=True,
     help="Disable automatic emergency cleanup when the driver returns a disk-full error.",
 )
+@click.option(
+    "--no-stuck-job-recovery", is_flag=True,
+    help="Disable automatic recovery of jobs stuck in processing.",
+)
+@click.option(
+    "--stuck-job-timeout-seconds", type=int, default=None,
+    help="Requeue processing jobs older than N seconds (default 3600).",
+)
+@click.option(
+    "--stuck-job-check-interval-seconds", type=int, default=None,
+    help="How often stuck-job recovery runs, in seconds (default 60).",
+)
 @click.pass_context
 def work(
     ctx: click.Context,
@@ -119,6 +131,9 @@ def work(
     prune_other_seconds: int | None,
     prune_interval_seconds: int | None,
     no_disk_full_cleanup: bool,
+    no_stuck_job_recovery: bool,
+    stuck_job_timeout_seconds: int | None,
+    stuck_job_check_interval_seconds: int | None,
 ) -> None:
     """Start processing jobs."""
     config: BaQueueConfig = ctx.obj["config"]
@@ -143,6 +158,13 @@ def work(
         sleep=sleep,
         timeout=timeout,
         max_jobs_per_worker=max_jobs,
+        recover_stuck_jobs=not no_stuck_job_recovery,
+        stuck_processing_seconds=(
+            3600 if stuck_job_timeout_seconds is None else stuck_job_timeout_seconds
+        ),
+        stuck_check_interval_seconds=(
+            60 if stuck_job_check_interval_seconds is None else stuck_job_check_interval_seconds
+        ),
     )
 
     _validate_driver(driver)
@@ -162,6 +184,13 @@ def work(
     click.echo(
         f"  Disk-full cleanup: {'enabled' if config.auto_cleanup_on_disk_full else 'disabled'}"
     )
+    if supervisor_config.recover_stuck_jobs:
+        click.echo(
+            f"  Stuck-job recovery: processing>{supervisor_config.stuck_processing_seconds}s, "
+            f"every {supervisor_config.stuck_check_interval_seconds}s"
+        )
+    else:
+        click.echo("  Stuck-job recovery: disabled")
     click.echo()
 
     _run_async(_run_worker, config, supervisor_config)

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/config.py

@@ -28,6 +28,9 @@ class SupervisorConfig(BaseModel):
     sleep: float = 1.0  # seconds to sleep when queue is empty
     timeout: int = 60  # max job execution time in seconds
     memory_limit: int = 128  # MB
+    recover_stuck_jobs: bool = True
+    stuck_processing_seconds: int = 3600
+    stuck_check_interval_seconds: int = 60
 
 
 class ScheduleEntry(BaseModel):

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/dashboard/api.py

@@ -117,7 +117,10 @@ class DashboardAPI:
             created_from=created_from, created_to=created_to,
         )
         return {
-            "jobs": [j.to_dict() for j in jobs],
+            # The list view never renders per-attempt history (the modal fetches
+            # job_detail for that), so omit it to keep the list and the live
+            # /ws/jobs push lean.
+            "jobs": [j.to_dict(include_history=False) for j in jobs],
             "page": page,
             "per_page": per_page,
             "count": len(jobs),
@@ -128,6 +131,10 @@ class DashboardAPI:
         job = await self.driver.get_job(job_id)
         return job.to_dict() if job else None
 
+    async def promote_job(self, job_id: str) -> bool:
+        """Make a scheduled/pending job runnable immediately. Returns True on success."""
+        return await self.driver.promote(job_id)
+
     async def retry_job(self, job_id: str) -> bool:
         job = await self.driver.get_job(job_id)
         if not job or job.status != "failed":

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/dashboard/server.py

@@ -150,6 +150,11 @@ def create_app(driver: BaseDriver, config: Optional[BaQueueConfig] = None) -> An
         ok = await api.retry_job(job_id)
         return JSONResponse({"success": ok})
 
+    @app.post("/api/jobs/{job_id}/execute")
+    async def execute_job(job_id: str):
+        ok = await api.promote_job(job_id)
+        return JSONResponse({"success": ok})
+
     @app.delete("/api/jobs/{job_id}")
     async def delete_job(job_id: str):
         ok = await api.delete_job(job_id)

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/dashboard/static/app.js

@@ -322,6 +322,14 @@ document.addEventListener("alpine:init", () => {
       this.fetchOverview();
     },
 
+    async executeJob(jobId) {
+      // Promote a scheduled/pending job so it runs immediately.
+      await fetch(`/api/jobs/${jobId}/execute`, { method: "POST" });
+      this.closeModal();
+      this.fetchJobs();
+      this.fetchOverview();
+    },
+
     async retryAllFailed() {
       const parts = [];
       if (this.jobsFilter.queue) parts.push(`queue "${this.jobsFilter.queue}"`);
@@ -437,6 +445,36 @@ document.addEventListener("alpine:init", () => {
       return Math.floor(diff / 60) + "m " + Math.floor(diff % 60) + "s";
     },
 
+    // ── Per-attempt timeline ────────────────────────────────
+
+    attemptHistory(job) {
+      return job && Array.isArray(job.history) ? job.history : [];
+    },
+
+    hasHistory(job) {
+      return this.attemptHistory(job).length > 0;
+    },
+
+    // A job currently processing has an in-flight attempt that isn't recorded in
+    // history yet (entries are appended only when an attempt concludes).
+    inFlightAttempt(job) {
+      return !!(job && job.status === "processing" && job.started_at);
+    },
+
+    attemptDotClass(entry) {
+      return entry && entry.status === "completed" ? "completed" : "failed";
+    },
+
+    attemptDuration(entry) {
+      if (!entry || !entry.started_at || !entry.finished_at) return "";
+      const diff = entry.finished_at - entry.started_at;
+      if (diff < 0) return "";
+      if (diff < 0.001) return "<1ms";
+      if (diff < 1) return Math.round(diff * 1000) + "ms";
+      if (diff < 60) return diff.toFixed(1) + "s";
+      return Math.floor(diff / 60) + "m " + Math.floor(diff % 60) + "s";
+    },
+
     shortId(id) {
       return id ? id.substring(0, 12) : "-";
     },

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/dashboard/static/index.html

@@ -511,27 +511,67 @@
                                     <span class="tl-time" x-text="formatTimeFull(selectedJob.delay_until)"></span>
                                 </div>
                             </div>
-                            <div class="tl-item" x-show="selectedJob.started_at">
-                                <div class="tl-dot processing"></div>
-                                <div class="tl-content">
-                                    <span class="tl-label">Started</span>
-                                    <span class="tl-time" x-text="formatTimeFull(selectedJob.started_at)"></span>
+                            <!-- Per-attempt history (jobs that ran at least once on a
+                                 driver that persists history). Each backoff retry is its
+                                 own entry. -->
+                            <template x-for="(entry, idx) in attemptHistory(selectedJob)" :key="idx">
+                                <div class="tl-item">
+                                    <div class="tl-dot" :class="attemptDotClass(entry)"></div>
+                                    <div class="tl-content">
+                                        <span class="tl-label">
+                                            Attempt <span x-text="entry.attempt"></span> &middot;
+                                            <span x-text="entry.status"></span>
+                                            <span class="tl-dur" x-show="attemptDuration(entry)" x-text="'(' + attemptDuration(entry) + ')'"></span>
+                                        </span>
+                                        <span class="tl-time" x-text="formatTimeFull(entry.started_at) + (entry.finished_at ? ' → ' + formatTimeFull(entry.finished_at) : '')"></span>
+                                        <span class="tl-retry" x-show="entry.will_retry">
+                                            Retry scheduled <span x-text="entry.next_retry_at ? scheduledIn(entry.next_retry_at) : ''"></span>
+                                        </span>
+                                        <pre class="tl-error" x-show="entry.error" x-text="entry.error"></pre>
+                                    </div>
                                 </div>
-                            </div>
-                            <div class="tl-item" x-show="selectedJob.completed_at">
-                                <div class="tl-dot completed"></div>
-                                <div class="tl-content">
-                                    <span class="tl-label">Completed</span>
-                                    <span class="tl-time" x-text="formatTimeFull(selectedJob.completed_at)"></span>
+                            </template>
+                            <!-- The currently-running attempt is not recorded in history
+                                 until it concludes, so surface it live. -->
+                            <template x-if="inFlightAttempt(selectedJob)">
+                                <div class="tl-item">
+                                    <div class="tl-dot processing"></div>
+                                    <div class="tl-content">
+                                        <span class="tl-label">Attempt <span x-text="selectedJob.attempts"></span> &middot; running&hellip;</span>
+                                        <span class="tl-time" x-text="formatTimeFull(selectedJob.started_at)"></span>
+                                    </div>
                                 </div>
-                            </div>
-                            <div class="tl-item" x-show="selectedJob.failed_at">
-                                <div class="tl-dot failed"></div>
-                                <div class="tl-content">
-                                    <span class="tl-label">Failed</span>
-                                    <span class="tl-time" x-text="formatTimeFull(selectedJob.failed_at)"></span>
+                            </template>
+
+                            <!-- Legacy single-attempt timeline: jobs created before history
+                                 tracking, or on drivers that don't persist history. -->
+                            <template x-if="!hasHistory(selectedJob) && !inFlightAttempt(selectedJob) && selectedJob.started_at">
+                                <div class="tl-item">
+                                    <div class="tl-dot processing"></div>
+                                    <div class="tl-content">
+                                        <span class="tl-label">Started</span>
+                                        <span class="tl-time" x-text="formatTimeFull(selectedJob.started_at)"></span>
+                                    </div>
                                 </div>
-                            </div>
+                            </template>
+                            <template x-if="!hasHistory(selectedJob) && selectedJob.completed_at">
+                                <div class="tl-item">
+                                    <div class="tl-dot completed"></div>
+                                    <div class="tl-content">
+                                        <span class="tl-label">Completed</span>
+                                        <span class="tl-time" x-text="formatTimeFull(selectedJob.completed_at)"></span>
+                                    </div>
+                                </div>
+                            </template>
+                            <template x-if="!hasHistory(selectedJob) && selectedJob.failed_at">
+                                <div class="tl-item">
+                                    <div class="tl-dot failed"></div>
+                                    <div class="tl-content">
+                                        <span class="tl-label">Failed</span>
+                                        <span class="tl-time" x-text="formatTimeFull(selectedJob.failed_at)"></span>
+                                    </div>
+                                </div>
+                            </template>
                         </div>
                     </div>
 
@@ -563,6 +603,10 @@
                 </div>
 
                 <div class="modal-actions">
+                    <button class="btn-primary" x-show="isScheduled(selectedJob)" @click="executeJob(selectedJob.id)">
+                        <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" width="16" height="16"><polygon points="5 3 19 12 5 21 5 3"/></svg>
+                        Execute Now
+                    </button>
                     <button class="btn-primary" x-show="selectedJob.status === 'failed'" @click="retryJob(selectedJob.id)">
                         <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" width="16" height="16"><polyline points="23 4 23 10 17 10"/><path d="M20.49 15a9 9 0 11-2.12-9.36L23 10"/></svg>
                         Retry Job

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/dashboard/static/style.css

@@ -1267,8 +1267,24 @@ body {
 .tl-dot.failed { border-color: var(--red); background: var(--red); }
 
 .tl-content { display: flex; flex-direction: column; gap: 1px; }
-.tl-label { font-size: 13px; font-weight: 600; }
+.tl-label { font-size: 13px; font-weight: 600; text-transform: capitalize; }
 .tl-time { font-size: 12px; color: var(--text-muted); font-family: 'JetBrains Mono', monospace; }
+.tl-dur { font-weight: 400; color: var(--text-muted); }
+.tl-retry { font-size: 12px; color: var(--amber); }
+.tl-error {
+    margin: 4px 0 0;
+    padding: 6px 8px;
+    font-size: 11px;
+    font-family: 'JetBrains Mono', monospace;
+    color: var(--red);
+    background: var(--bg-surface);
+    border: 1px solid var(--border);
+    border-radius: 6px;
+    white-space: pre-wrap;
+    word-break: break-word;
+    max-height: 140px;
+    overflow: auto;
+}
 
 /* ── Tags ───────────────────────────────────────────────── */
 

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/drivers/base.py

@@ -112,9 +112,32 @@ class BaseDriver(ABC):
         """Release a job back onto the queue (for retries)."""
         ...
 
+    @abstractmethod
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        """Move stale processing jobs back to pending. Returns count requeued."""
+        ...
+
     @abstractmethod
     async def delete(self, job_id: str) -> None: ...
 
+    async def promote(self, job_id: str) -> bool:
+        """Make a scheduled/pending job runnable immediately (clear its delay).
+
+        Returns True if the job was promoted, False if it does not exist or is not
+        in the ``pending`` state. Concrete (non-abstract) so existing third-party
+        drivers keep working; the built-in drivers override it with a race-safe,
+        index-aware version. The default relies on ``release(delay=0)`` to enqueue
+        the job for immediate processing."""
+        job = await self.get_job(job_id)
+        if job is None or job.status != "pending":
+            return False
+        await self.release(job, delay=0)
+        return True
+
     # ── Query ───────────────────────────────────────────────────
 
     @abstractmethod

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/drivers/memory_driver.py

@@ -107,6 +107,32 @@ class MemoryDriver(BaseDriver):
                 self._queues[payload.queue].append(payload.id)
             self._jobs[payload.id] = payload
 
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        cutoff = _now_ts() - older_than_seconds
+        now = _now_ts()
+        count = 0
+        async with self._lock:
+            for payload in self._jobs.values():
+                if queue and payload.queue != queue:
+                    continue
+                if payload.status != "processing":
+                    continue
+                started = payload.started_at or payload.updated_at
+                if started is None or started > cutoff:
+                    continue
+                payload.status = "pending"
+                payload.started_at = None
+                payload.delay_until = None
+                payload.updated_at = now
+                if payload.id not in self._queues[payload.queue]:
+                    self._queues[payload.queue].append(payload.id)
+                count += 1
+        return count
+
     async def delete(self, job_id: str) -> None:
         async with self._lock:
             self._jobs.pop(job_id, None)
@@ -116,6 +142,21 @@ class MemoryDriver(BaseDriver):
             if job_id in self._delayed:
                 self._delayed.remove(job_id)
 
+    async def promote(self, job_id: str) -> bool:
+        async with self._lock:
+            payload = self._jobs.get(job_id)
+            if payload is None or payload.status != "pending":
+                return False
+            payload.delay_until = None
+            payload.updated_at = _now_ts()
+            if job_id in self._delayed:
+                self._delayed.remove(job_id)
+            # Only enqueue if it isn't already ready, so promoting a non-delayed
+            # pending job can never duplicate it in the ready list.
+            if job_id not in self._queues[payload.queue]:
+                self._queues[payload.queue].append(job_id)
+            return True
+
     # ── Query ───────────────────────────────────────────────────
 
     async def get_job(self, job_id: str) -> JobPayload | None:

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/drivers/postgres_driver.py

@@ -317,6 +317,33 @@ class PostgresDriver(BaseDriver):
 
         await self._with_disk_full_recovery(_do)
 
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        now = _now_ts()
+        cutoff = now - older_than_seconds
+        conditions = ["status='processing'", "COALESCE(started_at, updated_at) <= $2"]
+        params: list[Any] = [now, cutoff]
+        idx = 3
+        if queue:
+            conditions.append(f"queue=${idx}")
+            params.append(queue)
+        where = " AND ".join(conditions)
+
+        async def _do():
+            async with self._pool.acquire() as conn:
+                return await conn.execute(
+                    f"""UPDATE {self._jobs_table}
+                        SET status='pending', started_at=NULL, delay_until=NULL, updated_at=$1
+                        WHERE {where}""",
+                    *params,
+                )
+
+        result = await self._with_disk_full_recovery(_do)
+        return int(result.split()[-1])
+
     async def delete(self, job_id: str) -> None:
         async def _do():
             async with self._pool.acquire() as conn:
@@ -324,6 +351,24 @@ class PostgresDriver(BaseDriver):
 
         await self._with_disk_full_recovery(_do)
 
+    async def promote(self, job_id: str) -> bool:
+        now = _now_ts()
+
+        async def _do():
+            async with self._pool.acquire() as conn:
+                # Clearing delay_until is enough: pop() already accepts a pending
+                # row whose delay_until IS NULL or has elapsed.
+                return await conn.fetchrow(
+                    f"""UPDATE {self._jobs_table}
+                        SET delay_until=NULL, updated_at=$1
+                        WHERE id=$2 AND status='pending'
+                        RETURNING id""",
+                    now, job_id,
+                )
+
+        row = await self._with_disk_full_recovery(_do)
+        return row is not None
+
     # ── Query ───────────────────────────────────────────────────
 
     async def get_job(self, job_id: str) -> JobPayload | None:

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/drivers/redis_driver.py

@@ -296,6 +296,88 @@ class RedisDriver(BaseDriver):
             await pipe.execute()
         await self._with_disk_full_recovery(_do)
 
+    async def _requeue_stuck_job(self, job_id: str, cutoff: float) -> int:
+        from redis.exceptions import WatchError
+
+        job_key = self._key("job", job_id)
+
+        async def _attempt() -> int:
+            pipe = self._redis.pipeline()
+            try:
+                await pipe.watch(job_key)
+                raw = await pipe.hget(job_key, "data")
+                if not raw:
+                    return 0
+
+                payload = JobPayload.from_json(raw)
+                if payload.status != "processing":
+                    return 0
+                started = payload.started_at or payload.updated_at
+                if started is None or started > cutoff:
+                    return 0
+
+                payload.status = "pending"
+                payload.started_at = None
+                payload.delay_until = None
+                payload.updated_at = _now_ts()
+
+                pipe.multi()
+                pipe.hset(job_key, mapping={"data": payload.to_json()})
+                pipe.zrem(self._key("delayed"), job_id)
+                pipe.lrem(self._key("queue", payload.queue), 0, job_id)
+                pipe.rpush(self._key("queue", payload.queue), job_id)
+                self._index_status_change(
+                    pipe,
+                    job_id,
+                    payload.queue,
+                    "processing",
+                    "pending",
+                    payload.created_at,
+                )
+                await pipe.execute()
+                return 1
+            finally:
+                await pipe.reset()
+
+        for _ in range(5):
+            try:
+                return int(await self._with_disk_full_recovery(_attempt) or 0)
+            except WatchError:
+                continue
+        return 0
+
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        cutoff = _now_ts() - older_than_seconds
+        index = self._index_key(queue, "processing")
+        ids = await self._redis.zrange(index, 0, -1)
+        if not ids:
+            return 0
+
+        pipe = self._redis.pipeline()
+        for jid in ids:
+            pipe.hget(self._key("job", jid), "data")
+        raws = await pipe.execute()
+
+        count = 0
+        for jid, raw in zip(ids, raws):
+            if not raw:
+                continue
+            try:
+                job = JobPayload.from_json(raw)
+            except (TypeError, ValueError, json.JSONDecodeError):
+                continue
+            if queue and job.queue != queue:
+                continue
+            started = job.started_at or job.updated_at
+            if started is None or started > cutoff:
+                continue
+            count += await self._requeue_stuck_job(jid, cutoff)
+        return count
+
     async def delete(self, job_id: str) -> None:
         raw = await self._redis.hget(self._key("job", job_id), "data")
 
@@ -310,6 +392,32 @@ class RedisDriver(BaseDriver):
             await pipe.execute()
         await self._with_disk_full_recovery(_do)
 
+    async def promote(self, job_id: str) -> bool:
+        raw = await self._redis.hget(self._key("job", job_id), "data")
+        if not raw:
+            return False
+        payload = JobPayload.from_json(raw)
+        if payload.status != "pending":
+            return False
+        now = _now_ts()
+        # Only a job actually sitting in the delayed ZSET needs to be moved into
+        # its ready list. A pending job that is already ready (delay_until None or
+        # in the past) must NOT be re-pushed, or Redis pop — which does not
+        # re-check status — would process it twice.
+        was_scheduled = payload.delay_until is not None and payload.delay_until > now
+        payload.delay_until = None
+        payload.updated_at = now
+
+        async def _do():
+            pipe = self._redis.pipeline()
+            pipe.hset(self._key("job", job_id), mapping={"data": payload.to_json()})
+            if was_scheduled:
+                pipe.zrem(self._key("delayed"), job_id)
+                pipe.rpush(self._key("queue", payload.queue), job_id)
+            await pipe.execute()
+        await self._with_disk_full_recovery(_do)
+        return True
+
     # ── Query ───────────────────────────────────────────────────
 
     async def get_job(self, job_id: str) -> JobPayload | None:

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/drivers/sqlite_driver.py

@@ -369,6 +369,36 @@ class SqliteDriver(BaseDriver):
                 c.commit()
             await self._execute_with_retry(_do)
 
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        now = _now_ts()
+        cutoff = now - older_than_seconds
+        conditions = ["status='processing'", "COALESCE(started_at, updated_at) <= ?"]
+        params: list[Any] = [cutoff]
+        if queue:
+            conditions.append("queue=?")
+            params.append(queue)
+        params.insert(0, now)
+        where = " AND ".join(conditions)
+        result = [0]
+
+        async with self._lock:
+            def _do():
+                c = self._get_conn()
+                cur = c.execute(
+                    f"""UPDATE jobs
+                        SET status='pending', started_at=NULL, delay_until=NULL, updated_at=?
+                        WHERE {where}""",
+                    params,
+                )
+                c.commit()
+                result[0] = cur.rowcount
+            await self._execute_with_retry(_do)
+        return result[0]
+
     async def delete(self, job_id: str) -> None:
         async with self._lock:
             def _do():
@@ -377,6 +407,23 @@ class SqliteDriver(BaseDriver):
                 c.commit()
             await self._execute_with_retry(_do)
 
+    async def promote(self, job_id: str) -> bool:
+        now = _now_ts()
+        async with self._lock:
+            result = [False]
+            def _do():
+                c = self._get_conn()
+                # Clearing delay_until is enough: pop() already accepts a pending
+                # row whose delay_until IS NULL or has elapsed.
+                cur = c.execute(
+                    "UPDATE jobs SET delay_until=NULL, updated_at=? WHERE id=? AND status='pending'",
+                    (now, job_id),
+                )
+                c.commit()
+                result[0] = cur.rowcount == 1
+            await self._execute_with_retry(_do)
+            return result[0]
+
     # ── Query ───────────────────────────────────────────────────
 
     async def get_job(self, job_id: str) -> JobPayload | None:

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/serializer.py

@@ -35,6 +35,7 @@ class JobPayload:
         "failed_at",
         "status",
         "error",
+        "history",
     )
 
     def __init__(
@@ -58,6 +59,7 @@ class JobPayload:
         failed_at: float | None = None,
         status: str = "pending",
         error: str | None = None,
+        history: list[dict[str, Any]] | None = None,
     ):
         self.id = id or uuid4().hex
         self.job_class = job_class
@@ -77,9 +79,14 @@ class JobPayload:
         self.failed_at = failed_at
         self.status = status
         self.error = error
-
-    def to_dict(self) -> dict[str, Any]:
-        return {
+        # Per-attempt execution history (one record per processing attempt).
+        # Bounded by the number of attempts; persisted only by drivers that store
+        # the full payload (memory, redis). Older payloads without this key load
+        # as an empty list, so the field is fully backward compatible.
+        self.history = history or []
+
+    def to_dict(self, *, include_history: bool = True) -> dict[str, Any]:
+        d = {
             "id": self.id,
             "job_class": self.job_class,
             "data": self.data,
@@ -99,6 +106,9 @@ class JobPayload:
             "status": self.status,
             "error": self.error,
         }
+        if include_history:
+            d["history"] = self.history
+        return d
 
     def to_json(self) -> str:
         return json.dumps(self.to_dict())

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/supervisor.py

@@ -43,6 +43,7 @@ class Supervisor:
         self._heartbeat_task: asyncio.Task | None = None
         self._balance_task: asyncio.Task | None = None
         self._pruner_task: asyncio.Task | None = None
+        self._stuck_recovery_task: asyncio.Task | None = None
 
     @property
     def is_running(self) -> bool:
@@ -80,6 +81,8 @@ class Supervisor:
         await self._report_stats()
         self._delayed_task = asyncio.create_task(self._poll_delayed())
         self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
+        if self.config.recover_stuck_jobs and self.config.stuck_processing_seconds > 0:
+            self._stuck_recovery_task = asyncio.create_task(self._recover_stuck_loop())
 
         if self.balancer:
             self._balance_task = asyncio.create_task(self._balance_loop())
@@ -112,9 +115,17 @@ class Supervisor:
             self._balance_task.cancel()
         if self._pruner_task:
             self._pruner_task.cancel()
+        if self._stuck_recovery_task:
+            self._stuck_recovery_task.cancel()
 
         aux_tasks = [
-            t for t in (self._delayed_task, self._heartbeat_task, self._balance_task, self._pruner_task)
+            t for t in (
+                self._delayed_task,
+                self._heartbeat_task,
+                self._balance_task,
+                self._pruner_task,
+                self._stuck_recovery_task,
+            )
             if t is not None
         ]
         if aux_tasks:
@@ -123,6 +134,7 @@ class Supervisor:
         self._heartbeat_task = None
         self._balance_task = None
         self._pruner_task = None
+        self._stuck_recovery_task = None
 
         for task in self._tasks:
             task.cancel()
@@ -187,6 +199,30 @@ class Supervisor:
                 logger.exception("Error in balance loop")
             await asyncio.sleep(5)
 
+    async def _recover_stuck_loop(self) -> None:
+        """Periodically requeue jobs left in processing after a worker crash."""
+        interval = max(1.0, float(self.config.stuck_check_interval_seconds))
+        while self._running:
+            try:
+                await self._recover_stuck_once()
+            except Exception:
+                logger.exception("Error recovering stuck processing jobs")
+            await asyncio.sleep(interval)
+
+    async def _recover_stuck_once(self) -> int:
+        timeout = float(self.config.stuck_processing_seconds)
+        if timeout <= 0:
+            return 0
+        total = 0
+        for queue in dict.fromkeys(self.config.queues):
+            total += await self.driver.requeue_stuck_jobs(timeout, queue=queue)
+        if total:
+            logger.warning(
+                "Requeued %d stuck processing job(s) older than %.0fs",
+                total, timeout,
+            )
+        return total
+
     async def _heartbeat_loop(self) -> None:
         while self._running:
             await self._report_stats()

{baqueue-1.0.2 → baqueue-1.2.0}/baqueue/worker.py

@@ -11,10 +11,16 @@ from baqueue.drivers.base import BaseDriver
 from baqueue.events import EventBus
 from baqueue.job import Job, FunctionJob
 from baqueue.retry import compute_delay, should_retry
-from baqueue.serializer import JobPayload, resolve_job_class
+from baqueue.serializer import JobPayload, resolve_job_class, _now_ts
 
 logger = logging.getLogger("baqueue.worker")
 
+# Per-attempt errors stored in JobPayload.history are truncated to this many
+# characters. The job's top-level `error` field keeps the full latest traceback;
+# this bound keeps the history (and therefore the stored payload) from growing
+# large across retries.
+_HISTORY_ERROR_MAXLEN = 1000
+
 
 class Worker:
     """Pulls and executes jobs from one or more queues."""
@@ -84,6 +90,33 @@ class Worker:
                 return job
         return None
 
+    @staticmethod
+    def _record_attempt(
+        payload: JobPayload,
+        *,
+        status: str,
+        finished_at: float,
+        error: str | None = None,
+        will_retry: bool = False,
+        next_retry_at: float | None = None,
+    ) -> None:
+        """Append one bounded record describing the attempt that just concluded.
+
+        Called once per attempt, right before the driver persists the new state, so
+        drivers that store the whole payload (memory, redis) keep the full history.
+        The list is bounded by the number of attempts and the error is truncated."""
+        if error is not None and len(error) > _HISTORY_ERROR_MAXLEN:
+            error = error[:_HISTORY_ERROR_MAXLEN] + "…"
+        payload.history.append({
+            "attempt": payload.attempts,
+            "started_at": payload.started_at,
+            "finished_at": finished_at,
+            "status": status,
+            "error": error,
+            "will_retry": will_retry,
+            "next_retry_at": next_retry_at,
+        })
+
     async def _process(self, payload: JobPayload) -> None:
         self._current_job = payload
         job_timeout = payload.timeout or self.timeout
@@ -99,6 +132,7 @@ class Worker:
                 timeout=job_timeout,
             )
 
+            self._record_attempt(payload, status="completed", finished_at=_now_ts())
             await self.driver.complete(payload)
             await self.driver.record_metric(payload.queue, "completed", 1)
             await self.events.emit("job.completed", payload=payload, result=result, worker=self.name)
@@ -118,9 +152,16 @@ class Worker:
 
             if should_retry(payload.attempts, payload.max_attempts):
                 delay = compute_delay(payload.backoff, payload.attempts)
+                self._record_attempt(
+                    payload, status="failed", finished_at=_now_ts(),
+                    error=error_msg, will_retry=True, next_retry_at=_now_ts() + delay,
+                )
                 await self.driver.release(payload, delay=delay)
                 await self.events.emit("job.retrying", payload=payload, error=error_msg, delay=delay)
             else:
+                self._record_attempt(
+                    payload, status="failed", finished_at=_now_ts(), error=error_msg,
+                )
                 await self.driver.fail(payload, error_msg)
                 await self.driver.record_metric(payload.queue, "failed", 1)
                 await self.events.emit("job.failed", payload=payload, error=error_msg, worker=self.name)

{baqueue-1.0.2 → baqueue-1.2.0/baqueue.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: baqueue
-Version: 1.0.2
+Version: 1.2.0
 Summary: A powerful Python queue management package inspired by Laravel Horizon
 Author: Basalam, BaQueue Contributors
 License: MIT
@@ -75,6 +75,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
   - [Dispatch Jobs](#dispatch-jobs)
   - [Batch Jobs](#batch-jobs)
   - [Run Workers](#run-workers)
+    - [Stuck Job Recovery](#stuck-job-recovery)
   - [Pruning](#pruning)
     - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)
     - [Manual pruning](#manual-pruning)
@@ -95,6 +96,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
 ## Features
 - **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory
 - **Auto-balancing**: Dynamically scale workers based on queue pressure
+- **Stuck-job recovery**: Jobs left in `processing` for more than 1 hour are requeued automatically
 - **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable
 - **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once
 - **Pruning**: Remove old jobs by status, tag, or age
@@ -213,6 +215,41 @@ Or via CLI:
 baqueue work -q emails -q payments -w 3 -b auto
 ```
 
+#### Stuck Job Recovery
+
+When `baqueue work` is running, the supervisor also checks for jobs that were
+claimed by a worker but never finished. By default, any job that has stayed in
+`processing` for more than 1 hour is moved back to `pending`, so another worker
+can pick it up and run it again.
+
+This is intended for worker crashes, process restarts, or other cases where a
+job was left in-flight. The original claim still counts as an attempt; when the
+job is picked up again, its attempt counter continues from there.
+
+Configure it from Python:
+
+```python
+supervisor = Supervisor(
+    driver=Queue.get_driver(),
+    config=SupervisorConfig(
+        queues=["emails"],
+        recover_stuck_jobs=True,
+        stuck_processing_seconds=3600,
+        stuck_check_interval_seconds=60,
+    ),
+)
+```
+
+Or from the CLI:
+
+```bash
+baqueue work --stuck-job-timeout-seconds 7200
+baqueue work --no-stuck-job-recovery
+```
+
+If you intentionally run jobs for longer than 1 hour, increase
+`stuck_processing_seconds` above your longest expected runtime.
+
 ### Pruning
 
 #### Auto-pruning (runs alongside `baqueue work`)
@@ -509,7 +546,7 @@ Coverage includes:
 - `Queue` facade — push / later / bulk / prune / `retry_failed`
 - Cross-driver contract tests (memory + sqlite, parameterized)
 - Worker lifecycle: success / failure / retry / timeout
-- Supervisor pool + delayed-job promotion
+- Supervisor pool + delayed-job promotion + stuck-job recovery
 - Scheduler interval dispatch
 - Pruner by status / tag / age
 - Batch builder + completion callbacks