baqueue 1.1.0__tar.gz → 1.2.0__tar.gz

{baqueue-1.1.0/baqueue.egg-info → baqueue-1.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: baqueue
-Version: 1.1.0
+Version: 1.2.0
 Summary: A powerful Python queue management package inspired by Laravel Horizon
 Author: Basalam, BaQueue Contributors
 License: MIT
@@ -75,6 +75,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
   - [Dispatch Jobs](#dispatch-jobs)
   - [Batch Jobs](#batch-jobs)
   - [Run Workers](#run-workers)
+    - [Stuck Job Recovery](#stuck-job-recovery)
   - [Pruning](#pruning)
     - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)
     - [Manual pruning](#manual-pruning)
@@ -95,6 +96,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
 ## Features
 - **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory
 - **Auto-balancing**: Dynamically scale workers based on queue pressure
+- **Stuck-job recovery**: Jobs left in `processing` for more than 1 hour are requeued automatically
 - **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable
 - **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once
 - **Pruning**: Remove old jobs by status, tag, or age
@@ -213,6 +215,41 @@ Or via CLI:
 baqueue work -q emails -q payments -w 3 -b auto
 ```
 
+#### Stuck Job Recovery
+
+When `baqueue work` is running, the supervisor also checks for jobs that were
+claimed by a worker but never finished. By default, any job that has stayed in
+`processing` for more than 1 hour is moved back to `pending`, so another worker
+can pick it up and run it again.
+
+This is intended for worker crashes, process restarts, or other cases where a
+job was left in-flight. The original claim still counts as an attempt; when the
+job is picked up again, its attempt counter continues from there.
+
+Configure it from Python:
+
+```python
+supervisor = Supervisor(
+    driver=Queue.get_driver(),
+    config=SupervisorConfig(
+        queues=["emails"],
+        recover_stuck_jobs=True,
+        stuck_processing_seconds=3600,
+        stuck_check_interval_seconds=60,
+    ),
+)
+```
+
+Or from the CLI:
+
+```bash
+baqueue work --stuck-job-timeout-seconds 7200
+baqueue work --no-stuck-job-recovery
+```
+
+If you intentionally run jobs for longer than 1 hour, increase
+`stuck_processing_seconds` above your longest expected runtime.
+
 ### Pruning
 
 #### Auto-pruning (runs alongside `baqueue work`)
@@ -509,7 +546,7 @@ Coverage includes:
 - `Queue` facade — push / later / bulk / prune / `retry_failed`
 - Cross-driver contract tests (memory + sqlite, parameterized)
 - Worker lifecycle: success / failure / retry / timeout
-- Supervisor pool + delayed-job promotion
+- Supervisor pool + delayed-job promotion + stuck-job recovery
 - Scheduler interval dispatch
 - Pruner by status / tag / age
 - Batch builder + completion callbacks

{baqueue-1.1.0 → baqueue-1.2.0}/README.md

@@ -23,6 +23,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
   - [Dispatch Jobs](#dispatch-jobs)
   - [Batch Jobs](#batch-jobs)
   - [Run Workers](#run-workers)
+    - [Stuck Job Recovery](#stuck-job-recovery)
   - [Pruning](#pruning)
     - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)
     - [Manual pruning](#manual-pruning)
@@ -43,6 +44,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
 ## Features
 - **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory
 - **Auto-balancing**: Dynamically scale workers based on queue pressure
+- **Stuck-job recovery**: Jobs left in `processing` for more than 1 hour are requeued automatically
 - **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable
 - **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once
 - **Pruning**: Remove old jobs by status, tag, or age
@@ -161,6 +163,41 @@ Or via CLI:
 baqueue work -q emails -q payments -w 3 -b auto
 ```
 
+#### Stuck Job Recovery
+
+When `baqueue work` is running, the supervisor also checks for jobs that were
+claimed by a worker but never finished. By default, any job that has stayed in
+`processing` for more than 1 hour is moved back to `pending`, so another worker
+can pick it up and run it again.
+
+This is intended for worker crashes, process restarts, or other cases where a
+job was left in-flight. The original claim still counts as an attempt; when the
+job is picked up again, its attempt counter continues from there.
+
+Configure it from Python:
+
+```python
+supervisor = Supervisor(
+    driver=Queue.get_driver(),
+    config=SupervisorConfig(
+        queues=["emails"],
+        recover_stuck_jobs=True,
+        stuck_processing_seconds=3600,
+        stuck_check_interval_seconds=60,
+    ),
+)
+```
+
+Or from the CLI:
+
+```bash
+baqueue work --stuck-job-timeout-seconds 7200
+baqueue work --no-stuck-job-recovery
+```
+
+If you intentionally run jobs for longer than 1 hour, increase
+`stuck_processing_seconds` above your longest expected runtime.
+
 ### Pruning
 
 #### Auto-pruning (runs alongside `baqueue work`)
@@ -457,7 +494,7 @@ Coverage includes:
 - `Queue` facade — push / later / bulk / prune / `retry_failed`
 - Cross-driver contract tests (memory + sqlite, parameterized)
 - Worker lifecycle: success / failure / retry / timeout
-- Supervisor pool + delayed-job promotion
+- Supervisor pool + delayed-job promotion + stuck-job recovery
 - Scheduler interval dispatch
 - Pruner by status / tag / age
 - Batch builder + completion callbacks

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/__init__.py

@@ -7,7 +7,7 @@ from baqueue.batch import Batch
 from baqueue.events import EventBus
 from baqueue.retry import BackoffStrategy
 
-__version__ = "1.1.0"
+__version__ = "1.2.0"
 
 __all__ = [
     "BaQueueConfig",

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/cli.py

@@ -103,6 +103,18 @@ def cli(ctx: click.Context, config: str | None, verbose: bool) -> None:
     "--no-disk-full-cleanup", is_flag=True,
     help="Disable automatic emergency cleanup when the driver returns a disk-full error.",
 )
+@click.option(
+    "--no-stuck-job-recovery", is_flag=True,
+    help="Disable automatic recovery of jobs stuck in processing.",
+)
+@click.option(
+    "--stuck-job-timeout-seconds", type=int, default=None,
+    help="Requeue processing jobs older than N seconds (default 3600).",
+)
+@click.option(
+    "--stuck-job-check-interval-seconds", type=int, default=None,
+    help="How often stuck-job recovery runs, in seconds (default 60).",
+)
 @click.pass_context
 def work(
     ctx: click.Context,
@@ -119,6 +131,9 @@ def work(
     prune_other_seconds: int | None,
     prune_interval_seconds: int | None,
     no_disk_full_cleanup: bool,
+    no_stuck_job_recovery: bool,
+    stuck_job_timeout_seconds: int | None,
+    stuck_job_check_interval_seconds: int | None,
 ) -> None:
     """Start processing jobs."""
     config: BaQueueConfig = ctx.obj["config"]
@@ -143,6 +158,13 @@ def work(
         sleep=sleep,
         timeout=timeout,
         max_jobs_per_worker=max_jobs,
+        recover_stuck_jobs=not no_stuck_job_recovery,
+        stuck_processing_seconds=(
+            3600 if stuck_job_timeout_seconds is None else stuck_job_timeout_seconds
+        ),
+        stuck_check_interval_seconds=(
+            60 if stuck_job_check_interval_seconds is None else stuck_job_check_interval_seconds
+        ),
     )
 
     _validate_driver(driver)
@@ -162,6 +184,13 @@ def work(
     click.echo(
         f"  Disk-full cleanup: {'enabled' if config.auto_cleanup_on_disk_full else 'disabled'}"
     )
+    if supervisor_config.recover_stuck_jobs:
+        click.echo(
+            f"  Stuck-job recovery: processing>{supervisor_config.stuck_processing_seconds}s, "
+            f"every {supervisor_config.stuck_check_interval_seconds}s"
+        )
+    else:
+        click.echo("  Stuck-job recovery: disabled")
     click.echo()
 
     _run_async(_run_worker, config, supervisor_config)

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/config.py

@@ -28,6 +28,9 @@ class SupervisorConfig(BaseModel):
     sleep: float = 1.0  # seconds to sleep when queue is empty
     timeout: int = 60  # max job execution time in seconds
     memory_limit: int = 128  # MB
+    recover_stuck_jobs: bool = True
+    stuck_processing_seconds: int = 3600
+    stuck_check_interval_seconds: int = 60
 
 
 class ScheduleEntry(BaseModel):

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/drivers/base.py

@@ -112,6 +112,15 @@ class BaseDriver(ABC):
         """Release a job back onto the queue (for retries)."""
         ...
 
+    @abstractmethod
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        """Move stale processing jobs back to pending. Returns count requeued."""
+        ...
+
     @abstractmethod
     async def delete(self, job_id: str) -> None: ...
 

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/drivers/memory_driver.py

@@ -107,6 +107,32 @@ class MemoryDriver(BaseDriver):
                 self._queues[payload.queue].append(payload.id)
             self._jobs[payload.id] = payload
 
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        cutoff = _now_ts() - older_than_seconds
+        now = _now_ts()
+        count = 0
+        async with self._lock:
+            for payload in self._jobs.values():
+                if queue and payload.queue != queue:
+                    continue
+                if payload.status != "processing":
+                    continue
+                started = payload.started_at or payload.updated_at
+                if started is None or started > cutoff:
+                    continue
+                payload.status = "pending"
+                payload.started_at = None
+                payload.delay_until = None
+                payload.updated_at = now
+                if payload.id not in self._queues[payload.queue]:
+                    self._queues[payload.queue].append(payload.id)
+                count += 1
+        return count
+
     async def delete(self, job_id: str) -> None:
         async with self._lock:
             self._jobs.pop(job_id, None)

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/drivers/postgres_driver.py

@@ -317,6 +317,33 @@ class PostgresDriver(BaseDriver):
 
         await self._with_disk_full_recovery(_do)
 
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        now = _now_ts()
+        cutoff = now - older_than_seconds
+        conditions = ["status='processing'", "COALESCE(started_at, updated_at) <= $2"]
+        params: list[Any] = [now, cutoff]
+        idx = 3
+        if queue:
+            conditions.append(f"queue=${idx}")
+            params.append(queue)
+        where = " AND ".join(conditions)
+
+        async def _do():
+            async with self._pool.acquire() as conn:
+                return await conn.execute(
+                    f"""UPDATE {self._jobs_table}
+                        SET status='pending', started_at=NULL, delay_until=NULL, updated_at=$1
+                        WHERE {where}""",
+                    *params,
+                )
+
+        result = await self._with_disk_full_recovery(_do)
+        return int(result.split()[-1])
+
     async def delete(self, job_id: str) -> None:
         async def _do():
             async with self._pool.acquire() as conn:

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/drivers/redis_driver.py

@@ -296,6 +296,88 @@ class RedisDriver(BaseDriver):
             await pipe.execute()
         await self._with_disk_full_recovery(_do)
 
+    async def _requeue_stuck_job(self, job_id: str, cutoff: float) -> int:
+        from redis.exceptions import WatchError
+
+        job_key = self._key("job", job_id)
+
+        async def _attempt() -> int:
+            pipe = self._redis.pipeline()
+            try:
+                await pipe.watch(job_key)
+                raw = await pipe.hget(job_key, "data")
+                if not raw:
+                    return 0
+
+                payload = JobPayload.from_json(raw)
+                if payload.status != "processing":
+                    return 0
+                started = payload.started_at or payload.updated_at
+                if started is None or started > cutoff:
+                    return 0
+
+                payload.status = "pending"
+                payload.started_at = None
+                payload.delay_until = None
+                payload.updated_at = _now_ts()
+
+                pipe.multi()
+                pipe.hset(job_key, mapping={"data": payload.to_json()})
+                pipe.zrem(self._key("delayed"), job_id)
+                pipe.lrem(self._key("queue", payload.queue), 0, job_id)
+                pipe.rpush(self._key("queue", payload.queue), job_id)
+                self._index_status_change(
+                    pipe,
+                    job_id,
+                    payload.queue,
+                    "processing",
+                    "pending",
+                    payload.created_at,
+                )
+                await pipe.execute()
+                return 1
+            finally:
+                await pipe.reset()
+
+        for _ in range(5):
+            try:
+                return int(await self._with_disk_full_recovery(_attempt) or 0)
+            except WatchError:
+                continue
+        return 0
+
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        cutoff = _now_ts() - older_than_seconds
+        index = self._index_key(queue, "processing")
+        ids = await self._redis.zrange(index, 0, -1)
+        if not ids:
+            return 0
+
+        pipe = self._redis.pipeline()
+        for jid in ids:
+            pipe.hget(self._key("job", jid), "data")
+        raws = await pipe.execute()
+
+        count = 0
+        for jid, raw in zip(ids, raws):
+            if not raw:
+                continue
+            try:
+                job = JobPayload.from_json(raw)
+            except (TypeError, ValueError, json.JSONDecodeError):
+                continue
+            if queue and job.queue != queue:
+                continue
+            started = job.started_at or job.updated_at
+            if started is None or started > cutoff:
+                continue
+            count += await self._requeue_stuck_job(jid, cutoff)
+        return count
+
     async def delete(self, job_id: str) -> None:
         raw = await self._redis.hget(self._key("job", job_id), "data")
 

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/drivers/sqlite_driver.py

@@ -369,6 +369,36 @@ class SqliteDriver(BaseDriver):
                 c.commit()
             await self._execute_with_retry(_do)
 
+    async def requeue_stuck_jobs(
+        self,
+        older_than_seconds: float,
+        queue: str | None = None,
+    ) -> int:
+        now = _now_ts()
+        cutoff = now - older_than_seconds
+        conditions = ["status='processing'", "COALESCE(started_at, updated_at) <= ?"]
+        params: list[Any] = [cutoff]
+        if queue:
+            conditions.append("queue=?")
+            params.append(queue)
+        params.insert(0, now)
+        where = " AND ".join(conditions)
+        result = [0]
+
+        async with self._lock:
+            def _do():
+                c = self._get_conn()
+                cur = c.execute(
+                    f"""UPDATE jobs
+                        SET status='pending', started_at=NULL, delay_until=NULL, updated_at=?
+                        WHERE {where}""",
+                    params,
+                )
+                c.commit()
+                result[0] = cur.rowcount
+            await self._execute_with_retry(_do)
+        return result[0]
+
     async def delete(self, job_id: str) -> None:
         async with self._lock:
             def _do():

{baqueue-1.1.0 → baqueue-1.2.0}/baqueue/supervisor.py

@@ -43,6 +43,7 @@ class Supervisor:
         self._heartbeat_task: asyncio.Task | None = None
         self._balance_task: asyncio.Task | None = None
         self._pruner_task: asyncio.Task | None = None
+        self._stuck_recovery_task: asyncio.Task | None = None
 
     @property
     def is_running(self) -> bool:
@@ -80,6 +81,8 @@ class Supervisor:
         await self._report_stats()
         self._delayed_task = asyncio.create_task(self._poll_delayed())
         self._heartbeat_task = asyncio.create_task(self._heartbeat_loop())
+        if self.config.recover_stuck_jobs and self.config.stuck_processing_seconds > 0:
+            self._stuck_recovery_task = asyncio.create_task(self._recover_stuck_loop())
 
         if self.balancer:
             self._balance_task = asyncio.create_task(self._balance_loop())
@@ -112,9 +115,17 @@ class Supervisor:
             self._balance_task.cancel()
         if self._pruner_task:
             self._pruner_task.cancel()
+        if self._stuck_recovery_task:
+            self._stuck_recovery_task.cancel()
 
         aux_tasks = [
-            t for t in (self._delayed_task, self._heartbeat_task, self._balance_task, self._pruner_task)
+            t for t in (
+                self._delayed_task,
+                self._heartbeat_task,
+                self._balance_task,
+                self._pruner_task,
+                self._stuck_recovery_task,
+            )
             if t is not None
         ]
         if aux_tasks:
@@ -123,6 +134,7 @@ class Supervisor:
         self._heartbeat_task = None
         self._balance_task = None
         self._pruner_task = None
+        self._stuck_recovery_task = None
 
         for task in self._tasks:
             task.cancel()
@@ -187,6 +199,30 @@ class Supervisor:
                 logger.exception("Error in balance loop")
             await asyncio.sleep(5)
 
+    async def _recover_stuck_loop(self) -> None:
+        """Periodically requeue jobs left in processing after a worker crash."""
+        interval = max(1.0, float(self.config.stuck_check_interval_seconds))
+        while self._running:
+            try:
+                await self._recover_stuck_once()
+            except Exception:
+                logger.exception("Error recovering stuck processing jobs")
+            await asyncio.sleep(interval)
+
+    async def _recover_stuck_once(self) -> int:
+        timeout = float(self.config.stuck_processing_seconds)
+        if timeout <= 0:
+            return 0
+        total = 0
+        for queue in dict.fromkeys(self.config.queues):
+            total += await self.driver.requeue_stuck_jobs(timeout, queue=queue)
+        if total:
+            logger.warning(
+                "Requeued %d stuck processing job(s) older than %.0fs",
+                total, timeout,
+            )
+        return total
+
     async def _heartbeat_loop(self) -> None:
         while self._running:
             await self._report_stats()

{baqueue-1.1.0 → baqueue-1.2.0/baqueue.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: baqueue
-Version: 1.1.0
+Version: 1.2.0
 Summary: A powerful Python queue management package inspired by Laravel Horizon
 Author: Basalam, BaQueue Contributors
 License: MIT
@@ -75,6 +75,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
   - [Dispatch Jobs](#dispatch-jobs)
   - [Batch Jobs](#batch-jobs)
   - [Run Workers](#run-workers)
+    - [Stuck Job Recovery](#stuck-job-recovery)
   - [Pruning](#pruning)
     - [Auto-pruning](#auto-pruning-runs-alongside-baqueue-work)
     - [Manual pruning](#manual-pruning)
@@ -95,6 +96,7 @@ A powerful Python queue management package. Multi-driver support, batch jobs, sc
 ## Features
 - **Multi-driver**: SQLite (default), Redis, PostgreSQL, or In-Memory
 - **Auto-balancing**: Dynamically scale workers based on queue pressure
+- **Stuck-job recovery**: Jobs left in `processing` for more than 1 hour are requeued automatically
 - **Auto-pruning**: Completed jobs are deleted about 5 seconds after they finish; failed/cancelled jobs are kept up to 1 day — all configurable
 - **Disk-full cleanup**: Storage-full/OOM driver errors trigger emergency cleanup of terminal jobs and old metrics, then retry once
 - **Pruning**: Remove old jobs by status, tag, or age
@@ -213,6 +215,41 @@ Or via CLI:
 baqueue work -q emails -q payments -w 3 -b auto
 ```
 
+#### Stuck Job Recovery
+
+When `baqueue work` is running, the supervisor also checks for jobs that were
+claimed by a worker but never finished. By default, any job that has stayed in
+`processing` for more than 1 hour is moved back to `pending`, so another worker
+can pick it up and run it again.
+
+This is intended for worker crashes, process restarts, or other cases where a
+job was left in-flight. The original claim still counts as an attempt; when the
+job is picked up again, its attempt counter continues from there.
+
+Configure it from Python:
+
+```python
+supervisor = Supervisor(
+    driver=Queue.get_driver(),
+    config=SupervisorConfig(
+        queues=["emails"],
+        recover_stuck_jobs=True,
+        stuck_processing_seconds=3600,
+        stuck_check_interval_seconds=60,
+    ),
+)
+```
+
+Or from the CLI:
+
+```bash
+baqueue work --stuck-job-timeout-seconds 7200
+baqueue work --no-stuck-job-recovery
+```
+
+If you intentionally run jobs for longer than 1 hour, increase
+`stuck_processing_seconds` above your longest expected runtime.
+
 ### Pruning
 
 #### Auto-pruning (runs alongside `baqueue work`)
@@ -509,7 +546,7 @@ Coverage includes:
 - `Queue` facade — push / later / bulk / prune / `retry_failed`
 - Cross-driver contract tests (memory + sqlite, parameterized)
 - Worker lifecycle: success / failure / retry / timeout
-- Supervisor pool + delayed-job promotion
+- Supervisor pool + delayed-job promotion + stuck-job recovery
 - Scheduler interval dispatch
 - Pruner by status / tag / age
 - Batch builder + completion callbacks