baqueue 1.2.0__tar.gz → 1.2.2__tar.gz

baqueue-1.2.2/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+## 1.2.2 - 2026-09-27
+
+- Made Redis ready-queue claiming crash-safe with `WATCH`/`MULTI`/`EXEC`.
+- Prevented future-dated jobs accidentally found in a ready list from becoming
+  orphaned; they are moved back to the delayed set atomically.
+- Added `RedisDriver.reconcile_pending_jobs()` and the
+  `baqueue reconcile-pending` command for repairing pending jobs missing from
+  both ready and delayed storage.
+- Kept Redis job status indexes, ready lists, and delayed membership consistent
+  during claims, stale-entry cleanup, retries, and reconciliation.
+- Preserved `attempts`, `max_attempts`, `backoff`, and per-attempt history across
+  Redis release and requeue paths.
+- Added Redis concurrency, failure-injection, delayed-job, reconciliation, and
+  index-consistency regression coverage.
+- Updated the dashboard Jobs table to show processing duration live to
+  centisecond precision.

{baqueue-1.2.0 → baqueue-1.2.2}/MANIFEST.in

@@ -1,4 +1,5 @@
 include README.md
+include CHANGELOG.md
 include LICENSE
 include pyproject.toml
 

{baqueue-1.2.0/baqueue.egg-info → baqueue-1.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: baqueue
-Version: 1.2.0
+Version: 1.2.2
 Summary: A powerful Python queue management package inspired by Laravel Horizon
 Author: Basalam, BaQueue Contributors
 License: MIT
@@ -327,6 +327,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
 Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
 to keep connect fast on large datasets).
 
+If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
+pending jobs that are present in the pending index but missing from both the ready list
+and delayed set:
+
+```bash
+baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
+```
+
+The repair is safe to repeat. Each job is handled in its own Redis transaction: future
+jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
+and missing hashes lose stale queue/index references.
+
 ### Retry Failed Jobs
 
 Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
@@ -552,6 +564,8 @@ Coverage includes:
 - Batch builder + completion callbacks
 - DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
 - CLI command surface (help text, validation, `retry-failed` abort flow)
+- Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
+  and secondary-index consistency
 
 ## CLI Commands
 
@@ -562,6 +576,7 @@ baqueue dashboard     Launch the monitoring dashboard
 baqueue prune         Prune old jobs
 baqueue retry-failed  Retry all failed jobs (filter by queue/tag/age)
 baqueue reconcile-indexes  Repair Redis secondary indexes (drop stale entries)
+baqueue reconcile-pending  Repair pending jobs missing from ready/delayed storage
 baqueue status        Show queue status
 baqueue test          Run the test suite
 ```

{baqueue-1.2.0 → baqueue-1.2.2}/README.md

@@ -275,6 +275,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
 Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
 to keep connect fast on large datasets).
 
+If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
+pending jobs that are present in the pending index but missing from both the ready list
+and delayed set:
+
+```bash
+baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
+```
+
+The repair is safe to repeat. Each job is handled in its own Redis transaction: future
+jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
+and missing hashes lose stale queue/index references.
+
 ### Retry Failed Jobs
 
 Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
@@ -500,6 +512,8 @@ Coverage includes:
 - Batch builder + completion callbacks
 - DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
 - CLI command surface (help text, validation, `retry-failed` abort flow)
+- Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
+  and secondary-index consistency
 
 ## CLI Commands
 
@@ -510,6 +524,7 @@ baqueue dashboard     Launch the monitoring dashboard
 baqueue prune         Prune old jobs
 baqueue retry-failed  Retry all failed jobs (filter by queue/tag/age)
 baqueue reconcile-indexes  Repair Redis secondary indexes (drop stale entries)
+baqueue reconcile-pending  Repair pending jobs missing from ready/delayed storage
 baqueue status        Show queue status
 baqueue test          Run the test suite
 ```

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/__init__.py

@@ -7,7 +7,7 @@ from baqueue.batch import Batch
 from baqueue.events import EventBus
 from baqueue.retry import BackoffStrategy
 
-__version__ = "1.2.0"
+__version__ = "1.2.2"
 
 __all__ = [
     "BaQueueConfig",

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/cli.py

@@ -425,6 +425,35 @@ async def _run_reconcile_indexes(config: BaQueueConfig, batch: int) -> int:
         await Queue.disconnect()
 
 
+@cli.command(name="reconcile-pending")
+@click.option("--batch", default=500, type=int, help="Pending jobs scanned per batch.")
+@click.option("--driver", "-d", default="redis", help="Driver name (sqlite, memory, redis, postgres).")
+@click.option("--driver-url", default=None, help="Driver connection URL.")
+@click.pass_context
+def reconcile_pending(
+    ctx: click.Context,
+    batch: int,
+    driver: str,
+    driver_url: str | None,
+) -> None:
+    """Repair pending jobs missing from both ready and delayed storage."""
+    _validate_driver(driver)
+    config: BaQueueConfig = ctx.obj["config"]
+    config.driver = DriverConfig(name=driver, url=driver_url or "")
+
+    repaired = _run_async(_run_reconcile_pending, config, batch)
+    click.echo(f"Reconciled {repaired or 0} pending job(s).")
+
+
+async def _run_reconcile_pending(config: BaQueueConfig, batch: int) -> int:
+    Queue.configure(config)
+    await Queue.connect()
+    try:
+        return await Queue.get_driver().reconcile_pending_jobs(batch=batch)
+    finally:
+        await Queue.disconnect()
+
+
 @cli.command()
 @click.option("--driver", "-d", default="sqlite", help="Driver name (sqlite, memory, redis, postgres).")
 @click.option("--driver-url", default=None, help="Driver connection URL.")

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/dashboard/static/app.js

@@ -25,6 +25,8 @@ document.addEventListener("alpine:init", () => {
     dateTo: "",
 
     refreshTimer: null,
+    durationTimer: null,
+    liveNow: Date.now(),
 
     get pageTitle() {
       const titles = { overview: "Overview", jobs: "Jobs", queues: "Queues", workers: "Workers" };
@@ -37,6 +39,17 @@ document.addEventListener("alpine:init", () => {
       this.fetchSupervisors();
       this.fetchRecentJobs();
       this.connectWS();
+      this.durationTimer = setInterval(() => {
+        if (
+          this.tab === "jobs" &&
+          this.jobs.some((job) => this.inFlightAttempt(job))
+        ) {
+          const now = Date.now();
+          if (Math.floor(now / 10) !== Math.floor(this.liveNow / 10)) {
+            this.liveNow = now;
+          }
+        }
+      }, 10);
       this.refreshTimer = setInterval(() => {
         if (this.tab === "overview") {
           this.fetchOverview();
@@ -49,6 +62,7 @@ document.addEventListener("alpine:init", () => {
 
     destroy() {
       if (this.refreshTimer) clearInterval(this.refreshTimer);
+      if (this.durationTimer) clearInterval(this.durationTimer);
       if (this.ws) this.ws.close();
       this.disconnectJobsWS();
     },
@@ -437,12 +451,26 @@ document.addEventListener("alpine:init", () => {
 
     jobDuration(job) {
       if (!job.started_at) return "-";
-      const end = job.completed_at || job.failed_at || (Date.now() / 1000);
-      const diff = end - job.started_at;
-      if (diff < 0.001) return "<1ms";
-      if (diff < 1) return Math.round(diff * 1000) + "ms";
-      if (diff < 60) return diff.toFixed(1) + "s";
-      return Math.floor(diff / 60) + "m " + Math.floor(diff % 60) + "s";
+      const end = job.completed_at || job.failed_at || (this.liveNow / 1000);
+      return this.formatDuration(end - job.started_at);
+    },
+
+    formatDuration(diff) {
+      if (!Number.isFinite(diff) || diff < 0) return "-";
+
+      // Work in centiseconds so a live duration never displays more precision
+      // than the requested hundredth of a second.
+      const centiseconds = Math.floor(diff * 100);
+      const totalSeconds = centiseconds / 100;
+      if (totalSeconds < 60) return totalSeconds.toFixed(2) + "s";
+
+      const minutes = Math.floor(totalSeconds / 60);
+      const seconds = (totalSeconds % 60).toFixed(2).padStart(5, "0");
+      if (minutes < 60) return minutes + "m " + seconds + "s";
+
+      const hours = Math.floor(minutes / 60);
+      const remainingMinutes = String(minutes % 60).padStart(2, "0");
+      return hours + "h " + remainingMinutes + "m " + seconds + "s";
     },
 
     // ── Per-attempt timeline ────────────────────────────────

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/dashboard/static/index.html

@@ -314,7 +314,7 @@
                                             </div>
                                         </td>
                                         <td class="cell-time" x-text="formatTime(job.created_at)"></td>
-                                        <td class="cell-time" x-text="jobDuration(job)"></td>
+                                        <td class="cell-time cell-duration" x-text="jobDuration(job)"></td>
                                         <td>
                                             <button class="btn-action" @click.stop="viewJob(job.id)">
                                                 <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" width="16" height="16"><polyline points="9 18 15 12 9 6"/></svg>

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/dashboard/static/style.css

@@ -732,7 +732,14 @@ body {
 
 .cell-id { font-size: 12px; }
 .cell-class { font-weight: 600; max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
-.cell-time { font-size: 12px; color: var(--text-muted); white-space: nowrap; }
+.cell-time {
+    font-size: 12px;
+    color: var(--text-muted);
+    white-space: nowrap;
+    font-variant-numeric: tabular-nums;
+}
+
+.cell-duration { min-width: 88px; }
 
 .cell-attempts { font-size: 13px; }
 

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/drivers/base.py

@@ -260,6 +260,14 @@ class BaseDriver(ABC):
         number of stale index entries removed."""
         return 0
 
+    async def reconcile_pending_jobs(self, batch: int = 500) -> int:
+        """Repair pending jobs that are absent from their runnable storage.
+
+        No-op for drivers without a separate ready/delayed index model. Redis
+        overrides this with an atomic per-job repair pass.
+        """
+        return 0
+
     @abstractmethod
     async def flush(self, queue: str | None = None) -> None:
         """Remove all jobs (optionally for a specific queue)."""

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/drivers/redis_driver.py

@@ -15,6 +15,7 @@ logger = logging.getLogger("baqueue.redis")
 # the job hash is gone, so we can't read its status — we ZREM from every global
 # status index to be sure the stale id is cleared.
 _ALL_STATUSES = ("pending", "processing", "completed", "failed", "cancelled")
+_MAX_STALE_POP_ATTEMPTS = 100
 
 
 class RedisDriver(BaseDriver):
@@ -161,6 +162,13 @@ class RedisDriver(BaseDriver):
         pipe.zadd(self._idx_status(new_status), {job_id: score})
         pipe.zadd(self._idx_queue_status(queue, new_status), {job_id: score})
 
+    def _index_rebuild(self, pipe: Any, job: JobPayload) -> None:
+        """Make every status index reflect the current payload in one transaction."""
+        for status in _ALL_STATUSES:
+            pipe.zrem(self._idx_status(status), job.id)
+            pipe.zrem(self._idx_queue_status(job.queue, status), job.id)
+        self._index_add(pipe, job)
+
     # ── Push / Pop ──────────────────────────────────────────────
 
     async def push(self, payload: JobPayload) -> str:
@@ -202,28 +210,136 @@ class RedisDriver(BaseDriver):
         await self._with_disk_full_recovery(_do)
         return ids
 
+    async def _pop_and_claim_pending_job(
+        self, queue: str,
+    ) -> tuple[bool, JobPayload | None]:
+        from redis.exceptions import WatchError
+
+        queue_key = self._key("queue", queue)
+        delayed_key = self._key("delayed")
+
+        async def _attempt() -> tuple[bool, JobPayload | None]:
+            pipe = self._redis.pipeline()
+            try:
+                await pipe.watch(queue_key)
+                job_id = await pipe.lindex(queue_key, 0)
+                if not job_id:
+                    return False, None
+
+                job_key = self._key("job", job_id)
+                await pipe.watch(job_key, delayed_key)
+                raw = await pipe.hget(job_key, "data")
+                if not raw:
+                    pipe.multi()
+                    pipe.lrem(queue_key, 0, job_id)
+                    pipe.zrem(delayed_key, job_id)
+                    self._index_remove_orphan(pipe, job_id, queue, None)
+                    await pipe.execute()
+                    return True, None
+
+                payload = JobPayload.from_json(raw)
+                if payload.status != "pending":
+                    pipe.multi()
+                    pipe.lrem(queue_key, 0, job_id)
+                    pipe.zrem(delayed_key, job_id)
+                    self._index_rebuild(pipe, payload)
+                    await pipe.execute()
+                    return True, None
+
+                now = _now_ts()
+                if payload.delay_until is not None and payload.delay_until > now:
+                    pipe.multi()
+                    # A future-dated job found in the ready list is corrupt
+                    # state. Move it back to delayed atomically instead of
+                    # dropping the only runnable reference.
+                    pipe.lrem(queue_key, 0, job_id)
+                    pipe.zadd(delayed_key, {job_id: payload.delay_until})
+                    self._index_rebuild(pipe, payload)
+                    await pipe.execute()
+                    return True, None
+
+                payload.status = "processing"
+                payload.started_at = now
+                payload.updated_at = now
+                payload.attempts += 1
+
+                pipe.multi()
+                # The list entry is removed in the same EXEC that persists the
+                # processing claim. A connection failure before EXEC leaves it
+                # available; an ambiguous post-EXEC failure leaves a processing
+                # job for the existing stuck-job recovery path.
+                pipe.lrem(queue_key, 0, job_id)
+                pipe.zrem(delayed_key, job_id)
+                pipe.hset(job_key, mapping={"data": payload.to_json()})
+                self._index_rebuild(pipe, payload)
+                await pipe.execute()
+                return True, payload
+            finally:
+                await pipe.reset()
+
+        for _ in range(5):
+            try:
+                return await self._with_disk_full_recovery(_attempt)
+            except WatchError:
+                continue
+        return False, None
+
     async def pop(self, queue: str) -> JobPayload | None:
-        job_id = await self._redis.lpop(self._key("queue", queue))
-        if not job_id:
-            return None
-        raw = await self._redis.hget(self._key("job", job_id), "data")
-        if not raw:
-            return None
-        payload = JobPayload.from_json(raw)
-        old_status = payload.status
-        payload.status = "processing"
-        now = _now_ts()
-        payload.started_at = now
-        payload.updated_at = now
-        payload.attempts += 1
+        for _ in range(_MAX_STALE_POP_ATTEMPTS):
+            found, payload = await self._pop_and_claim_pending_job(queue)
+            if not found:
+                return None
+            if payload is not None:
+                return payload
+        return None
+
+    async def _promote_delayed_job(self, job_id: str, now: float) -> JobPayload | None:
+        from redis.exceptions import WatchError
 
-        async def _do():
+        job_key = self._key("job", job_id)
+
+        async def _attempt() -> JobPayload | None:
             pipe = self._redis.pipeline()
-            pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
-            self._index_status_change(pipe, payload.id, payload.queue, old_status, payload.status, payload.created_at)
-            await pipe.execute()
-        await self._with_disk_full_recovery(_do)
-        return payload
+            try:
+                await pipe.watch(job_key)
+                raw = await pipe.hget(job_key, "data")
+                if not raw:
+                    pipe.multi()
+                    pipe.zrem(self._key("delayed"), job_id)
+                    await pipe.execute()
+                    return None
+
+                payload = JobPayload.from_json(raw)
+                if payload.status != "pending" or payload.delay_until is None:
+                    pipe.multi()
+                    pipe.zrem(self._key("delayed"), job_id)
+                    await pipe.execute()
+                    return None
+                if payload.delay_until > now:
+                    pipe.multi()
+                    pipe.zadd(self._key("delayed"), {job_id: payload.delay_until})
+                    await pipe.execute()
+                    return None
+
+                payload.delay_until = None
+                payload.updated_at = now
+
+                pipe.multi()
+                pipe.hset(job_key, mapping={"data": payload.to_json()})
+                pipe.zrem(self._key("delayed"), job_id)
+                pipe.lrem(self._key("queue", payload.queue), 0, job_id)
+                pipe.rpush(self._key("queue", payload.queue), job_id)
+                await pipe.execute()
+                return payload
+            finally:
+                await pipe.reset()
+
+        for _ in range(5):
+            try:
+                return await self._with_disk_full_recovery(_attempt)
+            except WatchError:
+                continue
+        return None
 
     async def pop_delayed(self) -> list[JobPayload]:
         now = _now_ts()
@@ -231,27 +347,10 @@ class RedisDriver(BaseDriver):
         if not job_ids:
             return []
 
-        async def _remove_delayed():
-            pipe = self._redis.pipeline()
-            for job_id in job_ids:
-                pipe.zrem(self._key("delayed"), job_id)
-            await pipe.execute()
-        await self._with_disk_full_recovery(_remove_delayed)
-
         moved: list[JobPayload] = []
         for job_id in job_ids:
-            raw = await self._redis.hget(self._key("job", job_id), "data")
-            if raw:
-                payload = JobPayload.from_json(raw)
-                payload.delay_until = None
-                payload.updated_at = _now_ts()
-
-                async def _move(payload=payload):
-                    pipe = self._redis.pipeline()
-                    pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
-                    pipe.rpush(self._key("queue", payload.queue), payload.id)
-                    await pipe.execute()
-                await self._with_disk_full_recovery(_move)
+            payload = await self._promote_delayed_job(job_id, now)
+            if payload is not None:
                 moved.append(payload)
         return moved
 
@@ -282,16 +381,20 @@ class RedisDriver(BaseDriver):
         old_status = payload.status
         payload.status = "pending"
         payload.updated_at = _now_ts()
+        delayed_key = self._key("delayed")
 
         async def _do():
             pipe = self._redis.pipeline()
+            pipe.lrem(self._key("queue", payload.queue), 0, payload.id)
             if delay > 0:
                 payload.delay_until = _now_ts() + delay
                 pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
-                pipe.zadd(self._key("delayed"), {payload.id: payload.delay_until})
+                pipe.zadd(delayed_key, {payload.id: payload.delay_until})
             else:
+                payload.delay_until = None
                 pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
                 pipe.rpush(self._key("queue", payload.queue), payload.id)
+                pipe.zrem(delayed_key, payload.id)
             self._index_status_change(pipe, payload.id, payload.queue, old_status, "pending", payload.created_at)
             await pipe.execute()
         await self._with_disk_full_recovery(_do)
@@ -402,8 +505,8 @@ class RedisDriver(BaseDriver):
         now = _now_ts()
         # Only a job actually sitting in the delayed ZSET needs to be moved into
         # its ready list. A pending job that is already ready (delay_until None or
-        # in the past) must NOT be re-pushed, or Redis pop — which does not
-        # re-check status — would process it twice.
+        # in the past) must NOT be re-pushed, or it would leave duplicate ready
+        # entries that every worker has to discard later.
         was_scheduled = payload.delay_until is not None and payload.delay_until > now
         payload.delay_until = None
         payload.updated_at = now
@@ -847,6 +950,143 @@ class RedisDriver(BaseDriver):
                     break
         return removed
 
+    async def _scan_keys(self, pattern: str, batch: int) -> list[str]:
+        keys: list[str] = []
+        cursor: Any = "0"
+        while True:
+            cursor, found = await self._redis.scan(cursor=cursor, match=pattern, count=batch)
+            keys.extend(found)
+            if cursor == "0" or cursor == 0:
+                break
+        return keys
+
+    async def _reconcile_pending_job(
+        self,
+        job_id: str,
+        ready_queue_keys: list[str],
+        index_keys: list[str],
+    ) -> int:
+        """Repair one pending-index member without exposing an intermediate state."""
+        from redis.exceptions import WatchError
+
+        job_key = self._key("job", job_id)
+        delayed_key = self._key("delayed")
+
+        async def _attempt() -> int:
+            pipe = self._redis.pipeline()
+            try:
+                watch_keys = [job_key, delayed_key, *ready_queue_keys]
+                await pipe.watch(*watch_keys)
+                raw = await pipe.hget(job_key, "data")
+
+                payload: JobPayload | None = None
+                if raw:
+                    try:
+                        payload = JobPayload.from_json(raw)
+                    except (TypeError, ValueError, json.JSONDecodeError):
+                        # Keep malformed data for manual inspection, but remove
+                        # its runnable/index references so workers do not spin on it.
+                        payload = None
+
+                if payload is None:
+                    pipe.multi()
+                    for queue_key in ready_queue_keys:
+                        pipe.lrem(queue_key, 0, job_id)
+                    pipe.zrem(delayed_key, job_id)
+                    for index_key in index_keys:
+                        pipe.zrem(index_key, job_id)
+                    await pipe.execute()
+                    return 1
+
+                queue_key = self._key("queue", payload.queue)
+                if queue_key not in watch_keys:
+                    await pipe.watch(queue_key)
+
+                ready_positions_raw = await pipe.lpos(queue_key, job_id, count=2)
+                delayed_score = await pipe.zscore(delayed_key, job_id)
+                if isinstance(ready_positions_raw, (list, tuple)):
+                    ready_positions = list(ready_positions_raw)
+                elif ready_positions_raw is None:
+                    ready_positions = []
+                else:
+                    ready_positions = [ready_positions_raw]
+
+                pipe.multi()
+                if payload.status != "pending":
+                    pipe.lrem(queue_key, 0, job_id)
+                    pipe.zrem(delayed_key, job_id)
+                    self._index_rebuild(pipe, payload)
+                elif payload.delay_until is not None and payload.delay_until > _now_ts():
+                    pipe.lrem(queue_key, 0, job_id)
+                    pipe.zadd(delayed_key, {job_id: payload.delay_until})
+                    self._index_rebuild(pipe, payload)
+                else:
+                    if payload.delay_until is not None:
+                        payload.delay_until = None
+                        payload.updated_at = _now_ts()
+                        pipe.hset(job_key, mapping={"data": payload.to_json()})
+                    if len(ready_positions) != 1 or delayed_score is not None:
+                        pipe.lrem(queue_key, 0, job_id)
+                        pipe.rpush(queue_key, job_id)
+                    pipe.zrem(delayed_key, job_id)
+                    self._index_rebuild(pipe, payload)
+                await pipe.execute()
+                return 1
+            finally:
+                await pipe.reset()
+
+        for _ in range(5):
+            try:
+                return int(await self._with_disk_full_recovery(_attempt) or 0)
+            except WatchError:
+                continue
+        return 0
+
+    async def reconcile_pending_jobs(self, batch: int = 500) -> int:
+        """Repair pending jobs that are absent from both ready and delayed storage.
+
+        The pending status index is the work source. Each job is repaired in its
+        own WATCH/MULTI/EXEC transaction: missing or malformed hashes lose stale
+        queue/index references, future jobs are restored to the delayed ZSET, and
+        runnable jobs are normalized to exactly one ready-list entry. The method
+        is safe to run repeatedly and concurrent workers can win a race without
+        duplicating or claiming a job.
+        """
+        batch = max(1, batch)
+        ready_queue_keys = await self._scan_keys(self._key("queue", "*"), batch)
+        index_keys = await self._scan_keys(self._key("jobs", "*"), batch)
+        pending_index = self._idx_status("pending")
+
+        repaired = 0
+        processed_ids: set[str] = set()
+        while True:
+            discovered = 0
+            cursor: Any = 0
+            while True:
+                cursor, members = await self._redis.zscan(
+                    pending_index, cursor=cursor, count=batch,
+                )
+                job_ids = [
+                    member[0] if isinstance(member, (tuple, list)) else member
+                    for member in members
+                ]
+                for job_id in job_ids:
+                    if job_id in processed_ids:
+                        continue
+                    processed_ids.add(job_id)
+                    discovered += 1
+                    repaired += await self._reconcile_pending_job(
+                        job_id, ready_queue_keys, index_keys,
+                    )
+                if cursor == 0 or cursor == "0":
+                    break
+            # Redis permits mutation while ZSCAN is in progress. Removing stale
+            # members can make a later member invisible to the current cursor, so
+            # take another pass until a complete scan discovers nothing new.
+            if discovered == 0:
+                break
+        return repaired
+
     async def prune_metrics(self, older_than_seconds: float) -> int:
         cutoff = _now_ts() - older_than_seconds
         removed = 0

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue/worker.py

@@ -44,6 +44,7 @@ class Worker:
         self._current_job: JobPayload | None = None
         self._jobs_processed = 0
         self._jobs_failed = 0
+        self._queue_cursor = 0
 
     @property
     def is_running(self) -> bool:
@@ -84,10 +85,18 @@ class Worker:
         self._running = False
 
     async def _fetch_next(self) -> JobPayload | None:
-        for queue in self.queues:
-            job = await self.driver.pop(queue)
+        queue_count = len(self.queues)
+        if queue_count == 0:
+            return None
+
+        for offset in range(queue_count):
+            index = (self._queue_cursor + offset) % queue_count
+            job = await self.driver.pop(self.queues[index])
             if job:
+                self._queue_cursor = (index + 1) % queue_count
                 return job
+
+        self._queue_cursor = (self._queue_cursor + 1) % queue_count
         return None
 
     @staticmethod

{baqueue-1.2.0 → baqueue-1.2.2/baqueue.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: baqueue
-Version: 1.2.0
+Version: 1.2.2
 Summary: A powerful Python queue management package inspired by Laravel Horizon
 Author: Basalam, BaQueue Contributors
 License: MIT
@@ -327,6 +327,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
 Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
 to keep connect fast on large datasets).
 
+If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
+pending jobs that are present in the pending index but missing from both the ready list
+and delayed set:
+
+```bash
+baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
+```
+
+The repair is safe to repeat. Each job is handled in its own Redis transaction: future
+jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
+and missing hashes lose stale queue/index references.
+
 ### Retry Failed Jobs
 
 Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
@@ -552,6 +564,8 @@ Coverage includes:
 - Batch builder + completion callbacks
 - DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
 - CLI command surface (help text, validation, `retry-failed` abort flow)
+- Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
+  and secondary-index consistency
 
 ## CLI Commands
 
@@ -562,6 +576,7 @@ baqueue dashboard     Launch the monitoring dashboard
 baqueue prune         Prune old jobs
 baqueue retry-failed  Retry all failed jobs (filter by queue/tag/age)
 baqueue reconcile-indexes  Repair Redis secondary indexes (drop stale entries)
+baqueue reconcile-pending  Repair pending jobs missing from ready/delayed storage
 baqueue status        Show queue status
 baqueue test          Run the test suite
 ```

{baqueue-1.2.0 → baqueue-1.2.2}/baqueue.egg-info/SOURCES.txt

@@ -1,3 +1,4 @@
+CHANGELOG.md
 LICENSE
 MANIFEST.in
 README.md