baqueue 1.2.1__tar.gz → 1.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (39) hide show
  1. baqueue-1.2.2/CHANGELOG.md +18 -0
  2. {baqueue-1.2.1 → baqueue-1.2.2}/MANIFEST.in +1 -0
  3. {baqueue-1.2.1/baqueue.egg-info → baqueue-1.2.2}/PKG-INFO +16 -1
  4. {baqueue-1.2.1 → baqueue-1.2.2}/README.md +15 -0
  5. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/__init__.py +1 -1
  6. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/cli.py +29 -0
  7. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/static/app.js +34 -6
  8. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/static/index.html +1 -1
  9. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/static/style.css +8 -1
  10. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/base.py +8 -0
  11. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/redis_driver.py +196 -23
  12. {baqueue-1.2.1 → baqueue-1.2.2/baqueue.egg-info}/PKG-INFO +16 -1
  13. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/SOURCES.txt +1 -0
  14. {baqueue-1.2.1 → baqueue-1.2.2}/LICENSE +0 -0
  15. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/balancer.py +0 -0
  16. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/batch.py +0 -0
  17. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/config.py +0 -0
  18. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/__init__.py +0 -0
  19. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/api.py +0 -0
  20. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/server.py +0 -0
  21. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/__init__.py +0 -0
  22. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/memory_driver.py +0 -0
  23. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/postgres_driver.py +0 -0
  24. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/sqlite_driver.py +0 -0
  25. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/events.py +0 -0
  26. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/job.py +0 -0
  27. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/pruner.py +0 -0
  28. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/queue.py +0 -0
  29. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/retry.py +0 -0
  30. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/scheduler.py +0 -0
  31. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/serializer.py +0 -0
  32. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/supervisor.py +0 -0
  33. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/worker.py +0 -0
  34. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/dependency_links.txt +0 -0
  35. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/entry_points.txt +0 -0
  36. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/requires.txt +0 -0
  37. {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/top_level.txt +0 -0
  38. {baqueue-1.2.1 → baqueue-1.2.2}/pyproject.toml +0 -0
  39. {baqueue-1.2.1 → baqueue-1.2.2}/setup.cfg +0 -0
@@ -0,0 +1,18 @@
1
+ # Changelog
2
+
3
+ ## 1.2.2 - 2026-09-27
4
+
5
+ - Made Redis ready-queue claiming crash-safe with `WATCH`/`MULTI`/`EXEC`.
6
+ - Prevented future-dated jobs accidentally found in a ready list from becoming
7
+ orphaned; they are moved back to the delayed set atomically.
8
+ - Added `RedisDriver.reconcile_pending_jobs()` and the
9
+ `baqueue reconcile-pending` command for repairing pending jobs missing from
10
+ both ready and delayed storage.
11
+ - Kept Redis job status indexes, ready lists, and delayed membership consistent
12
+ during claims, stale-entry cleanup, retries, and reconciliation.
13
+ - Preserved `attempts`, `max_attempts`, `backoff`, and per-attempt history across
14
+ Redis release and requeue paths.
15
+ - Added Redis concurrency, failure-injection, delayed-job, reconciliation, and
16
+ index-consistency regression coverage.
17
+ - Updated the dashboard Jobs table to show processing duration live to
18
+ centisecond precision.
@@ -1,4 +1,5 @@
1
1
  include README.md
2
+ include CHANGELOG.md
2
3
  include LICENSE
3
4
  include pyproject.toml
4
5
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: baqueue
3
- Version: 1.2.1
3
+ Version: 1.2.2
4
4
  Summary: A powerful Python queue management package inspired by Laravel Horizon
5
5
  Author: Basalam, BaQueue Contributors
6
6
  License: MIT
@@ -327,6 +327,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
327
327
  Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
328
328
  to keep connect fast on large datasets).
329
329
 
330
+ If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
331
+ pending jobs that are present in the pending index but missing from both the ready list
332
+ and delayed set:
333
+
334
+ ```bash
335
+ baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
336
+ ```
337
+
338
+ The repair is safe to repeat. Each job is handled in its own Redis transaction: future
339
+ jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
340
+ and missing hashes lose stale queue/index references.
341
+
330
342
  ### Retry Failed Jobs
331
343
 
332
344
  Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
@@ -552,6 +564,8 @@ Coverage includes:
552
564
  - Batch builder + completion callbacks
553
565
  - DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
554
566
  - CLI command surface (help text, validation, `retry-failed` abort flow)
567
+ - Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
568
+ and secondary-index consistency
555
569
 
556
570
  ## CLI Commands
557
571
 
@@ -562,6 +576,7 @@ baqueue dashboard Launch the monitoring dashboard
562
576
  baqueue prune Prune old jobs
563
577
  baqueue retry-failed Retry all failed jobs (filter by queue/tag/age)
564
578
  baqueue reconcile-indexes Repair Redis secondary indexes (drop stale entries)
579
+ baqueue reconcile-pending Repair pending jobs missing from ready/delayed storage
565
580
  baqueue status Show queue status
566
581
  baqueue test Run the test suite
567
582
  ```
@@ -275,6 +275,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
275
275
  Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
276
276
  to keep connect fast on large datasets).
277
277
 
278
+ If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
279
+ pending jobs that are present in the pending index but missing from both the ready list
280
+ and delayed set:
281
+
282
+ ```bash
283
+ baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
284
+ ```
285
+
286
+ The repair is safe to repeat. Each job is handled in its own Redis transaction: future
287
+ jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
288
+ and missing hashes lose stale queue/index references.
289
+
278
290
  ### Retry Failed Jobs
279
291
 
280
292
  Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
@@ -500,6 +512,8 @@ Coverage includes:
500
512
  - Batch builder + completion callbacks
501
513
  - DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
502
514
  - CLI command surface (help text, validation, `retry-failed` abort flow)
515
+ - Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
516
+ and secondary-index consistency
503
517
 
504
518
  ## CLI Commands
505
519
 
@@ -510,6 +524,7 @@ baqueue dashboard Launch the monitoring dashboard
510
524
  baqueue prune Prune old jobs
511
525
  baqueue retry-failed Retry all failed jobs (filter by queue/tag/age)
512
526
  baqueue reconcile-indexes Repair Redis secondary indexes (drop stale entries)
527
+ baqueue reconcile-pending Repair pending jobs missing from ready/delayed storage
513
528
  baqueue status Show queue status
514
529
  baqueue test Run the test suite
515
530
  ```
@@ -7,7 +7,7 @@ from baqueue.batch import Batch
7
7
  from baqueue.events import EventBus
8
8
  from baqueue.retry import BackoffStrategy
9
9
 
10
- __version__ = "1.2.1"
10
+ __version__ = "1.2.2"
11
11
 
12
12
  __all__ = [
13
13
  "BaQueueConfig",
@@ -425,6 +425,35 @@ async def _run_reconcile_indexes(config: BaQueueConfig, batch: int) -> int:
425
425
  await Queue.disconnect()
426
426
 
427
427
 
428
+ @cli.command(name="reconcile-pending")
429
+ @click.option("--batch", default=500, type=int, help="Pending jobs scanned per batch.")
430
+ @click.option("--driver", "-d", default="redis", help="Driver name (sqlite, memory, redis, postgres).")
431
+ @click.option("--driver-url", default=None, help="Driver connection URL.")
432
+ @click.pass_context
433
+ def reconcile_pending(
434
+ ctx: click.Context,
435
+ batch: int,
436
+ driver: str,
437
+ driver_url: str | None,
438
+ ) -> None:
439
+ """Repair pending jobs missing from both ready and delayed storage."""
440
+ _validate_driver(driver)
441
+ config: BaQueueConfig = ctx.obj["config"]
442
+ config.driver = DriverConfig(name=driver, url=driver_url or "")
443
+
444
+ repaired = _run_async(_run_reconcile_pending, config, batch)
445
+ click.echo(f"Reconciled {repaired or 0} pending job(s).")
446
+
447
+
448
+ async def _run_reconcile_pending(config: BaQueueConfig, batch: int) -> int:
449
+ Queue.configure(config)
450
+ await Queue.connect()
451
+ try:
452
+ return await Queue.get_driver().reconcile_pending_jobs(batch=batch)
453
+ finally:
454
+ await Queue.disconnect()
455
+
456
+
428
457
  @cli.command()
429
458
  @click.option("--driver", "-d", default="sqlite", help="Driver name (sqlite, memory, redis, postgres).")
430
459
  @click.option("--driver-url", default=None, help="Driver connection URL.")
@@ -25,6 +25,8 @@ document.addEventListener("alpine:init", () => {
25
25
  dateTo: "",
26
26
 
27
27
  refreshTimer: null,
28
+ durationTimer: null,
29
+ liveNow: Date.now(),
28
30
 
29
31
  get pageTitle() {
30
32
  const titles = { overview: "Overview", jobs: "Jobs", queues: "Queues", workers: "Workers" };
@@ -37,6 +39,17 @@ document.addEventListener("alpine:init", () => {
37
39
  this.fetchSupervisors();
38
40
  this.fetchRecentJobs();
39
41
  this.connectWS();
42
+ this.durationTimer = setInterval(() => {
43
+ if (
44
+ this.tab === "jobs" &&
45
+ this.jobs.some((job) => this.inFlightAttempt(job))
46
+ ) {
47
+ const now = Date.now();
48
+ if (Math.floor(now / 10) !== Math.floor(this.liveNow / 10)) {
49
+ this.liveNow = now;
50
+ }
51
+ }
52
+ }, 10);
40
53
  this.refreshTimer = setInterval(() => {
41
54
  if (this.tab === "overview") {
42
55
  this.fetchOverview();
@@ -49,6 +62,7 @@ document.addEventListener("alpine:init", () => {
49
62
 
50
63
  destroy() {
51
64
  if (this.refreshTimer) clearInterval(this.refreshTimer);
65
+ if (this.durationTimer) clearInterval(this.durationTimer);
52
66
  if (this.ws) this.ws.close();
53
67
  this.disconnectJobsWS();
54
68
  },
@@ -437,12 +451,26 @@ document.addEventListener("alpine:init", () => {
437
451
 
438
452
  jobDuration(job) {
439
453
  if (!job.started_at) return "-";
440
- const end = job.completed_at || job.failed_at || (Date.now() / 1000);
441
- const diff = end - job.started_at;
442
- if (diff < 0.001) return "<1ms";
443
- if (diff < 1) return Math.round(diff * 1000) + "ms";
444
- if (diff < 60) return diff.toFixed(1) + "s";
445
- return Math.floor(diff / 60) + "m " + Math.floor(diff % 60) + "s";
454
+ const end = job.completed_at || job.failed_at || (this.liveNow / 1000);
455
+ return this.formatDuration(end - job.started_at);
456
+ },
457
+
458
+ formatDuration(diff) {
459
+ if (!Number.isFinite(diff) || diff < 0) return "-";
460
+
461
+ // Work in centiseconds so a live duration never displays more precision
462
+ // than the requested hundredth of a second.
463
+ const centiseconds = Math.floor(diff * 100);
464
+ const totalSeconds = centiseconds / 100;
465
+ if (totalSeconds < 60) return totalSeconds.toFixed(2) + "s";
466
+
467
+ const minutes = Math.floor(totalSeconds / 60);
468
+ const seconds = (totalSeconds % 60).toFixed(2).padStart(5, "0");
469
+ if (minutes < 60) return minutes + "m " + seconds + "s";
470
+
471
+ const hours = Math.floor(minutes / 60);
472
+ const remainingMinutes = String(minutes % 60).padStart(2, "0");
473
+ return hours + "h " + remainingMinutes + "m " + seconds + "s";
446
474
  },
447
475
 
448
476
  // ── Per-attempt timeline ────────────────────────────────
@@ -314,7 +314,7 @@
314
314
  </div>
315
315
  </td>
316
316
  <td class="cell-time" x-text="formatTime(job.created_at)"></td>
317
- <td class="cell-time" x-text="jobDuration(job)"></td>
317
+ <td class="cell-time cell-duration" x-text="jobDuration(job)"></td>
318
318
  <td>
319
319
  <button class="btn-action" @click.stop="viewJob(job.id)">
320
320
  <svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" width="16" height="16"><polyline points="9 18 15 12 9 6"/></svg>
@@ -732,7 +732,14 @@ body {
732
732
 
733
733
  .cell-id { font-size: 12px; }
734
734
  .cell-class { font-weight: 600; max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
735
- .cell-time { font-size: 12px; color: var(--text-muted); white-space: nowrap; }
735
+ .cell-time {
736
+ font-size: 12px;
737
+ color: var(--text-muted);
738
+ white-space: nowrap;
739
+ font-variant-numeric: tabular-nums;
740
+ }
741
+
742
+ .cell-duration { min-width: 88px; }
736
743
 
737
744
  .cell-attempts { font-size: 13px; }
738
745
 
@@ -260,6 +260,14 @@ class BaseDriver(ABC):
260
260
  number of stale index entries removed."""
261
261
  return 0
262
262
 
263
+ async def reconcile_pending_jobs(self, batch: int = 500) -> int:
264
+ """Repair pending jobs that are absent from their runnable storage.
265
+
266
+ No-op for drivers without a separate ready/delayed index model. Redis
267
+ overrides this with an atomic per-job repair pass.
268
+ """
269
+ return 0
270
+
263
271
  @abstractmethod
264
272
  async def flush(self, queue: str | None = None) -> None:
265
273
  """Remove all jobs (optionally for a specific queue)."""
@@ -162,6 +162,13 @@ class RedisDriver(BaseDriver):
162
162
  pipe.zadd(self._idx_status(new_status), {job_id: score})
163
163
  pipe.zadd(self._idx_queue_status(queue, new_status), {job_id: score})
164
164
 
165
+ def _index_rebuild(self, pipe: Any, job: JobPayload) -> None:
166
+ """Make every status index reflect the current payload in one transaction."""
167
+ for status in _ALL_STATUSES:
168
+ pipe.zrem(self._idx_status(status), job.id)
169
+ pipe.zrem(self._idx_queue_status(job.queue, status), job.id)
170
+ self._index_add(pipe, job)
171
+
165
172
  # ── Push / Pop ──────────────────────────────────────────────
166
173
 
167
174
  async def push(self, payload: JobPayload) -> str:
@@ -203,43 +210,70 @@ class RedisDriver(BaseDriver):
203
210
  await self._with_disk_full_recovery(_do)
204
211
  return ids
205
212
 
206
- async def _claim_pending_job(self, job_id: str) -> JobPayload | None:
213
+ async def _pop_and_claim_pending_job(
214
+ self, queue: str,
215
+ ) -> tuple[bool, JobPayload | None]:
207
216
  from redis.exceptions import WatchError
208
217
 
209
- job_key = self._key("job", job_id)
218
+ queue_key = self._key("queue", queue)
219
+ delayed_key = self._key("delayed")
210
220
 
211
- async def _attempt() -> JobPayload | None:
221
+ async def _attempt() -> tuple[bool, JobPayload | None]:
212
222
  pipe = self._redis.pipeline()
213
223
  try:
214
- await pipe.watch(job_key)
224
+ await pipe.watch(queue_key)
225
+ job_id = await pipe.lindex(queue_key, 0)
226
+ if not job_id:
227
+ return False, None
228
+
229
+ job_key = self._key("job", job_id)
230
+ await pipe.watch(job_key, delayed_key)
215
231
  raw = await pipe.hget(job_key, "data")
216
232
  if not raw:
217
- return None
233
+ pipe.multi()
234
+ pipe.lrem(queue_key, 0, job_id)
235
+ pipe.zrem(delayed_key, job_id)
236
+ self._index_remove_orphan(pipe, job_id, queue, None)
237
+ await pipe.execute()
238
+ return True, None
218
239
 
219
240
  payload = JobPayload.from_json(raw)
220
241
  if payload.status != "pending":
221
- return None
222
- if payload.delay_until is not None and payload.delay_until > _now_ts():
223
- return None
242
+ pipe.multi()
243
+ pipe.lrem(queue_key, 0, job_id)
244
+ pipe.zrem(delayed_key, job_id)
245
+ self._index_rebuild(pipe, payload)
246
+ await pipe.execute()
247
+ return True, None
224
248
 
225
- payload.status = "processing"
226
249
  now = _now_ts()
250
+ if payload.delay_until is not None and payload.delay_until > now:
251
+ pipe.multi()
252
+ # A future-dated job found in the ready list is corrupt
253
+ # state. Move it back to delayed atomically instead of
254
+ # dropping the only runnable reference.
255
+ pipe.lrem(queue_key, 0, job_id)
256
+ pipe.zadd(delayed_key, {job_id: payload.delay_until})
257
+ self._index_rebuild(pipe, payload)
258
+ await pipe.execute()
259
+ return True, None
260
+
261
+ payload.status = "processing"
227
262
  payload.started_at = now
228
263
  payload.updated_at = now
229
264
  payload.attempts += 1
230
265
 
231
266
  pipe.multi()
267
+ # The list entry is removed in the same EXEC that persists the
268
+ # processing claim. A connection failure before EXEC leaves it
269
+ # available; an ambiguous post-EXEC failure leaves a processing
270
+ # job for the existing stuck-job recovery path.
271
+ pipe.lrem(queue_key, 0, job_id)
272
+ pipe.zrem(delayed_key, job_id)
232
273
  pipe.hset(job_key, mapping={"data": payload.to_json()})
233
- self._index_status_change(
234
- pipe,
235
- payload.id,
236
- payload.queue,
237
- "pending",
238
- "processing",
239
- payload.created_at,
240
- )
274
+ self._index_rebuild(pipe, payload)
241
275
  await pipe.execute()
242
- return payload
276
+ return True, payload
243
277
  finally:
244
278
  await pipe.reset()
245
279
 
@@ -248,14 +282,13 @@ class RedisDriver(BaseDriver):
248
282
  return await self._with_disk_full_recovery(_attempt)
249
283
  except WatchError:
250
284
  continue
251
- return None
285
+ return False, None
252
286
 
253
287
  async def pop(self, queue: str) -> JobPayload | None:
254
288
  for _ in range(_MAX_STALE_POP_ATTEMPTS):
255
- job_id = await self._redis.lpop(self._key("queue", queue))
256
- if not job_id:
289
+ found, payload = await self._pop_and_claim_pending_job(queue)
290
+ if not found:
257
291
  return None
258
- payload = await self._claim_pending_job(job_id)
259
292
  if payload is not None:
260
293
  return payload
261
294
  return None
@@ -348,6 +381,7 @@ class RedisDriver(BaseDriver):
348
381
  old_status = payload.status
349
382
  payload.status = "pending"
350
383
  payload.updated_at = _now_ts()
384
+ delayed_key = self._key("delayed")
351
385
 
352
386
  async def _do():
353
387
  pipe = self._redis.pipeline()
@@ -355,10 +389,12 @@ class RedisDriver(BaseDriver):
355
389
  if delay > 0:
356
390
  payload.delay_until = _now_ts() + delay
357
391
  pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
358
- pipe.zadd(self._key("delayed"), {payload.id: payload.delay_until})
392
+ pipe.zadd(delayed_key, {payload.id: payload.delay_until})
359
393
  else:
394
+ payload.delay_until = None
360
395
  pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
361
396
  pipe.rpush(self._key("queue", payload.queue), payload.id)
397
+ pipe.zrem(delayed_key, payload.id)
362
398
  self._index_status_change(pipe, payload.id, payload.queue, old_status, "pending", payload.created_at)
363
399
  await pipe.execute()
364
400
  await self._with_disk_full_recovery(_do)
@@ -914,6 +950,143 @@ class RedisDriver(BaseDriver):
914
950
  break
915
951
  return removed
916
952
 
953
+ async def _scan_keys(self, pattern: str, batch: int) -> list[str]:
954
+ keys: list[str] = []
955
+ cursor: Any = "0"
956
+ while True:
957
+ cursor, found = await self._redis.scan(cursor=cursor, match=pattern, count=batch)
958
+ keys.extend(found)
959
+ if cursor == "0" or cursor == 0:
960
+ break
961
+ return keys
962
+
963
+ async def _reconcile_pending_job(
964
+ self,
965
+ job_id: str,
966
+ ready_queue_keys: list[str],
967
+ index_keys: list[str],
968
+ ) -> int:
969
+ """Repair one pending-index member without exposing an intermediate state."""
970
+ from redis.exceptions import WatchError
971
+
972
+ job_key = self._key("job", job_id)
973
+ delayed_key = self._key("delayed")
974
+
975
+ async def _attempt() -> int:
976
+ pipe = self._redis.pipeline()
977
+ try:
978
+ watch_keys = [job_key, delayed_key, *ready_queue_keys]
979
+ await pipe.watch(*watch_keys)
980
+ raw = await pipe.hget(job_key, "data")
981
+
982
+ payload: JobPayload | None = None
983
+ if raw:
984
+ try:
985
+ payload = JobPayload.from_json(raw)
986
+ except (TypeError, ValueError, json.JSONDecodeError):
987
+ # Keep malformed data for manual inspection, but remove
988
+ # its runnable/index references so workers do not spin on it.
989
+ payload = None
990
+
991
+ if payload is None:
992
+ pipe.multi()
993
+ for queue_key in ready_queue_keys:
994
+ pipe.lrem(queue_key, 0, job_id)
995
+ pipe.zrem(delayed_key, job_id)
996
+ for index_key in index_keys:
997
+ pipe.zrem(index_key, job_id)
998
+ await pipe.execute()
999
+ return 1
1000
+
1001
+ queue_key = self._key("queue", payload.queue)
1002
+ if queue_key not in watch_keys:
1003
+ await pipe.watch(queue_key)
1004
+
1005
+ ready_positions_raw = await pipe.lpos(queue_key, job_id, count=2)
1006
+ delayed_score = await pipe.zscore(delayed_key, job_id)
1007
+ if isinstance(ready_positions_raw, (list, tuple)):
1008
+ ready_positions = list(ready_positions_raw)
1009
+ elif ready_positions_raw is None:
1010
+ ready_positions = []
1011
+ else:
1012
+ ready_positions = [ready_positions_raw]
1013
+
1014
+ pipe.multi()
1015
+ if payload.status != "pending":
1016
+ pipe.lrem(queue_key, 0, job_id)
1017
+ pipe.zrem(delayed_key, job_id)
1018
+ self._index_rebuild(pipe, payload)
1019
+ elif payload.delay_until is not None and payload.delay_until > _now_ts():
1020
+ pipe.lrem(queue_key, 0, job_id)
1021
+ pipe.zadd(delayed_key, {job_id: payload.delay_until})
1022
+ self._index_rebuild(pipe, payload)
1023
+ else:
1024
+ if payload.delay_until is not None:
1025
+ payload.delay_until = None
1026
+ payload.updated_at = _now_ts()
1027
+ pipe.hset(job_key, mapping={"data": payload.to_json()})
1028
+ if len(ready_positions) != 1 or delayed_score is not None:
1029
+ pipe.lrem(queue_key, 0, job_id)
1030
+ pipe.rpush(queue_key, job_id)
1031
+ pipe.zrem(delayed_key, job_id)
1032
+ self._index_rebuild(pipe, payload)
1033
+ await pipe.execute()
1034
+ return 1
1035
+ finally:
1036
+ await pipe.reset()
1037
+
1038
+ for _ in range(5):
1039
+ try:
1040
+ return int(await self._with_disk_full_recovery(_attempt) or 0)
1041
+ except WatchError:
1042
+ continue
1043
+ return 0
1044
+
1045
+ async def reconcile_pending_jobs(self, batch: int = 500) -> int:
1046
+ """Repair pending jobs that are absent from both ready and delayed storage.
1047
+
1048
+ The pending status index is the work source. Each job is repaired in its
1049
+ own WATCH/MULTI/EXEC transaction: missing or malformed hashes lose stale
1050
+ queue/index references, future jobs are restored to the delayed ZSET, and
1051
+ runnable jobs are normalized to exactly one ready-list entry. The method
1052
+ is safe to run repeatedly and concurrent workers can win a race without
1053
+ duplicating or claiming a job.
1054
+ """
1055
+ batch = max(1, batch)
1056
+ ready_queue_keys = await self._scan_keys(self._key("queue", "*"), batch)
1057
+ index_keys = await self._scan_keys(self._key("jobs", "*"), batch)
1058
+ pending_index = self._idx_status("pending")
1059
+
1060
+ repaired = 0
1061
+ processed_ids: set[str] = set()
1062
+ while True:
1063
+ discovered = 0
1064
+ cursor: Any = 0
1065
+ while True:
1066
+ cursor, members = await self._redis.zscan(
1067
+ pending_index, cursor=cursor, count=batch,
1068
+ )
1069
+ job_ids = [
1070
+ member[0] if isinstance(member, (tuple, list)) else member
1071
+ for member in members
1072
+ ]
1073
+ for job_id in job_ids:
1074
+ if job_id in processed_ids:
1075
+ continue
1076
+ processed_ids.add(job_id)
1077
+ discovered += 1
1078
+ repaired += await self._reconcile_pending_job(
1079
+ job_id, ready_queue_keys, index_keys,
1080
+ )
1081
+ if cursor == 0 or cursor == "0":
1082
+ break
1083
+ # Redis permits mutation while ZSCAN is in progress. Removing stale
1084
+ # members can make a later member invisible to the current cursor, so
1085
+ # take another pass until a complete scan discovers nothing new.
1086
+ if discovered == 0:
1087
+ break
1088
+ return repaired
1089
+
917
1090
  async def prune_metrics(self, older_than_seconds: float) -> int:
918
1091
  cutoff = _now_ts() - older_than_seconds
919
1092
  removed = 0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: baqueue
3
- Version: 1.2.1
3
+ Version: 1.2.2
4
4
  Summary: A powerful Python queue management package inspired by Laravel Horizon
5
5
  Author: Basalam, BaQueue Contributors
6
6
  License: MIT
@@ -327,6 +327,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
327
327
  Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
328
328
  to keep connect fast on large datasets).
329
329
 
330
+ If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
331
+ pending jobs that are present in the pending index but missing from both the ready list
332
+ and delayed set:
333
+
334
+ ```bash
335
+ baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
336
+ ```
337
+
338
+ The repair is safe to repeat. Each job is handled in its own Redis transaction: future
339
+ jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
340
+ and missing hashes lose stale queue/index references.
341
+
330
342
  ### Retry Failed Jobs
331
343
 
332
344
  Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
@@ -552,6 +564,8 @@ Coverage includes:
552
564
  - Batch builder + completion callbacks
553
565
  - DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
554
566
  - CLI command surface (help text, validation, `retry-failed` abort flow)
567
+ - Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
568
+ and secondary-index consistency
555
569
 
556
570
  ## CLI Commands
557
571
 
@@ -562,6 +576,7 @@ baqueue dashboard Launch the monitoring dashboard
562
576
  baqueue prune Prune old jobs
563
577
  baqueue retry-failed Retry all failed jobs (filter by queue/tag/age)
564
578
  baqueue reconcile-indexes Repair Redis secondary indexes (drop stale entries)
579
+ baqueue reconcile-pending Repair pending jobs missing from ready/delayed storage
565
580
  baqueue status Show queue status
566
581
  baqueue test Run the test suite
567
582
  ```
@@ -1,3 +1,4 @@
1
+ CHANGELOG.md
1
2
  LICENSE
2
3
  MANIFEST.in
3
4
  README.md
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes