baqueue 1.2.1__tar.gz → 1.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- baqueue-1.2.2/CHANGELOG.md +18 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/MANIFEST.in +1 -0
- {baqueue-1.2.1/baqueue.egg-info → baqueue-1.2.2}/PKG-INFO +16 -1
- {baqueue-1.2.1 → baqueue-1.2.2}/README.md +15 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/__init__.py +1 -1
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/cli.py +29 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/static/app.js +34 -6
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/static/index.html +1 -1
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/static/style.css +8 -1
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/base.py +8 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/redis_driver.py +196 -23
- {baqueue-1.2.1 → baqueue-1.2.2/baqueue.egg-info}/PKG-INFO +16 -1
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/SOURCES.txt +1 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/LICENSE +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/balancer.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/batch.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/config.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/__init__.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/api.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/dashboard/server.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/__init__.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/memory_driver.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/postgres_driver.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/drivers/sqlite_driver.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/events.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/job.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/pruner.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/queue.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/retry.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/scheduler.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/serializer.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/supervisor.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue/worker.py +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/dependency_links.txt +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/entry_points.txt +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/requires.txt +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/baqueue.egg-info/top_level.txt +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/pyproject.toml +0 -0
- {baqueue-1.2.1 → baqueue-1.2.2}/setup.cfg +0 -0
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 1.2.2 - 2026-09-27
|
|
4
|
+
|
|
5
|
+
- Made Redis ready-queue claiming crash-safe with `WATCH`/`MULTI`/`EXEC`.
|
|
6
|
+
- Prevented future-dated jobs accidentally found in a ready list from becoming
|
|
7
|
+
orphaned; they are moved back to the delayed set atomically.
|
|
8
|
+
- Added `RedisDriver.reconcile_pending_jobs()` and the
|
|
9
|
+
`baqueue reconcile-pending` command for repairing pending jobs missing from
|
|
10
|
+
both ready and delayed storage.
|
|
11
|
+
- Kept Redis job status indexes, ready lists, and delayed membership consistent
|
|
12
|
+
during claims, stale-entry cleanup, retries, and reconciliation.
|
|
13
|
+
- Preserved `attempts`, `max_attempts`, `backoff`, and per-attempt history across
|
|
14
|
+
Redis release and requeue paths.
|
|
15
|
+
- Added Redis concurrency, failure-injection, delayed-job, reconciliation, and
|
|
16
|
+
index-consistency regression coverage.
|
|
17
|
+
- Updated the dashboard Jobs table to show processing duration live to
|
|
18
|
+
centisecond precision.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: baqueue
|
|
3
|
-
Version: 1.2.
|
|
3
|
+
Version: 1.2.2
|
|
4
4
|
Summary: A powerful Python queue management package inspired by Laravel Horizon
|
|
5
5
|
Author: Basalam, BaQueue Contributors
|
|
6
6
|
License: MIT
|
|
@@ -327,6 +327,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
|
|
|
327
327
|
Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
|
|
328
328
|
to keep connect fast on large datasets).
|
|
329
329
|
|
|
330
|
+
If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
|
|
331
|
+
pending jobs that are present in the pending index but missing from both the ready list
|
|
332
|
+
and delayed set:
|
|
333
|
+
|
|
334
|
+
```bash
|
|
335
|
+
baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
The repair is safe to repeat. Each job is handled in its own Redis transaction: future
|
|
339
|
+
jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
|
|
340
|
+
and missing hashes lose stale queue/index references.
|
|
341
|
+
|
|
330
342
|
### Retry Failed Jobs
|
|
331
343
|
|
|
332
344
|
Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
|
|
@@ -552,6 +564,8 @@ Coverage includes:
|
|
|
552
564
|
- Batch builder + completion callbacks
|
|
553
565
|
- DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
|
|
554
566
|
- CLI command surface (help text, validation, `retry-failed` abort flow)
|
|
567
|
+
- Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
|
|
568
|
+
and secondary-index consistency
|
|
555
569
|
|
|
556
570
|
## CLI Commands
|
|
557
571
|
|
|
@@ -562,6 +576,7 @@ baqueue dashboard Launch the monitoring dashboard
|
|
|
562
576
|
baqueue prune Prune old jobs
|
|
563
577
|
baqueue retry-failed Retry all failed jobs (filter by queue/tag/age)
|
|
564
578
|
baqueue reconcile-indexes Repair Redis secondary indexes (drop stale entries)
|
|
579
|
+
baqueue reconcile-pending Repair pending jobs missing from ready/delayed storage
|
|
565
580
|
baqueue status Show queue status
|
|
566
581
|
baqueue test Run the test suite
|
|
567
582
|
```
|
|
@@ -275,6 +275,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
|
|
|
275
275
|
Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
|
|
276
276
|
to keep connect fast on large datasets).
|
|
277
277
|
|
|
278
|
+
If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
|
|
279
|
+
pending jobs that are present in the pending index but missing from both the ready list
|
|
280
|
+
and delayed set:
|
|
281
|
+
|
|
282
|
+
```bash
|
|
283
|
+
baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
|
|
284
|
+
```
|
|
285
|
+
|
|
286
|
+
The repair is safe to repeat. Each job is handled in its own Redis transaction: future
|
|
287
|
+
jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
|
|
288
|
+
and missing hashes lose stale queue/index references.
|
|
289
|
+
|
|
278
290
|
### Retry Failed Jobs
|
|
279
291
|
|
|
280
292
|
Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
|
|
@@ -500,6 +512,8 @@ Coverage includes:
|
|
|
500
512
|
- Batch builder + completion callbacks
|
|
501
513
|
- DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
|
|
502
514
|
- CLI command surface (help text, validation, `retry-failed` abort flow)
|
|
515
|
+
- Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
|
|
516
|
+
and secondary-index consistency
|
|
503
517
|
|
|
504
518
|
## CLI Commands
|
|
505
519
|
|
|
@@ -510,6 +524,7 @@ baqueue dashboard Launch the monitoring dashboard
|
|
|
510
524
|
baqueue prune Prune old jobs
|
|
511
525
|
baqueue retry-failed Retry all failed jobs (filter by queue/tag/age)
|
|
512
526
|
baqueue reconcile-indexes Repair Redis secondary indexes (drop stale entries)
|
|
527
|
+
baqueue reconcile-pending Repair pending jobs missing from ready/delayed storage
|
|
513
528
|
baqueue status Show queue status
|
|
514
529
|
baqueue test Run the test suite
|
|
515
530
|
```
|
|
@@ -425,6 +425,35 @@ async def _run_reconcile_indexes(config: BaQueueConfig, batch: int) -> int:
|
|
|
425
425
|
await Queue.disconnect()
|
|
426
426
|
|
|
427
427
|
|
|
428
|
+
@cli.command(name="reconcile-pending")
|
|
429
|
+
@click.option("--batch", default=500, type=int, help="Pending jobs scanned per batch.")
|
|
430
|
+
@click.option("--driver", "-d", default="redis", help="Driver name (sqlite, memory, redis, postgres).")
|
|
431
|
+
@click.option("--driver-url", default=None, help="Driver connection URL.")
|
|
432
|
+
@click.pass_context
|
|
433
|
+
def reconcile_pending(
|
|
434
|
+
ctx: click.Context,
|
|
435
|
+
batch: int,
|
|
436
|
+
driver: str,
|
|
437
|
+
driver_url: str | None,
|
|
438
|
+
) -> None:
|
|
439
|
+
"""Repair pending jobs missing from both ready and delayed storage."""
|
|
440
|
+
_validate_driver(driver)
|
|
441
|
+
config: BaQueueConfig = ctx.obj["config"]
|
|
442
|
+
config.driver = DriverConfig(name=driver, url=driver_url or "")
|
|
443
|
+
|
|
444
|
+
repaired = _run_async(_run_reconcile_pending, config, batch)
|
|
445
|
+
click.echo(f"Reconciled {repaired or 0} pending job(s).")
|
|
446
|
+
|
|
447
|
+
|
|
448
|
+
async def _run_reconcile_pending(config: BaQueueConfig, batch: int) -> int:
|
|
449
|
+
Queue.configure(config)
|
|
450
|
+
await Queue.connect()
|
|
451
|
+
try:
|
|
452
|
+
return await Queue.get_driver().reconcile_pending_jobs(batch=batch)
|
|
453
|
+
finally:
|
|
454
|
+
await Queue.disconnect()
|
|
455
|
+
|
|
456
|
+
|
|
428
457
|
@cli.command()
|
|
429
458
|
@click.option("--driver", "-d", default="sqlite", help="Driver name (sqlite, memory, redis, postgres).")
|
|
430
459
|
@click.option("--driver-url", default=None, help="Driver connection URL.")
|
|
@@ -25,6 +25,8 @@ document.addEventListener("alpine:init", () => {
|
|
|
25
25
|
dateTo: "",
|
|
26
26
|
|
|
27
27
|
refreshTimer: null,
|
|
28
|
+
durationTimer: null,
|
|
29
|
+
liveNow: Date.now(),
|
|
28
30
|
|
|
29
31
|
get pageTitle() {
|
|
30
32
|
const titles = { overview: "Overview", jobs: "Jobs", queues: "Queues", workers: "Workers" };
|
|
@@ -37,6 +39,17 @@ document.addEventListener("alpine:init", () => {
|
|
|
37
39
|
this.fetchSupervisors();
|
|
38
40
|
this.fetchRecentJobs();
|
|
39
41
|
this.connectWS();
|
|
42
|
+
this.durationTimer = setInterval(() => {
|
|
43
|
+
if (
|
|
44
|
+
this.tab === "jobs" &&
|
|
45
|
+
this.jobs.some((job) => this.inFlightAttempt(job))
|
|
46
|
+
) {
|
|
47
|
+
const now = Date.now();
|
|
48
|
+
if (Math.floor(now / 10) !== Math.floor(this.liveNow / 10)) {
|
|
49
|
+
this.liveNow = now;
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
}, 10);
|
|
40
53
|
this.refreshTimer = setInterval(() => {
|
|
41
54
|
if (this.tab === "overview") {
|
|
42
55
|
this.fetchOverview();
|
|
@@ -49,6 +62,7 @@ document.addEventListener("alpine:init", () => {
|
|
|
49
62
|
|
|
50
63
|
destroy() {
|
|
51
64
|
if (this.refreshTimer) clearInterval(this.refreshTimer);
|
|
65
|
+
if (this.durationTimer) clearInterval(this.durationTimer);
|
|
52
66
|
if (this.ws) this.ws.close();
|
|
53
67
|
this.disconnectJobsWS();
|
|
54
68
|
},
|
|
@@ -437,12 +451,26 @@ document.addEventListener("alpine:init", () => {
|
|
|
437
451
|
|
|
438
452
|
jobDuration(job) {
|
|
439
453
|
if (!job.started_at) return "-";
|
|
440
|
-
const end = job.completed_at || job.failed_at || (
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
454
|
+
const end = job.completed_at || job.failed_at || (this.liveNow / 1000);
|
|
455
|
+
return this.formatDuration(end - job.started_at);
|
|
456
|
+
},
|
|
457
|
+
|
|
458
|
+
formatDuration(diff) {
|
|
459
|
+
if (!Number.isFinite(diff) || diff < 0) return "-";
|
|
460
|
+
|
|
461
|
+
// Work in centiseconds so a live duration never displays more precision
|
|
462
|
+
// than the requested hundredth of a second.
|
|
463
|
+
const centiseconds = Math.floor(diff * 100);
|
|
464
|
+
const totalSeconds = centiseconds / 100;
|
|
465
|
+
if (totalSeconds < 60) return totalSeconds.toFixed(2) + "s";
|
|
466
|
+
|
|
467
|
+
const minutes = Math.floor(totalSeconds / 60);
|
|
468
|
+
const seconds = (totalSeconds % 60).toFixed(2).padStart(5, "0");
|
|
469
|
+
if (minutes < 60) return minutes + "m " + seconds + "s";
|
|
470
|
+
|
|
471
|
+
const hours = Math.floor(minutes / 60);
|
|
472
|
+
const remainingMinutes = String(minutes % 60).padStart(2, "0");
|
|
473
|
+
return hours + "h " + remainingMinutes + "m " + seconds + "s";
|
|
446
474
|
},
|
|
447
475
|
|
|
448
476
|
// ── Per-attempt timeline ────────────────────────────────
|
|
@@ -314,7 +314,7 @@
|
|
|
314
314
|
</div>
|
|
315
315
|
</td>
|
|
316
316
|
<td class="cell-time" x-text="formatTime(job.created_at)"></td>
|
|
317
|
-
<td class="cell-time" x-text="jobDuration(job)"></td>
|
|
317
|
+
<td class="cell-time cell-duration" x-text="jobDuration(job)"></td>
|
|
318
318
|
<td>
|
|
319
319
|
<button class="btn-action" @click.stop="viewJob(job.id)">
|
|
320
320
|
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" width="16" height="16"><polyline points="9 18 15 12 9 6"/></svg>
|
|
@@ -732,7 +732,14 @@ body {
|
|
|
732
732
|
|
|
733
733
|
.cell-id { font-size: 12px; }
|
|
734
734
|
.cell-class { font-weight: 600; max-width: 200px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
|
735
|
-
.cell-time {
|
|
735
|
+
.cell-time {
|
|
736
|
+
font-size: 12px;
|
|
737
|
+
color: var(--text-muted);
|
|
738
|
+
white-space: nowrap;
|
|
739
|
+
font-variant-numeric: tabular-nums;
|
|
740
|
+
}
|
|
741
|
+
|
|
742
|
+
.cell-duration { min-width: 88px; }
|
|
736
743
|
|
|
737
744
|
.cell-attempts { font-size: 13px; }
|
|
738
745
|
|
|
@@ -260,6 +260,14 @@ class BaseDriver(ABC):
|
|
|
260
260
|
number of stale index entries removed."""
|
|
261
261
|
return 0
|
|
262
262
|
|
|
263
|
+
async def reconcile_pending_jobs(self, batch: int = 500) -> int:
|
|
264
|
+
"""Repair pending jobs that are absent from their runnable storage.
|
|
265
|
+
|
|
266
|
+
No-op for drivers without a separate ready/delayed index model. Redis
|
|
267
|
+
overrides this with an atomic per-job repair pass.
|
|
268
|
+
"""
|
|
269
|
+
return 0
|
|
270
|
+
|
|
263
271
|
@abstractmethod
|
|
264
272
|
async def flush(self, queue: str | None = None) -> None:
|
|
265
273
|
"""Remove all jobs (optionally for a specific queue)."""
|
|
@@ -162,6 +162,13 @@ class RedisDriver(BaseDriver):
|
|
|
162
162
|
pipe.zadd(self._idx_status(new_status), {job_id: score})
|
|
163
163
|
pipe.zadd(self._idx_queue_status(queue, new_status), {job_id: score})
|
|
164
164
|
|
|
165
|
+
def _index_rebuild(self, pipe: Any, job: JobPayload) -> None:
|
|
166
|
+
"""Make every status index reflect the current payload in one transaction."""
|
|
167
|
+
for status in _ALL_STATUSES:
|
|
168
|
+
pipe.zrem(self._idx_status(status), job.id)
|
|
169
|
+
pipe.zrem(self._idx_queue_status(job.queue, status), job.id)
|
|
170
|
+
self._index_add(pipe, job)
|
|
171
|
+
|
|
165
172
|
# ── Push / Pop ──────────────────────────────────────────────
|
|
166
173
|
|
|
167
174
|
async def push(self, payload: JobPayload) -> str:
|
|
@@ -203,43 +210,70 @@ class RedisDriver(BaseDriver):
|
|
|
203
210
|
await self._with_disk_full_recovery(_do)
|
|
204
211
|
return ids
|
|
205
212
|
|
|
206
|
-
async def
|
|
213
|
+
async def _pop_and_claim_pending_job(
|
|
214
|
+
self, queue: str,
|
|
215
|
+
) -> tuple[bool, JobPayload | None]:
|
|
207
216
|
from redis.exceptions import WatchError
|
|
208
217
|
|
|
209
|
-
|
|
218
|
+
queue_key = self._key("queue", queue)
|
|
219
|
+
delayed_key = self._key("delayed")
|
|
210
220
|
|
|
211
|
-
async def _attempt() -> JobPayload | None:
|
|
221
|
+
async def _attempt() -> tuple[bool, JobPayload | None]:
|
|
212
222
|
pipe = self._redis.pipeline()
|
|
213
223
|
try:
|
|
214
|
-
await pipe.watch(
|
|
224
|
+
await pipe.watch(queue_key)
|
|
225
|
+
job_id = await pipe.lindex(queue_key, 0)
|
|
226
|
+
if not job_id:
|
|
227
|
+
return False, None
|
|
228
|
+
|
|
229
|
+
job_key = self._key("job", job_id)
|
|
230
|
+
await pipe.watch(job_key, delayed_key)
|
|
215
231
|
raw = await pipe.hget(job_key, "data")
|
|
216
232
|
if not raw:
|
|
217
|
-
|
|
233
|
+
pipe.multi()
|
|
234
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
235
|
+
pipe.zrem(delayed_key, job_id)
|
|
236
|
+
self._index_remove_orphan(pipe, job_id, queue, None)
|
|
237
|
+
await pipe.execute()
|
|
238
|
+
return True, None
|
|
218
239
|
|
|
219
240
|
payload = JobPayload.from_json(raw)
|
|
220
241
|
if payload.status != "pending":
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
242
|
+
pipe.multi()
|
|
243
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
244
|
+
pipe.zrem(delayed_key, job_id)
|
|
245
|
+
self._index_rebuild(pipe, payload)
|
|
246
|
+
await pipe.execute()
|
|
247
|
+
return True, None
|
|
224
248
|
|
|
225
|
-
payload.status = "processing"
|
|
226
249
|
now = _now_ts()
|
|
250
|
+
if payload.delay_until is not None and payload.delay_until > now:
|
|
251
|
+
pipe.multi()
|
|
252
|
+
# A future-dated job found in the ready list is corrupt
|
|
253
|
+
# state. Move it back to delayed atomically instead of
|
|
254
|
+
# dropping the only runnable reference.
|
|
255
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
256
|
+
pipe.zadd(delayed_key, {job_id: payload.delay_until})
|
|
257
|
+
self._index_rebuild(pipe, payload)
|
|
258
|
+
await pipe.execute()
|
|
259
|
+
return True, None
|
|
260
|
+
|
|
261
|
+
payload.status = "processing"
|
|
227
262
|
payload.started_at = now
|
|
228
263
|
payload.updated_at = now
|
|
229
264
|
payload.attempts += 1
|
|
230
265
|
|
|
231
266
|
pipe.multi()
|
|
267
|
+
# The list entry is removed in the same EXEC that persists the
|
|
268
|
+
# processing claim. A connection failure before EXEC leaves it
|
|
269
|
+
# available; an ambiguous post-EXEC failure leaves a processing
|
|
270
|
+
# job for the existing stuck-job recovery path.
|
|
271
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
272
|
+
pipe.zrem(delayed_key, job_id)
|
|
232
273
|
pipe.hset(job_key, mapping={"data": payload.to_json()})
|
|
233
|
-
self.
|
|
234
|
-
pipe,
|
|
235
|
-
payload.id,
|
|
236
|
-
payload.queue,
|
|
237
|
-
"pending",
|
|
238
|
-
"processing",
|
|
239
|
-
payload.created_at,
|
|
240
|
-
)
|
|
274
|
+
self._index_rebuild(pipe, payload)
|
|
241
275
|
await pipe.execute()
|
|
242
|
-
return payload
|
|
276
|
+
return True, payload
|
|
243
277
|
finally:
|
|
244
278
|
await pipe.reset()
|
|
245
279
|
|
|
@@ -248,14 +282,13 @@ class RedisDriver(BaseDriver):
|
|
|
248
282
|
return await self._with_disk_full_recovery(_attempt)
|
|
249
283
|
except WatchError:
|
|
250
284
|
continue
|
|
251
|
-
return None
|
|
285
|
+
return False, None
|
|
252
286
|
|
|
253
287
|
async def pop(self, queue: str) -> JobPayload | None:
|
|
254
288
|
for _ in range(_MAX_STALE_POP_ATTEMPTS):
|
|
255
|
-
|
|
256
|
-
if not
|
|
289
|
+
found, payload = await self._pop_and_claim_pending_job(queue)
|
|
290
|
+
if not found:
|
|
257
291
|
return None
|
|
258
|
-
payload = await self._claim_pending_job(job_id)
|
|
259
292
|
if payload is not None:
|
|
260
293
|
return payload
|
|
261
294
|
return None
|
|
@@ -348,6 +381,7 @@ class RedisDriver(BaseDriver):
|
|
|
348
381
|
old_status = payload.status
|
|
349
382
|
payload.status = "pending"
|
|
350
383
|
payload.updated_at = _now_ts()
|
|
384
|
+
delayed_key = self._key("delayed")
|
|
351
385
|
|
|
352
386
|
async def _do():
|
|
353
387
|
pipe = self._redis.pipeline()
|
|
@@ -355,10 +389,12 @@ class RedisDriver(BaseDriver):
|
|
|
355
389
|
if delay > 0:
|
|
356
390
|
payload.delay_until = _now_ts() + delay
|
|
357
391
|
pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
|
|
358
|
-
pipe.zadd(
|
|
392
|
+
pipe.zadd(delayed_key, {payload.id: payload.delay_until})
|
|
359
393
|
else:
|
|
394
|
+
payload.delay_until = None
|
|
360
395
|
pipe.hset(self._key("job", payload.id), mapping={"data": payload.to_json()})
|
|
361
396
|
pipe.rpush(self._key("queue", payload.queue), payload.id)
|
|
397
|
+
pipe.zrem(delayed_key, payload.id)
|
|
362
398
|
self._index_status_change(pipe, payload.id, payload.queue, old_status, "pending", payload.created_at)
|
|
363
399
|
await pipe.execute()
|
|
364
400
|
await self._with_disk_full_recovery(_do)
|
|
@@ -914,6 +950,143 @@ class RedisDriver(BaseDriver):
|
|
|
914
950
|
break
|
|
915
951
|
return removed
|
|
916
952
|
|
|
953
|
+
async def _scan_keys(self, pattern: str, batch: int) -> list[str]:
|
|
954
|
+
keys: list[str] = []
|
|
955
|
+
cursor: Any = "0"
|
|
956
|
+
while True:
|
|
957
|
+
cursor, found = await self._redis.scan(cursor=cursor, match=pattern, count=batch)
|
|
958
|
+
keys.extend(found)
|
|
959
|
+
if cursor == "0" or cursor == 0:
|
|
960
|
+
break
|
|
961
|
+
return keys
|
|
962
|
+
|
|
963
|
+
async def _reconcile_pending_job(
|
|
964
|
+
self,
|
|
965
|
+
job_id: str,
|
|
966
|
+
ready_queue_keys: list[str],
|
|
967
|
+
index_keys: list[str],
|
|
968
|
+
) -> int:
|
|
969
|
+
"""Repair one pending-index member without exposing an intermediate state."""
|
|
970
|
+
from redis.exceptions import WatchError
|
|
971
|
+
|
|
972
|
+
job_key = self._key("job", job_id)
|
|
973
|
+
delayed_key = self._key("delayed")
|
|
974
|
+
|
|
975
|
+
async def _attempt() -> int:
|
|
976
|
+
pipe = self._redis.pipeline()
|
|
977
|
+
try:
|
|
978
|
+
watch_keys = [job_key, delayed_key, *ready_queue_keys]
|
|
979
|
+
await pipe.watch(*watch_keys)
|
|
980
|
+
raw = await pipe.hget(job_key, "data")
|
|
981
|
+
|
|
982
|
+
payload: JobPayload | None = None
|
|
983
|
+
if raw:
|
|
984
|
+
try:
|
|
985
|
+
payload = JobPayload.from_json(raw)
|
|
986
|
+
except (TypeError, ValueError, json.JSONDecodeError):
|
|
987
|
+
# Keep malformed data for manual inspection, but remove
|
|
988
|
+
# its runnable/index references so workers do not spin on it.
|
|
989
|
+
payload = None
|
|
990
|
+
|
|
991
|
+
if payload is None:
|
|
992
|
+
pipe.multi()
|
|
993
|
+
for queue_key in ready_queue_keys:
|
|
994
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
995
|
+
pipe.zrem(delayed_key, job_id)
|
|
996
|
+
for index_key in index_keys:
|
|
997
|
+
pipe.zrem(index_key, job_id)
|
|
998
|
+
await pipe.execute()
|
|
999
|
+
return 1
|
|
1000
|
+
|
|
1001
|
+
queue_key = self._key("queue", payload.queue)
|
|
1002
|
+
if queue_key not in watch_keys:
|
|
1003
|
+
await pipe.watch(queue_key)
|
|
1004
|
+
|
|
1005
|
+
ready_positions_raw = await pipe.lpos(queue_key, job_id, count=2)
|
|
1006
|
+
delayed_score = await pipe.zscore(delayed_key, job_id)
|
|
1007
|
+
if isinstance(ready_positions_raw, (list, tuple)):
|
|
1008
|
+
ready_positions = list(ready_positions_raw)
|
|
1009
|
+
elif ready_positions_raw is None:
|
|
1010
|
+
ready_positions = []
|
|
1011
|
+
else:
|
|
1012
|
+
ready_positions = [ready_positions_raw]
|
|
1013
|
+
|
|
1014
|
+
pipe.multi()
|
|
1015
|
+
if payload.status != "pending":
|
|
1016
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
1017
|
+
pipe.zrem(delayed_key, job_id)
|
|
1018
|
+
self._index_rebuild(pipe, payload)
|
|
1019
|
+
elif payload.delay_until is not None and payload.delay_until > _now_ts():
|
|
1020
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
1021
|
+
pipe.zadd(delayed_key, {job_id: payload.delay_until})
|
|
1022
|
+
self._index_rebuild(pipe, payload)
|
|
1023
|
+
else:
|
|
1024
|
+
if payload.delay_until is not None:
|
|
1025
|
+
payload.delay_until = None
|
|
1026
|
+
payload.updated_at = _now_ts()
|
|
1027
|
+
pipe.hset(job_key, mapping={"data": payload.to_json()})
|
|
1028
|
+
if len(ready_positions) != 1 or delayed_score is not None:
|
|
1029
|
+
pipe.lrem(queue_key, 0, job_id)
|
|
1030
|
+
pipe.rpush(queue_key, job_id)
|
|
1031
|
+
pipe.zrem(delayed_key, job_id)
|
|
1032
|
+
self._index_rebuild(pipe, payload)
|
|
1033
|
+
await pipe.execute()
|
|
1034
|
+
return 1
|
|
1035
|
+
finally:
|
|
1036
|
+
await pipe.reset()
|
|
1037
|
+
|
|
1038
|
+
for _ in range(5):
|
|
1039
|
+
try:
|
|
1040
|
+
return int(await self._with_disk_full_recovery(_attempt) or 0)
|
|
1041
|
+
except WatchError:
|
|
1042
|
+
continue
|
|
1043
|
+
return 0
|
|
1044
|
+
|
|
1045
|
+
async def reconcile_pending_jobs(self, batch: int = 500) -> int:
|
|
1046
|
+
"""Repair pending jobs that are absent from both ready and delayed storage.
|
|
1047
|
+
|
|
1048
|
+
The pending status index is the work source. Each job is repaired in its
|
|
1049
|
+
own WATCH/MULTI/EXEC transaction: missing or malformed hashes lose stale
|
|
1050
|
+
queue/index references, future jobs are restored to the delayed ZSET, and
|
|
1051
|
+
runnable jobs are normalized to exactly one ready-list entry. The method
|
|
1052
|
+
is safe to run repeatedly and concurrent workers can win a race without
|
|
1053
|
+
duplicating or claiming a job.
|
|
1054
|
+
"""
|
|
1055
|
+
batch = max(1, batch)
|
|
1056
|
+
ready_queue_keys = await self._scan_keys(self._key("queue", "*"), batch)
|
|
1057
|
+
index_keys = await self._scan_keys(self._key("jobs", "*"), batch)
|
|
1058
|
+
pending_index = self._idx_status("pending")
|
|
1059
|
+
|
|
1060
|
+
repaired = 0
|
|
1061
|
+
processed_ids: set[str] = set()
|
|
1062
|
+
while True:
|
|
1063
|
+
discovered = 0
|
|
1064
|
+
cursor: Any = 0
|
|
1065
|
+
while True:
|
|
1066
|
+
cursor, members = await self._redis.zscan(
|
|
1067
|
+
pending_index, cursor=cursor, count=batch,
|
|
1068
|
+
)
|
|
1069
|
+
job_ids = [
|
|
1070
|
+
member[0] if isinstance(member, (tuple, list)) else member
|
|
1071
|
+
for member in members
|
|
1072
|
+
]
|
|
1073
|
+
for job_id in job_ids:
|
|
1074
|
+
if job_id in processed_ids:
|
|
1075
|
+
continue
|
|
1076
|
+
processed_ids.add(job_id)
|
|
1077
|
+
discovered += 1
|
|
1078
|
+
repaired += await self._reconcile_pending_job(
|
|
1079
|
+
job_id, ready_queue_keys, index_keys,
|
|
1080
|
+
)
|
|
1081
|
+
if cursor == 0 or cursor == "0":
|
|
1082
|
+
break
|
|
1083
|
+
# Redis permits mutation while ZSCAN is in progress. Removing stale
|
|
1084
|
+
# members can make a later member invisible to the current cursor, so
|
|
1085
|
+
# take another pass until a complete scan discovers nothing new.
|
|
1086
|
+
if discovered == 0:
|
|
1087
|
+
break
|
|
1088
|
+
return repaired
|
|
1089
|
+
|
|
917
1090
|
async def prune_metrics(self, older_than_seconds: float) -> int:
|
|
918
1091
|
cutoff = _now_ts() - older_than_seconds
|
|
919
1092
|
removed = 0
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: baqueue
|
|
3
|
-
Version: 1.2.
|
|
3
|
+
Version: 1.2.2
|
|
4
4
|
Summary: A powerful Python queue management package inspired by Laravel Horizon
|
|
5
5
|
Author: Basalam, BaQueue Contributors
|
|
6
6
|
License: MIT
|
|
@@ -327,6 +327,18 @@ baqueue reconcile-indexes -d redis --driver-url redis://localhost:6379/0
|
|
|
327
327
|
Set `reconcile_on_connect=True` to run that repair once on every startup (off by default
|
|
328
328
|
to keep connect fast on large datasets).
|
|
329
329
|
|
|
330
|
+
If a Redis failure occurred during the old non-atomic ready-list dequeue path, repair
|
|
331
|
+
pending jobs that are present in the pending index but missing from both the ready list
|
|
332
|
+
and delayed set:
|
|
333
|
+
|
|
334
|
+
```bash
|
|
335
|
+
baqueue reconcile-pending -d redis --driver-url redis://localhost:6379/0
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
The repair is safe to repeat. Each job is handled in its own Redis transaction: future
|
|
339
|
+
jobs return to the delayed sorted set, runnable jobs get exactly one ready-list entry,
|
|
340
|
+
and missing hashes lose stale queue/index references.
|
|
341
|
+
|
|
330
342
|
### Retry Failed Jobs
|
|
331
343
|
|
|
332
344
|
Bulk-retry failed jobs from the CLI, from Python, or from the dashboard.
|
|
@@ -552,6 +564,8 @@ Coverage includes:
|
|
|
552
564
|
- Batch builder + completion callbacks
|
|
553
565
|
- DashboardAPI (overview, jobs_list, retry, bulk retry-failed, prune, stats)
|
|
554
566
|
- CLI command surface (help text, validation, `retry-failed` abort flow)
|
|
567
|
+
- Redis claim atomicity, concurrent workers, delayed recovery, pending reconciliation,
|
|
568
|
+
and secondary-index consistency
|
|
555
569
|
|
|
556
570
|
## CLI Commands
|
|
557
571
|
|
|
@@ -562,6 +576,7 @@ baqueue dashboard Launch the monitoring dashboard
|
|
|
562
576
|
baqueue prune Prune old jobs
|
|
563
577
|
baqueue retry-failed Retry all failed jobs (filter by queue/tag/age)
|
|
564
578
|
baqueue reconcile-indexes Repair Redis secondary indexes (drop stale entries)
|
|
579
|
+
baqueue reconcile-pending Repair pending jobs missing from ready/delayed storage
|
|
565
580
|
baqueue status Show queue status
|
|
566
581
|
baqueue test Run the test suite
|
|
567
582
|
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|