good_job 3.9.0 → 3.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +14 -0
- data/README.md +194 -23
- data/app/controllers/good_job/batches_controller.rb +12 -0
- data/app/filters/good_job/batches_filter.rb +21 -0
- data/app/models/good_job/batch.rb +141 -0
- data/app/models/good_job/batch_record.rb +84 -0
- data/app/models/good_job/execution.rb +47 -21
- data/app/models/good_job/job.rb +1 -0
- data/app/views/good_job/batches/_jobs.erb +108 -0
- data/app/views/good_job/batches/_table.erb +61 -0
- data/app/views/good_job/batches/index.html.erb +16 -0
- data/app/views/good_job/batches/show.html.erb +30 -0
- data/app/views/good_job/shared/_navbar.erb +7 -0
- data/config/routes.rb +2 -0
- data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb +19 -0
- data/lib/generators/good_job/templates/update/migrations/04_create_good_job_batches.rb.erb +34 -0
- data/lib/good_job/active_job_extensions/batches.rb +13 -0
- data/lib/good_job/bulk.rb +25 -21
- data/lib/good_job/log_subscriber.rb +1 -1
- data/lib/good_job/version.rb +1 -1
- data/lib/good_job.rb +16 -6
- metadata +12 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 22496fe725c6ad616c39f243cf8f53381dee64a5402d3a199c8c604bff2c4eb3
|
|
4
|
+
data.tar.gz: 6d9791dd28f28ddf9317f2770029323754fa32c60403f336f23db8020cdecc51
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: e182b3aa78b38fd75baa5e0c14704baa99b7aa98f34ff011006543ce7a356e6eae8ac45e605d65d985b8631f6f185666f78e5bb1be1f39feebd62c159535a7e5
|
|
7
|
+
data.tar.gz: acf8a1f601e07142b0dbaea2f2e229fa3d9b7f8c50dd6232531f8ff50cf888d61f88f2a02ae423ffaa8a9a6533bd1cd0522e7cfa849abb20db8dc538adf75ae3
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [v3.10.0](https://github.com/bensheldon/good_job/tree/v3.10.0) (2023-02-04)
|
|
4
|
+
|
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.9.0...v3.10.0)
|
|
6
|
+
|
|
7
|
+
**Implemented enhancements:**
|
|
8
|
+
|
|
9
|
+
- Implement `GoodJob::Batch` [\#712](https://github.com/bensheldon/good_job/pull/712) ([bensheldon](https://github.com/bensheldon))
|
|
10
|
+
|
|
11
|
+
**Closed issues:**
|
|
12
|
+
|
|
13
|
+
- Support for Ruby 3.2 [\#785](https://github.com/bensheldon/good_job/issues/785)
|
|
14
|
+
- Custom table names [\#748](https://github.com/bensheldon/good_job/issues/748)
|
|
15
|
+
- Health check issue with cron scheduler job [\#741](https://github.com/bensheldon/good_job/issues/741)
|
|
16
|
+
|
|
3
17
|
## [v3.9.0](https://github.com/bensheldon/good_job/tree/v3.9.0) (2023-01-31)
|
|
4
18
|
|
|
5
19
|
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.8.0...v3.9.0)
|
data/README.md
CHANGED
|
@@ -41,9 +41,11 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
|
41
41
|
- [Dashboard](#dashboard)
|
|
42
42
|
- [API-only Rails applications](#api-only-rails-applications)
|
|
43
43
|
- [Live Polling](#live-polling)
|
|
44
|
-
- [
|
|
44
|
+
- [Concurrency controls](#concurrency-controls)
|
|
45
45
|
- [How concurrency controls work](#how-concurrency-controls-work)
|
|
46
46
|
- [Cron-style repeating/recurring jobs](#cron-style-repeatingrecurring-jobs)
|
|
47
|
+
- [Bulk enqueue](#bulk-enqueue)
|
|
48
|
+
- [Batches](#batches)
|
|
47
49
|
- [Updating](#updating)
|
|
48
50
|
- [Upgrading minor versions](#upgrading-minor-versions)
|
|
49
51
|
- [Upgrading v2 to v3](#upgrading-v2-to-v3)
|
|
@@ -58,7 +60,6 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
|
58
60
|
- [Database connections](#database-connections)
|
|
59
61
|
- [Production setup](#production-setup)
|
|
60
62
|
- [Queue performance with Queue Select Limit](#queue-performance-with-queue-select-limit)
|
|
61
|
-
- [Bulk enqueue](#bulk-enqueue)
|
|
62
63
|
- [Execute jobs async / in-process](#execute-jobs-async--in-process)
|
|
63
64
|
- [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
|
|
64
65
|
- [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
|
|
@@ -392,7 +393,7 @@ end
|
|
|
392
393
|
|
|
393
394
|
The Dashboard can be set to automatically refresh by checking "Live Poll" in the Dashboard header, or by setting `?poll=10` with the interval in seconds (default 30 seconds).
|
|
394
395
|
|
|
395
|
-
###
|
|
396
|
+
### Concurrency controls
|
|
396
397
|
|
|
397
398
|
GoodJob can extend ActiveJob to provide limits on concurrently running jobs, either at time of _enqueue_ or at _perform_. Limiting concurrency can help prevent duplicate, double or unnecessary jobs from being enqueued, or race conditions when performing, for example when interacting with 3rd-party APIs.
|
|
398
399
|
|
|
@@ -485,6 +486,196 @@ config.good_job.cron = {
|
|
|
485
486
|
}
|
|
486
487
|
```
|
|
487
488
|
|
|
489
|
+
### Bulk enqueue
|
|
490
|
+
|
|
491
|
+
GoodJob's Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can more performant when enqueuing a large number of jobs.
|
|
492
|
+
|
|
493
|
+
```ruby
|
|
494
|
+
# Capture jobs using `.perform_later`:
|
|
495
|
+
active_jobs = GoodJob::Bulk.enqueue do
|
|
496
|
+
MyJob.perform_later
|
|
497
|
+
AnotherJob.perform_later
|
|
498
|
+
# If an exception is raised within this block, no jobs will be inserted.
|
|
499
|
+
end
|
|
500
|
+
|
|
501
|
+
# All ActiveJob instances are returned from GoodJob::Bulk.enqueue.
|
|
502
|
+
# Jobs that have been successfully enqueued have a `provider_job_id` set.
|
|
503
|
+
active_jobs.all?(&:provider_job_id)
|
|
504
|
+
|
|
505
|
+
# Bulk enqueue ActiveJob instances directly without using `.perform_later`:
|
|
506
|
+
GoodJob::Bulk.enqueue(MyJob.new, AnotherJob.new)
|
|
507
|
+
```
|
|
508
|
+
|
|
509
|
+
### Batches
|
|
510
|
+
|
|
511
|
+
Batches track a set of jobs, and enqueue an optional callback job when all of the jobs have finished (succeeded or discarded).
|
|
512
|
+
|
|
513
|
+
- A simple example that enqueues your `MyBatchCallbackJob` after the two jobs have finished, and passes along the current user as a batch property:
|
|
514
|
+
|
|
515
|
+
```ruby
|
|
516
|
+
GoodJob::Batch.enqueue(on_finish: MyBatchCallbackJob, user: current_user) do
|
|
517
|
+
MyJob.perform_later
|
|
518
|
+
OtherJob.perform_later
|
|
519
|
+
end
|
|
520
|
+
|
|
521
|
+
# When these jobs have finished, it will enqueue your `MyBatchCallbackJob.perform_later(batch, options)`
|
|
522
|
+
class MyBatchCallbackJob < ApplicationJob
|
|
523
|
+
# Callback jobs must accept a `batch` and `options` argument
|
|
524
|
+
def perform(batch, params)
|
|
525
|
+
# The batch object will contain the Batch's properties, which are mutable
|
|
526
|
+
batch.properties[:user] # => <User id: 1, ...>
|
|
527
|
+
|
|
528
|
+
# Params is a hash containing additional context (more may be added in the future)
|
|
529
|
+
params[:event] # => :finish, :success, :discard
|
|
530
|
+
end
|
|
531
|
+
end
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
- Jobs can be added to an existing batch. Jobs in a batch are enqueued and performed immediately/asynchronously. The final callback job will not be enqueued until `GoodJob::Batch#enqueue` is called.
|
|
535
|
+
|
|
536
|
+
```ruby
|
|
537
|
+
batch = GoodJob::Batch.add do
|
|
538
|
+
10.times { MyJob.perform_later }
|
|
539
|
+
end
|
|
540
|
+
batch.add do
|
|
541
|
+
10.times { OtherJob.perform_later }
|
|
542
|
+
end
|
|
543
|
+
batch.enqueue(on_finish: MyBatchCallbackJob, age: 42)
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
- If you need to access the batch within a job that is part of the batch, include [`GoodJob::ActiveJobExtensions::Batches`](lib/good_job/active_job_extensions/batches.rb) in your job class:
|
|
547
|
+
|
|
548
|
+
```ruby
|
|
549
|
+
class MyJob < ApplicationJob
|
|
550
|
+
include GoodJob::ActiveJobExtensions::Batches
|
|
551
|
+
|
|
552
|
+
def perform
|
|
553
|
+
self.batch # => <GoodJob::Batch id: 1, ...>
|
|
554
|
+
end
|
|
555
|
+
end
|
|
556
|
+
```
|
|
557
|
+
|
|
558
|
+
- [`GoodJob::Batch`](app/models/good_job/batch.rb) has a number of assignable attributes and methods:
|
|
559
|
+
|
|
560
|
+
```ruby
|
|
561
|
+
batch = GoodJob::Batch.new
|
|
562
|
+
batch.description = "My batch"
|
|
563
|
+
batch.on_finish = "MyBatchCallbackJob" # Callback job when all jobs have finished
|
|
564
|
+
batch.on_success = "MyBatchCallbackJob" # Callback job when/if all jobs have succeeded
|
|
565
|
+
batch.on_discard = "MyBatchCallbackJob" # Callback job when the first job in the batch is discarded
|
|
566
|
+
batch.callback_queue_name = "special_queue" # Optional queue for callback jobs, otherwise will defer to job class
|
|
567
|
+
batch.callback_priority = 10 # Optional priority name for callback jobs, otherwise will defer to job class
|
|
568
|
+
batch.properties = { age: 42 } # Custom data and state to attach to the batch
|
|
569
|
+
batch.add do
|
|
570
|
+
MyJob.perform_later
|
|
571
|
+
end
|
|
572
|
+
batch.enqueue
|
|
573
|
+
|
|
574
|
+
batch.discarded? # => Boolean
|
|
575
|
+
batch.discarded_at # => <DateTime>
|
|
576
|
+
batch.finished? # => Boolean
|
|
577
|
+
batch.finished_at # => <DateTime>
|
|
578
|
+
batch.succeeded? # => Boolean
|
|
579
|
+
batch.active_jobs # => Array of ActiveJob::Base-inherited jobs that are part of the batch
|
|
580
|
+
|
|
581
|
+
batch = GoodJob::Batch.find(batch.id)
|
|
582
|
+
batch.description = "Updated batch description"
|
|
583
|
+
batch.save
|
|
584
|
+
batch.reload
|
|
585
|
+
```
|
|
586
|
+
|
|
587
|
+
### Batch callback jobs
|
|
588
|
+
|
|
589
|
+
Batch callbacks are Active Job jobs that are enqueued at certain events during the execution of jobs within the batch:
|
|
590
|
+
|
|
591
|
+
- `:finish` - Enqueued when all jobs in the batch have finished, after all retries. Jobs will either be discarded or succeeded.
|
|
592
|
+
- `:success` - Enqueued only when all jobs in the batch have finished and succeeded.
|
|
593
|
+
- `:discard` - Enqueued immediately the first time a job in the batch is discarded.
|
|
594
|
+
|
|
595
|
+
Callback jobs must accept a `batch` and `params` argument in their `perform` method:
|
|
596
|
+
|
|
597
|
+
```ruby
|
|
598
|
+
class MyBatchCallbackJob < ApplicationJob
|
|
599
|
+
def perform(batch, params)
|
|
600
|
+
# The batch object will contain the Batch's properties
|
|
601
|
+
batch.properties[:user] # => <User id: 1, ...>
|
|
602
|
+
# Batches are mutable
|
|
603
|
+
batch.properties[:user] = User.find(2)
|
|
604
|
+
batch.save
|
|
605
|
+
|
|
606
|
+
# Params is a hash containing additional context (more may be added in the future)
|
|
607
|
+
params[:event] # => :finish, :success, :discard
|
|
608
|
+
end
|
|
609
|
+
end
|
|
610
|
+
```
|
|
611
|
+
|
|
612
|
+
#### Complex batches
|
|
613
|
+
|
|
614
|
+
Consider a multi-stage batch with both parallel and serial job steps:
|
|
615
|
+
|
|
616
|
+
```mermaid
|
|
617
|
+
graph TD
|
|
618
|
+
0{"BatchJob\n{ stage: nil }"}
|
|
619
|
+
0 --> a["WorkJob]\n{ step: a }"]
|
|
620
|
+
0 --> b["WorkJob]\n{ step: b }"]
|
|
621
|
+
0 --> c["WorkJob]\n{ step: c }"]
|
|
622
|
+
a --> 1
|
|
623
|
+
b --> 1
|
|
624
|
+
c --> 1
|
|
625
|
+
1{"BatchJob\n{ stage: 1 }"}
|
|
626
|
+
1 --> d["WorkJob]\n{ step: d }"]
|
|
627
|
+
1 --> e["WorkJob]\n{ step: e }"]
|
|
628
|
+
e --> f["WorkJob]\n{ step: f }"]
|
|
629
|
+
d --> 2
|
|
630
|
+
f --> 2
|
|
631
|
+
2{"BatchJob\n{ stage: 2 }"}
|
|
632
|
+
```
|
|
633
|
+
|
|
634
|
+
This can be implemented with a single, mutable batch job:
|
|
635
|
+
|
|
636
|
+
```ruby
|
|
637
|
+
class WorkJob < ApplicationJob
|
|
638
|
+
include GoodJob::ActiveJobExtensions::Batches
|
|
639
|
+
|
|
640
|
+
def perform(step)
|
|
641
|
+
# ...
|
|
642
|
+
if step == 'e'
|
|
643
|
+
batch.add { WorkJob.perform_later('f') }
|
|
644
|
+
end
|
|
645
|
+
end
|
|
646
|
+
end
|
|
647
|
+
|
|
648
|
+
class BatchJob < ApplicationJob
|
|
649
|
+
def perform(batch, options)
|
|
650
|
+
if batch.properties[:stage].nil?
|
|
651
|
+
batch.enqueue(stage: 1) do
|
|
652
|
+
WorkJob.perform_later('a')
|
|
653
|
+
WorkJob.perform_later('b')
|
|
654
|
+
WorkJob.perform_later('c')
|
|
655
|
+
end
|
|
656
|
+
elsif batch.properties[:stage] == 1
|
|
657
|
+
batch.enqueue(stage: 2) do
|
|
658
|
+
WorkJob.perform_later('d')
|
|
659
|
+
WorkJob.perform_later('e')
|
|
660
|
+
end
|
|
661
|
+
elsif batch.properties[:stage] == 2
|
|
662
|
+
# ...
|
|
663
|
+
end
|
|
664
|
+
end
|
|
665
|
+
end
|
|
666
|
+
|
|
667
|
+
GoodJob::Batch.enqueue(on_finish: BatchJob)
|
|
668
|
+
```
|
|
669
|
+
|
|
670
|
+
#### Other batch details
|
|
671
|
+
|
|
672
|
+
- Whether to enqueue a callback job is evaluated once the batch is in an `enqueued?`-state by using `GoodJob::Batch.enqueue` or `batch.enqueue`.
|
|
673
|
+
- Callback job enqueueing will be re-triggered if additional jobs are `enqueue`'d to the batch; use `add` to add jobs to the batch without retriggering callback jobs.
|
|
674
|
+
- Callback jobs will be enqueued even if the batch contains no jobs.
|
|
675
|
+
- Callback jobs perform asynchronously. It's possible that `:finish` and `:success` or `:discard` callback jobs perform at the same time. Keep this in mind when updating batch properties.
|
|
676
|
+
- Batch properties are serialized using Active Job serialization. This is flexible, but can lead to deserialization errors if a GlobalID record is directly referenced but is subsequently deleted and thus unloadable.
|
|
677
|
+
- 🚧Batches are a work in progress. Please let us know what would be helpful to improve their functionality and usefulness.
|
|
678
|
+
|
|
488
679
|
### Updating
|
|
489
680
|
|
|
490
681
|
GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.
|
|
@@ -794,26 +985,6 @@ To explain where this value is used, here is the pseudo-query that GoodJob uses
|
|
|
794
985
|
)
|
|
795
986
|
```
|
|
796
987
|
|
|
797
|
-
### Bulk enqueue
|
|
798
|
-
|
|
799
|
-
GoodJob's Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can more performant when enqueuing a large number of jobs.
|
|
800
|
-
|
|
801
|
-
```ruby
|
|
802
|
-
# Capture jobs using `.perform_later`:
|
|
803
|
-
active_jobs = GoodJob::Bulk.enqueue do
|
|
804
|
-
MyJob.perform_later
|
|
805
|
-
AnotherJob.perform_later
|
|
806
|
-
# If an exception is raised within this block, no jobs will be inserted.
|
|
807
|
-
end
|
|
808
|
-
|
|
809
|
-
# All ActiveJob instances are returned from GoodJob::Bulk.enqueue.
|
|
810
|
-
# Jobs that have been successfully enqueued have a `provider_job_id` set.
|
|
811
|
-
active_jobs.all?(&:provider_job_id)
|
|
812
|
-
|
|
813
|
-
# Bulk enqueue ActiveJob instances directly without using `.perform_later`:
|
|
814
|
-
GoodJob::Bulk.enqueue(MyJob.new, AnotherJob.new)
|
|
815
|
-
```
|
|
816
|
-
|
|
817
988
|
### Execute jobs async / in-process
|
|
818
989
|
|
|
819
990
|
GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
module GoodJob
|
|
3
|
+
class BatchesFilter < BaseFilter
|
|
4
|
+
def records
|
|
5
|
+
after_created_at = params[:after_created_at].present? ? Time.zone.parse(params[:after_created_at]) : nil
|
|
6
|
+
|
|
7
|
+
filtered_query.display_all(
|
|
8
|
+
after_created_at: after_created_at,
|
|
9
|
+
after_id: params[:after_id]
|
|
10
|
+
).limit(params.fetch(:limit, DEFAULT_LIMIT))
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def filtered_query(_filtered_params = params)
|
|
14
|
+
base_query
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def default_base_query
|
|
18
|
+
GoodJob::BatchRecord.all.includes(:jobs)
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module GoodJob
|
|
4
|
+
# NOTE: This class delegates to {GoodJob::BatchRecord} and is intended to be the public interface for Batches.
|
|
5
|
+
class Batch
|
|
6
|
+
include GlobalID::Identification
|
|
7
|
+
|
|
8
|
+
thread_cattr_accessor :current_batch_id
|
|
9
|
+
thread_cattr_accessor :current_batch_callback_id
|
|
10
|
+
|
|
11
|
+
PROTECTED_PROPERTIES = %i[
|
|
12
|
+
on_finish
|
|
13
|
+
on_success
|
|
14
|
+
on_discard
|
|
15
|
+
callback_queue_name
|
|
16
|
+
callback_priority
|
|
17
|
+
description
|
|
18
|
+
properties
|
|
19
|
+
].freeze
|
|
20
|
+
|
|
21
|
+
delegate(
|
|
22
|
+
:id,
|
|
23
|
+
:created_at,
|
|
24
|
+
:updated_at,
|
|
25
|
+
:persisted?,
|
|
26
|
+
:enqueued_at,
|
|
27
|
+
:finished_at,
|
|
28
|
+
:discarded_at,
|
|
29
|
+
:enqueued?,
|
|
30
|
+
:finished?,
|
|
31
|
+
:succeeded?,
|
|
32
|
+
:discarded?,
|
|
33
|
+
:description,
|
|
34
|
+
:description=,
|
|
35
|
+
:on_finish,
|
|
36
|
+
:on_finish=,
|
|
37
|
+
:on_success,
|
|
38
|
+
:on_success=,
|
|
39
|
+
:on_discard,
|
|
40
|
+
:on_discard=,
|
|
41
|
+
:callback_queue_name,
|
|
42
|
+
:callback_queue_name=,
|
|
43
|
+
:callback_priority,
|
|
44
|
+
:callback_priority=,
|
|
45
|
+
:properties,
|
|
46
|
+
:properties=,
|
|
47
|
+
:save,
|
|
48
|
+
:reload,
|
|
49
|
+
to: :record
|
|
50
|
+
)
|
|
51
|
+
|
|
52
|
+
# Create a new batch and enqueue it
|
|
53
|
+
# @param on_finish [String, Object] The class name of the callback job to be enqueued after the batch is finished
|
|
54
|
+
# @param properties [Hash] Additional properties to be stored on the batch
|
|
55
|
+
# @param block [Proc] Enqueue jobs within the block to add them to the batch
|
|
56
|
+
# @return [GoodJob::BatchRecord]
|
|
57
|
+
def self.enqueue(active_jobs = [], **properties, &block)
|
|
58
|
+
new.tap do |batch|
|
|
59
|
+
batch.enqueue(active_jobs, **properties, &block)
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
def self.find(id)
|
|
64
|
+
new _record: BatchRecord.find(id)
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
# Helper method to enqueue jobs and assign them to a batch
|
|
68
|
+
def self.within_thread(batch_id: nil, batch_callback_id: nil)
|
|
69
|
+
original_batch_id = current_batch_id
|
|
70
|
+
original_batch_callback_id = current_batch_callback_id
|
|
71
|
+
|
|
72
|
+
self.current_batch_id = batch_id
|
|
73
|
+
self.current_batch_callback_id = batch_callback_id
|
|
74
|
+
|
|
75
|
+
yield
|
|
76
|
+
ensure
|
|
77
|
+
self.current_batch_id = original_batch_id
|
|
78
|
+
self.current_batch_callback_id = original_batch_callback_id
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def initialize(_record: nil, **properties) # rubocop:disable Lint/UnderscorePrefixedVariableName
|
|
82
|
+
self.record = _record || BatchRecord.new
|
|
83
|
+
assign_properties(properties)
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
# @return [Array<ActiveJob::Base>] Active jobs added to the batch
|
|
87
|
+
def enqueue(active_jobs = [], **properties, &block)
|
|
88
|
+
assign_properties(properties)
|
|
89
|
+
record.save!
|
|
90
|
+
|
|
91
|
+
active_jobs = add(active_jobs, &block)
|
|
92
|
+
|
|
93
|
+
record.finished_at = nil
|
|
94
|
+
record.enqueued_at = Time.current if enqueued_at.nil?
|
|
95
|
+
record.save!
|
|
96
|
+
|
|
97
|
+
record._continue_discard_or_finish
|
|
98
|
+
active_jobs
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
# Enqueue jobs and add them to the batch
|
|
102
|
+
# @param block [Proc] Enqueue jobs within the block to add them to the batch
|
|
103
|
+
# @return [Array<ActiveJob::Base>] Active jobs added to the batch
|
|
104
|
+
def add(active_jobs = nil, &block)
|
|
105
|
+
record.save if record.new_record?
|
|
106
|
+
|
|
107
|
+
buffer = Bulk::Buffer.new
|
|
108
|
+
buffer.add(active_jobs)
|
|
109
|
+
buffer.capture(&block) if block
|
|
110
|
+
|
|
111
|
+
self.class.within_thread(batch_id: id) do
|
|
112
|
+
buffer.enqueue
|
|
113
|
+
end
|
|
114
|
+
|
|
115
|
+
buffer.active_jobs
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
def active_jobs
|
|
119
|
+
record.jobs.map(&:head_execution).map(&:active_job)
|
|
120
|
+
end
|
|
121
|
+
|
|
122
|
+
def callback_active_jobs
|
|
123
|
+
record.callback_jobs.map(&:head_execution).map(&:active_job)
|
|
124
|
+
end
|
|
125
|
+
|
|
126
|
+
def assign_properties(properties)
|
|
127
|
+
properties = properties.dup
|
|
128
|
+
batch_attrs = PROTECTED_PROPERTIES.index_with { |key| properties.delete(key) }.compact
|
|
129
|
+
record.assign_attributes(batch_attrs)
|
|
130
|
+
self.properties.merge!(properties)
|
|
131
|
+
end
|
|
132
|
+
|
|
133
|
+
def _record
|
|
134
|
+
record
|
|
135
|
+
end
|
|
136
|
+
|
|
137
|
+
private
|
|
138
|
+
|
|
139
|
+
attr_accessor :record
|
|
140
|
+
end
|
|
141
|
+
end
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module GoodJob
|
|
4
|
+
class BatchRecord < BaseRecord
|
|
5
|
+
include Lockable
|
|
6
|
+
|
|
7
|
+
self.table_name = 'good_job_batches'
|
|
8
|
+
|
|
9
|
+
has_many :jobs, class_name: 'GoodJob::Job', inverse_of: :batch, foreign_key: :batch_id, dependent: nil
|
|
10
|
+
has_many :executions, class_name: 'GoodJob::Execution', foreign_key: :batch_id, inverse_of: :batch, dependent: nil
|
|
11
|
+
has_many :callback_jobs, class_name: 'GoodJob::Job', foreign_key: :batch_callback_id, dependent: nil # rubocop:disable Rails/InverseOf
|
|
12
|
+
|
|
13
|
+
scope :finished, -> { where.not(finished_at: nil) }
|
|
14
|
+
scope :discarded, -> { where.not(discarded_at: nil) }
|
|
15
|
+
scope :not_discarded, -> { where(discarded_at: nil) }
|
|
16
|
+
scope :succeeded, -> { finished.not_discarded }
|
|
17
|
+
|
|
18
|
+
alias_attribute :enqueued?, :enqueued_at
|
|
19
|
+
alias_attribute :discarded?, :discarded_at
|
|
20
|
+
alias_attribute :finished?, :finished_at
|
|
21
|
+
|
|
22
|
+
scope :display_all, (lambda do |after_created_at: nil, after_id: nil|
|
|
23
|
+
query = order(created_at: :desc, id: :desc)
|
|
24
|
+
if after_created_at.present? && after_id.present?
|
|
25
|
+
query = query.where(Arel.sql('(created_at, id) < (:after_created_at, :after_id)'), after_created_at: after_created_at, after_id: after_id)
|
|
26
|
+
elsif after_created_at.present?
|
|
27
|
+
query = query.where(Arel.sql('(after_created_at) < (:after_created_at)'), after_created_at: after_created_at)
|
|
28
|
+
end
|
|
29
|
+
query
|
|
30
|
+
end)
|
|
31
|
+
|
|
32
|
+
# Whether the batch has finished and no jobs were discarded
|
|
33
|
+
# @return [Boolean]
|
|
34
|
+
def succeeded?
|
|
35
|
+
!discarded? && finished?
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def to_batch
|
|
39
|
+
Batch.new(_record: self)
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def display_attributes
|
|
43
|
+
attributes.except('serialized_properties').merge(properties: properties)
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
def _continue_discard_or_finish(execution = nil)
|
|
47
|
+
execution_discarded = execution && execution.error.present? && execution.retried_good_job_id.nil?
|
|
48
|
+
with_advisory_lock(function: "pg_advisory_lock") do
|
|
49
|
+
Batch.within_thread(batch_id: nil, batch_callback_id: id) do
|
|
50
|
+
if execution_discarded && discarded_at.blank?
|
|
51
|
+
update(discarded_at: Time.current)
|
|
52
|
+
on_discard.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :discard }) if on_discard.present?
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
if !finished_at && enqueued_at && jobs.where(finished_at: nil).count.zero?
|
|
56
|
+
update(finished_at: Time.current)
|
|
57
|
+
on_success.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :success }) if !discarded_at && on_success.present?
|
|
58
|
+
on_finish.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :finish }) if on_finish.present?
|
|
59
|
+
end
|
|
60
|
+
end
|
|
61
|
+
end
|
|
62
|
+
end
|
|
63
|
+
|
|
64
|
+
class PropertySerializer
|
|
65
|
+
def self.dump(value)
|
|
66
|
+
ActiveJob::Arguments.serialize([value]).first
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def self.load(value)
|
|
70
|
+
ActiveJob::Arguments.deserialize([value]).first
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
attribute :serialized_properties, :json, default: -> { {} } # Can be set as default value in `serialize` as of Rails v6.1
|
|
75
|
+
serialize :serialized_properties, PropertySerializer
|
|
76
|
+
alias_attribute :properties, :serialized_properties
|
|
77
|
+
|
|
78
|
+
def properties=(value)
|
|
79
|
+
raise ArgumentError, "Properties must be a Hash" unless value.is_a?(Hash)
|
|
80
|
+
|
|
81
|
+
self.serialized_properties = value
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
end
|
|
@@ -20,8 +20,12 @@ module GoodJob
|
|
|
20
20
|
self.table_name = 'good_jobs'
|
|
21
21
|
self.advisory_lockable_column = 'active_job_id'
|
|
22
22
|
|
|
23
|
+
define_model_callbacks :perform
|
|
23
24
|
define_model_callbacks :perform_unlocked, only: :after
|
|
24
25
|
|
|
26
|
+
set_callback :perform, :around, :reset_batch_values
|
|
27
|
+
set_callback :perform_unlocked, :after, :continue_discard_or_finish_batch
|
|
28
|
+
|
|
25
29
|
# Parse a string representing a group of queues into a more readable data
|
|
26
30
|
# structure.
|
|
27
31
|
# @param string [String] Queue string
|
|
@@ -65,6 +69,9 @@ module GoodJob
|
|
|
65
69
|
end
|
|
66
70
|
end
|
|
67
71
|
|
|
72
|
+
belongs_to :batch, class_name: 'GoodJob::BatchRecord', optional: true, inverse_of: :executions
|
|
73
|
+
belongs_to :batch_callback, class_name: 'GoodJob::Batch', optional: true
|
|
74
|
+
|
|
68
75
|
belongs_to :job, class_name: 'GoodJob::Job', foreign_key: 'active_job_id', primary_key: 'active_job_id', optional: true, inverse_of: :executions
|
|
69
76
|
after_destroy -> { self.class.active_job_id(active_job_id).delete_all }, if: -> { @_destroy_job }
|
|
70
77
|
|
|
@@ -206,14 +213,24 @@ module GoodJob
|
|
|
206
213
|
serialized_params: active_job.serialize,
|
|
207
214
|
scheduled_at: active_job.scheduled_at,
|
|
208
215
|
}
|
|
209
|
-
|
|
210
216
|
execution_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
|
|
211
217
|
|
|
212
|
-
|
|
218
|
+
reenqueued_current_execution = CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
|
219
|
+
current_execution = CurrentThread.execution
|
|
220
|
+
|
|
221
|
+
if reenqueued_current_execution
|
|
222
|
+
if GoodJob::BatchRecord.migrated?
|
|
223
|
+
execution_args[:batch_id] = current_execution.batch_id
|
|
224
|
+
execution_args[:batch_callback_id] = current_execution.batch_callback_id
|
|
225
|
+
end
|
|
226
|
+
execution_args[:cron_key] = current_execution.cron_key
|
|
227
|
+
else
|
|
228
|
+
if GoodJob::BatchRecord.migrated?
|
|
229
|
+
execution_args[:batch_id] = GoodJob::Batch.current_batch_id
|
|
230
|
+
execution_args[:batch_callback_id] = GoodJob::Batch.current_batch_callback_id
|
|
231
|
+
end
|
|
213
232
|
execution_args[:cron_key] = CurrentThread.cron_key
|
|
214
233
|
execution_args[:cron_at] = CurrentThread.cron_at
|
|
215
|
-
elsif CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
|
216
|
-
execution_args[:cron_key] = CurrentThread.execution.cron_key
|
|
217
234
|
end
|
|
218
235
|
|
|
219
236
|
new(**execution_args.merge(overrides))
|
|
@@ -283,7 +300,6 @@ module GoodJob
|
|
|
283
300
|
|
|
284
301
|
execution.save!
|
|
285
302
|
active_job.provider_job_id = execution.id
|
|
286
|
-
|
|
287
303
|
CurrentThread.execution.retried_good_job_id = execution.id if CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
|
288
304
|
|
|
289
305
|
execution
|
|
@@ -296,27 +312,29 @@ module GoodJob
|
|
|
296
312
|
# exception raised by the job, if any. If the job completed successfully,
|
|
297
313
|
# the second array entry (the exception) will be +nil+ and vice versa.
|
|
298
314
|
def perform
|
|
299
|
-
|
|
315
|
+
run_callbacks(:perform) do
|
|
316
|
+
raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
|
|
300
317
|
|
|
301
|
-
|
|
302
|
-
|
|
318
|
+
self.performed_at = Time.current
|
|
319
|
+
save! if GoodJob.preserve_job_records
|
|
303
320
|
|
|
304
|
-
|
|
321
|
+
result = execute
|
|
305
322
|
|
|
306
|
-
|
|
307
|
-
|
|
323
|
+
job_error = result.handled_error || result.unhandled_error
|
|
324
|
+
self.error = [job_error.class, ERROR_MESSAGE_SEPARATOR, job_error.message].join if job_error
|
|
308
325
|
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
326
|
+
reenqueued = result.retried? || retried_good_job_id.present?
|
|
327
|
+
if result.unhandled_error && GoodJob.retry_on_unhandled_error
|
|
328
|
+
save!
|
|
329
|
+
elsif GoodJob.preserve_job_records == true || reenqueued || (result.unhandled_error && GoodJob.preserve_job_records == :on_unhandled_error) || cron_key.present?
|
|
330
|
+
self.finished_at = Time.current
|
|
331
|
+
save!
|
|
332
|
+
else
|
|
333
|
+
destroy_job
|
|
334
|
+
end
|
|
318
335
|
|
|
319
|
-
|
|
336
|
+
result
|
|
337
|
+
end
|
|
320
338
|
end
|
|
321
339
|
|
|
322
340
|
# Tests whether this job is safe to be executed by this thread.
|
|
@@ -411,5 +429,13 @@ module GoodJob
|
|
|
411
429
|
end
|
|
412
430
|
end
|
|
413
431
|
end
|
|
432
|
+
|
|
433
|
+
def reset_batch_values(&block)
|
|
434
|
+
GoodJob::Batch.within_thread(batch_id: nil, batch_callback_id: nil, &block)
|
|
435
|
+
end
|
|
436
|
+
|
|
437
|
+
def continue_discard_or_finish_batch
|
|
438
|
+
batch._continue_discard_or_finish(self) if GoodJob::BatchRecord.migrated? && batch.present?
|
|
439
|
+
end
|
|
414
440
|
end
|
|
415
441
|
end
|
data/app/models/good_job/job.rb
CHANGED
|
@@ -28,6 +28,7 @@ module GoodJob
|
|
|
28
28
|
self.primary_key = 'active_job_id'
|
|
29
29
|
self.advisory_lockable_column = 'active_job_id'
|
|
30
30
|
|
|
31
|
+
belongs_to :batch, class_name: 'GoodJob::BatchRecord', inverse_of: :jobs, optional: true
|
|
31
32
|
has_many :executions, -> { order(created_at: :asc) }, class_name: 'GoodJob::Execution', foreign_key: 'active_job_id', inverse_of: :job # rubocop:disable Rails/HasManyOrHasOneDependent
|
|
32
33
|
|
|
33
34
|
# Only the most-recent unretried execution represents a "Job"
|
|
@@ -0,0 +1,108 @@
|
|
|
1
|
+
<div class="my-3 card" data-gj-poll-replace id="jobs-table">
|
|
2
|
+
<div class="list-group list-group-flush text-nowrap table-jobs" role="table">
|
|
3
|
+
<header class="list-group-item bg-light">
|
|
4
|
+
<div class="row small text-muted text-uppercase align-items-center">
|
|
5
|
+
<div class="col-4">Jobs</div>
|
|
6
|
+
<div class="col-1">Queue</div>
|
|
7
|
+
<div class="col-1">Priority</div>
|
|
8
|
+
<div class="col-1 text-end">Attempts</div>
|
|
9
|
+
<div class="col text-end">
|
|
10
|
+
<%= tag.button type: "button", class: "btn btn-sm text-muted", role: "button",
|
|
11
|
+
data: { bs_toggle: "collapse", bs_target: ".job-params" },
|
|
12
|
+
aria: { expanded: false, controls: jobs.map { |job| "##{dom_id(job, "params")}" }.join(" ") } do %>
|
|
13
|
+
<%= render_icon "info" %>
|
|
14
|
+
<span class="visually-hidden">Inspect</span>
|
|
15
|
+
<% end %>
|
|
16
|
+
</div>
|
|
17
|
+
</div>
|
|
18
|
+
</header>
|
|
19
|
+
|
|
20
|
+
<% if jobs.present? %>
|
|
21
|
+
<% jobs.each do |job| %>
|
|
22
|
+
<div role="row" class="list-group-item list-group-item-action py-3">
|
|
23
|
+
<div class="row align-items-center">
|
|
24
|
+
<div class="col-4">
|
|
25
|
+
<%= tag.code link_to(job.id, job_path(job), class: "small text-muted text-decoration-none") %>
|
|
26
|
+
<%= tag.h5 tag.code(link_to(job.job_class, job_path(job), class: "text-reset text-decoration-none")), class: "text-reset mb-0" %>
|
|
27
|
+
</div>
|
|
28
|
+
<div class="col-1">
|
|
29
|
+
<span class="badge bg-primary bg-opacity-25 text-dark font-monospace"><%= job.queue_name %></span>
|
|
30
|
+
</div>
|
|
31
|
+
<div class="col-1 small text-center">
|
|
32
|
+
<span class="font-monospace fw-bold"><%= job.priority %></span>
|
|
33
|
+
</div>
|
|
34
|
+
<div class="col-1 text-center">
|
|
35
|
+
<% if job.executions_count > 0 && job.status != :finished %>
|
|
36
|
+
<%= tag.span job.executions_count, class: "badge rounded-pill bg-danger", data: {
|
|
37
|
+
bs_toggle: "popover",
|
|
38
|
+
bs_trigger: "hover focus click",
|
|
39
|
+
bs_placement: "bottom",
|
|
40
|
+
bs_content: job.recent_error
|
|
41
|
+
} %>
|
|
42
|
+
<% else %>
|
|
43
|
+
<span class="badge bg-secondary bg-opacity-50 rounded-pill"><%= job.executions_count %></span>
|
|
44
|
+
<% end %>
|
|
45
|
+
</div>
|
|
46
|
+
<div class="col d-flex gap-3 align-items-center justify-content-end">
|
|
47
|
+
<%= tag.span relative_time(job.last_status_at), class: "small" %>
|
|
48
|
+
<%= status_badge job.status %>
|
|
49
|
+
</div>
|
|
50
|
+
<div class="col-auto">
|
|
51
|
+
<div class="dropdown float-end">
|
|
52
|
+
<button class="d-flex align-items-center btn btn-sm" type="button" id="<%= dom_id(job, :actions) %>" data-bs-toggle="dropdown" aria-expanded="false">
|
|
53
|
+
<%= render "good_job/shared/icons/dots" %>
|
|
54
|
+
<span class="visually-hidden">Actions</span>
|
|
55
|
+
</button>
|
|
56
|
+
<ul class="dropdown-menu shadow" aria-labelledby="<%= dom_id(job, :actions) %>">
|
|
57
|
+
<li>
|
|
58
|
+
<% job_reschedulable = job.status.in? [:scheduled, :retried, :queued] %>
|
|
59
|
+
<%= link_to reschedule_job_path(job.id), method: :put, class: "dropdown-item #{'disabled' unless job_reschedulable}", title: "Reschedule job", data: { confirm: "Confirm reschedule", disable: true } do %>
|
|
60
|
+
<%= render "good_job/shared/icons/skip_forward" %>
|
|
61
|
+
Reschedule
|
|
62
|
+
<% end %>
|
|
63
|
+
</li>
|
|
64
|
+
<li>
|
|
65
|
+
<% job_discardable = job.status.in? [:scheduled, :retried, :queued] %>
|
|
66
|
+
<%= link_to discard_job_path(job.id), method: :put, class: "dropdown-item #{'disabled' unless job_discardable}", title: "Discard job", data: { confirm: "Confirm discard", disable: true } do %>
|
|
67
|
+
<%= render "good_job/shared/icons/stop" %>
|
|
68
|
+
Discard
|
|
69
|
+
<% end %>
|
|
70
|
+
</li>
|
|
71
|
+
<li>
|
|
72
|
+
<%= link_to retry_job_path(job.id), method: :put, class: "dropdown-item #{'disabled' unless job.status == :discarded}", title: "Retry job", data: { confirm: "Confirm retry", disable: true } do %>
|
|
73
|
+
<%= render "good_job/shared/icons/arrow_clockwise" %>
|
|
74
|
+
Retry
|
|
75
|
+
<% end %>
|
|
76
|
+
</li>
|
|
77
|
+
<li>
|
|
78
|
+
<%= link_to job_path(job.id), method: :delete, class: "dropdown-item #{'disabled' unless job.status.in? [:discarded, :finished]}", title: "Destroy job", data: { confirm: "Confirm destroy", disable: true } do %>
|
|
79
|
+
<%= render_icon "trash" %>
|
|
80
|
+
Destroy
|
|
81
|
+
<% end %>
|
|
82
|
+
</li>
|
|
83
|
+
|
|
84
|
+
<li>
|
|
85
|
+
<%= link_to "##{dom_id(job, 'params')}",
|
|
86
|
+
class: "dropdown-item",
|
|
87
|
+
data: { bs_toggle: "collapse" },
|
|
88
|
+
aria: { expanded: false, controls: dom_id(job, "params") } do %>
|
|
89
|
+
<%= render_icon "info" %>
|
|
90
|
+
Inspect
|
|
91
|
+
<% end %>
|
|
92
|
+
</li>
|
|
93
|
+
</ul>
|
|
94
|
+
</div>
|
|
95
|
+
</div>
|
|
96
|
+
</div>
|
|
97
|
+
</div>
|
|
98
|
+
<%= tag.div id: dom_id(job, "params"), class: "job-params list-group-item collapse small bg-dark text-light" do %>
|
|
99
|
+
<%= tag.pre JSON.pretty_generate(job.display_serialized_params) %>
|
|
100
|
+
<% end %>
|
|
101
|
+
<% end %>
|
|
102
|
+
<% else %>
|
|
103
|
+
<div class="list-group-item py-4 text-center text-muted">
|
|
104
|
+
No jobs found.
|
|
105
|
+
</div>
|
|
106
|
+
<% end %>
|
|
107
|
+
</>
|
|
108
|
+
</div>
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
<div class="my-3 card" data-gj-poll-replace id="batches-table">
|
|
2
|
+
<div class="list-group list-group-flush text-nowrap table-batches" role="table">
|
|
3
|
+
<header class="list-group-item bg-light">
|
|
4
|
+
<div class="row small text-muted text-uppercase align-items-center">
|
|
5
|
+
<div class="col-4">Name</div>
|
|
6
|
+
<div class="col-1">Created</div>
|
|
7
|
+
<div class="col-1">Enqueued</div>
|
|
8
|
+
<div class="col-1">Discarded</div>
|
|
9
|
+
<div class="col-1">Finished</div>
|
|
10
|
+
<div class="col">Jobs</div>
|
|
11
|
+
<div class="col text-end">
|
|
12
|
+
<%= tag.button type: "button", class: "btn btn-sm text-muted", role: "button",
|
|
13
|
+
data: { bs_toggle: "collapse", bs_target: ".batch-properties" },
|
|
14
|
+
aria: { expanded: false, controls: batches.map { |batch| "##{dom_id(batch, "params")}" }.join(" ") } do %>
|
|
15
|
+
<%= render_icon "info" %>
|
|
16
|
+
<span class="visually-hidden">Inspect</span>
|
|
17
|
+
<% end %>
|
|
18
|
+
</div>
|
|
19
|
+
</div>
|
|
20
|
+
</header>
|
|
21
|
+
|
|
22
|
+
<% if batches.present? %>
|
|
23
|
+
<% batches.each do |batch| %>
|
|
24
|
+
<div id="<%= dom_id(batch) %>" class="list-group-item py-3" role="row">
|
|
25
|
+
<div class="row align-items-center">
|
|
26
|
+
<div class="col-4">
|
|
27
|
+
<%= link_to batch_path(batch), class: "text-decoration-none" do %>
|
|
28
|
+
<code class="small text-muted">
|
|
29
|
+
<%= batch.id %>
|
|
30
|
+
</code>
|
|
31
|
+
<h5 class=""><code><%= batch.on_finish %></code></h5>
|
|
32
|
+
<div class="text-muted"><%= batch.description %></div>
|
|
33
|
+
<% end %>
|
|
34
|
+
</div>
|
|
35
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.created_at) %></div>
|
|
36
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.enqueued_at) if batch.enqueued_at %></div>
|
|
37
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.discarded_at) if batch.discarded_at %></div>
|
|
38
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.finished_at) if batch.finished_at %></div>
|
|
39
|
+
<div class="col"><%= batch.jobs.count %></div>
|
|
40
|
+
<div class="col text-end">
|
|
41
|
+
<%= tag.button type: "button", class: "btn btn-sm text-muted ms-auto", role: "button",
|
|
42
|
+
title: "Inspect",
|
|
43
|
+
data: { bs_toggle: "collapse", bs_target: "##{dom_id(batch, 'properties')}" },
|
|
44
|
+
aria: { expanded: false, controls: dom_id(batch, "state") } do %>
|
|
45
|
+
<%= render_icon "info" %>
|
|
46
|
+
<span class="visually-hidden">Inspect</span>
|
|
47
|
+
<% end %>
|
|
48
|
+
</div>
|
|
49
|
+
</div>
|
|
50
|
+
</div>
|
|
51
|
+
<%= tag.div id: dom_id(batch, "properties"), class: "batch-properties list-group-item collapse small bg-dark text-light" do %>
|
|
52
|
+
<%= tag.pre JSON.pretty_generate(batch.properties) %>
|
|
53
|
+
<% end %>
|
|
54
|
+
<% end %>
|
|
55
|
+
<% else %>
|
|
56
|
+
<div class="list-group-item py-4 text-center text-muted">
|
|
57
|
+
No batches found.
|
|
58
|
+
</div>
|
|
59
|
+
<% end %>
|
|
60
|
+
</div>
|
|
61
|
+
</div>
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
<% if GoodJob::BatchRecord.migrated? %>
|
|
2
|
+
<%= render 'good_job/batches/table', batches: @filter.records, filter: @filter %>
|
|
3
|
+
<% if @filter.records.present? %>
|
|
4
|
+
<nav aria-label="Batch pagination" class="mt-3">
|
|
5
|
+
<ul class="pagination">
|
|
6
|
+
<li class="page-item">
|
|
7
|
+
<%= link_to(@filter.to_params(after_created_at: @filter.last.created_at, after_id: @filter.last.id), class: "page-link") do %>
|
|
8
|
+
Older batches <span aria-hidden="true">»</span>
|
|
9
|
+
<% end %>
|
|
10
|
+
</li>
|
|
11
|
+
</ul>
|
|
12
|
+
</nav>
|
|
13
|
+
<% end %>
|
|
14
|
+
<% else %>
|
|
15
|
+
<h3 class="text-center my-5">GoodJob has pending database migrations.</h3>
|
|
16
|
+
<% end %>
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
<div class="break-out bg-light border-bottom py-2 mb-3">
|
|
2
|
+
<div class="container-fluid pt-2">
|
|
3
|
+
<div class="row align-items-center">
|
|
4
|
+
<div class="col-5">
|
|
5
|
+
<nav aria-label="breadcrumb">
|
|
6
|
+
<ol class="breadcrumb small mb-0">
|
|
7
|
+
<li class="breadcrumb-item"><%= link_to "Batches", batches_path %></li>
|
|
8
|
+
<li class="breadcrumb-item active" aria-current="page"><%= tag.code @batch.id, class: "text-muted" %></li>
|
|
9
|
+
</ol>
|
|
10
|
+
<h2 class="h5 mt-2"><%= @batch.description %></h2>
|
|
11
|
+
</nav>
|
|
12
|
+
</div>
|
|
13
|
+
</div>
|
|
14
|
+
</div>
|
|
15
|
+
</div>
|
|
16
|
+
|
|
17
|
+
<div class="my-4">
|
|
18
|
+
<h5>Attributes</h5>
|
|
19
|
+
<%= tag.pre JSON.pretty_generate @batch.display_attributes, class: 'text-wrap text-break' %>
|
|
20
|
+
</div>
|
|
21
|
+
|
|
22
|
+
<div class="my-4">
|
|
23
|
+
<h5>Callback Jobs</h5>
|
|
24
|
+
<%= render 'jobs', jobs: @batch.callback_jobs.reverse %>
|
|
25
|
+
</div>
|
|
26
|
+
|
|
27
|
+
<div class="my-4">
|
|
28
|
+
<h5>Batched Jobs</h5>
|
|
29
|
+
<%= render 'jobs', jobs: @batch.jobs.reverse %>
|
|
30
|
+
</div>
|
|
@@ -14,6 +14,13 @@
|
|
|
14
14
|
<span class="badge bg-secondary rounded-pill"><%= number_to_human(jobs_count) %></span>
|
|
15
15
|
<% end %>
|
|
16
16
|
</li>
|
|
17
|
+
<li class="nav-item">
|
|
18
|
+
<%= link_to batches_path, class: ["nav-link", ("active" if controller_name == 'batches')] do %>
|
|
19
|
+
<%= "Batches" %>
|
|
20
|
+
<% batches_count = GoodJob::BatchRecord.migrated? ? GoodJob::BatchRecord.all.size : 0 %>
|
|
21
|
+
<span class="badge bg-secondary rounded-pill"><%= batches_count %></span>
|
|
22
|
+
<% end %>
|
|
23
|
+
</li>
|
|
17
24
|
<li class="nav-item">
|
|
18
25
|
<%= link_to cron_entries_path, class: ["nav-link", ("active" if controller_name == 'cron_entries')] do %>
|
|
19
26
|
<%= t(".cron_schedules") %>
|
data/config/routes.rb
CHANGED
|
@@ -19,6 +19,23 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
|
|
|
19
19
|
t.text :cron_key
|
|
20
20
|
t.uuid :retried_good_job_id
|
|
21
21
|
t.datetime :cron_at
|
|
22
|
+
|
|
23
|
+
t.uuid :batch_id
|
|
24
|
+
t.uuid :batch_callback_id
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
create_table :good_job_batches, id: :uuid do |t|
|
|
28
|
+
t.timestamps
|
|
29
|
+
t.text :description
|
|
30
|
+
t.jsonb :serialized_properties
|
|
31
|
+
t.text :on_finish
|
|
32
|
+
t.text :on_success
|
|
33
|
+
t.text :on_discard
|
|
34
|
+
t.text :callback_queue_name
|
|
35
|
+
t.integer :callback_priority
|
|
36
|
+
t.datetime :enqueued_at
|
|
37
|
+
t.datetime :discarded_at
|
|
38
|
+
t.datetime :finished_at
|
|
22
39
|
end
|
|
23
40
|
|
|
24
41
|
create_table :good_job_processes, id: :uuid do |t|
|
|
@@ -43,5 +60,7 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
|
|
|
43
60
|
add_index :good_jobs, [:finished_at], where: "retried_good_job_id IS NULL AND finished_at IS NOT NULL", name: :index_good_jobs_jobs_on_finished_at
|
|
44
61
|
add_index :good_jobs, [:priority, :created_at], order: { priority: "DESC NULLS LAST", created_at: :asc },
|
|
45
62
|
where: "finished_at IS NULL", name: :index_good_jobs_jobs_on_priority_created_at_when_unfinished
|
|
63
|
+
add_index :good_jobs, [:batch_id], where: "batch_id IS NOT NULL"
|
|
64
|
+
add_index :good_jobs, [:batch_callback_id], where: "batch_callback_id IS NOT NULL"
|
|
46
65
|
end
|
|
47
66
|
end
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
class CreateGoodJobBatches < ActiveRecord::Migration<%= migration_version %>
|
|
3
|
+
def change
|
|
4
|
+
reversible do |dir|
|
|
5
|
+
dir.up do
|
|
6
|
+
# Ensure this incremental update migration is idempotent
|
|
7
|
+
# with monolithic install migration.
|
|
8
|
+
return if connection.table_exists?(:good_job_batches)
|
|
9
|
+
end
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
create_table :good_job_batches, id: :uuid do |t|
|
|
13
|
+
t.timestamps
|
|
14
|
+
t.text :description
|
|
15
|
+
t.jsonb :serialized_properties
|
|
16
|
+
t.text :on_finish
|
|
17
|
+
t.text :on_success
|
|
18
|
+
t.text :on_discard
|
|
19
|
+
t.text :callback_queue_name
|
|
20
|
+
t.integer :callback_priority
|
|
21
|
+
t.datetime :enqueued_at
|
|
22
|
+
t.datetime :discarded_at
|
|
23
|
+
t.datetime :finished_at
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
change_table :good_jobs do |t|
|
|
27
|
+
t.uuid :batch_id
|
|
28
|
+
t.uuid :batch_callback_id
|
|
29
|
+
|
|
30
|
+
t.index :batch_id, where: "batch_id IS NOT NULL"
|
|
31
|
+
t.index :batch_callback_id, where: "batch_callback_id IS NOT NULL"
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
end
|
data/lib/good_job/bulk.rb
CHANGED
|
@@ -15,19 +15,14 @@ module GoodJob
|
|
|
15
15
|
# @param active_jobs [Array<ActiveJob::Base>] Active Jobs to be buffered.
|
|
16
16
|
# @param queue_adapter Override the jobs implict queue adapter with an explicit one.
|
|
17
17
|
# @return [nil, Array<ActiveJob::Base>] The ActiveJob instances that have been buffered; nil if no active buffer
|
|
18
|
-
def self.capture(active_jobs = nil, queue_adapter: nil)
|
|
19
|
-
raise(ArgumentError, "Use either the block form or the argument form, not both") if
|
|
20
|
-
|
|
21
|
-
if
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
current_buffer.active_jobs
|
|
27
|
-
ensure
|
|
28
|
-
self.current_buffer = original_buffer
|
|
29
|
-
end
|
|
30
|
-
else
|
|
18
|
+
def self.capture(active_jobs = nil, queue_adapter: nil, &block)
|
|
19
|
+
raise(ArgumentError, "Use either the block form or the argument form, not both") if block && active_jobs
|
|
20
|
+
|
|
21
|
+
if block
|
|
22
|
+
buffer = Buffer.new
|
|
23
|
+
buffer.capture(&block)
|
|
24
|
+
buffer.active_jobs
|
|
25
|
+
elsif current_buffer
|
|
31
26
|
current_buffer&.add(active_jobs, queue_adapter: queue_adapter)
|
|
32
27
|
end
|
|
33
28
|
end
|
|
@@ -35,16 +30,15 @@ module GoodJob
|
|
|
35
30
|
# Capture jobs to a buffer and enqueue them all at once; or enqueue the current buffer.
|
|
36
31
|
# @param active_jobs [Array<ActiveJob::Base>] Active Jobs to be enqueued.
|
|
37
32
|
# @return [Array<ActiveJob::Base>] The ActiveJob instances that have been captured; check provider_job_id to confirm enqueued.
|
|
38
|
-
def self.enqueue(active_jobs = nil)
|
|
39
|
-
raise(ArgumentError, "Use either the block form or the argument form, not both") if
|
|
33
|
+
def self.enqueue(active_jobs = nil, &block)
|
|
34
|
+
raise(ArgumentError, "Use either the block form or the argument form, not both") if block && active_jobs
|
|
40
35
|
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
36
|
+
buffer = Buffer.new
|
|
37
|
+
if block
|
|
38
|
+
buffer.capture(&block)
|
|
39
|
+
buffer.enqueue
|
|
40
|
+
buffer.active_jobs
|
|
46
41
|
elsif active_jobs.present?
|
|
47
|
-
buffer = Buffer.new
|
|
48
42
|
buffer.add(active_jobs)
|
|
49
43
|
buffer.enqueue
|
|
50
44
|
buffer.active_jobs
|
|
@@ -69,8 +63,18 @@ module GoodJob
|
|
|
69
63
|
@values = []
|
|
70
64
|
end
|
|
71
65
|
|
|
66
|
+
def capture
|
|
67
|
+
original_buffer = Bulk.current_buffer
|
|
68
|
+
Bulk.current_buffer = self
|
|
69
|
+
yield
|
|
70
|
+
ensure
|
|
71
|
+
Bulk.current_buffer = original_buffer
|
|
72
|
+
end
|
|
73
|
+
|
|
72
74
|
def add(active_jobs, queue_adapter: nil)
|
|
73
75
|
new_pairs = Array(active_jobs).map do |active_job|
|
|
76
|
+
raise ArgumentError, "Expected an ActiveJob::Base instance, got #{active_job.class}" unless active_job.is_a?(ActiveJob::Base)
|
|
77
|
+
|
|
74
78
|
adapter = queue_adapter || active_job.class.queue_adapter
|
|
75
79
|
raise Error, "Jobs must have a Queue Adapter" unless adapter
|
|
76
80
|
|
|
@@ -156,7 +156,7 @@ module GoodJob
|
|
|
156
156
|
destroyed_records_count = event.payload[:destroyed_records_count]
|
|
157
157
|
|
|
158
158
|
info do
|
|
159
|
-
"GoodJob destroyed #{destroyed_records_count} preserved #{'
|
|
159
|
+
"GoodJob destroyed #{destroyed_records_count} preserved job execution #{'records'.pluralize(destroyed_records_count)} finished before #{timestamp}."
|
|
160
160
|
end
|
|
161
161
|
end
|
|
162
162
|
|
data/lib/good_job/version.rb
CHANGED
data/lib/good_job.rb
CHANGED
|
@@ -8,6 +8,7 @@ require "good_job/engine"
|
|
|
8
8
|
require "good_job/adapter"
|
|
9
9
|
require "active_job/queue_adapters/good_job_adapter"
|
|
10
10
|
require "good_job/active_job_extensions/concurrency"
|
|
11
|
+
require "good_job/active_job_extensions/batches"
|
|
11
12
|
|
|
12
13
|
require "good_job/assignable_connection"
|
|
13
14
|
require "good_job/bulk"
|
|
@@ -143,7 +144,7 @@ module GoodJob
|
|
|
143
144
|
end
|
|
144
145
|
end
|
|
145
146
|
|
|
146
|
-
# Destroys preserved job records.
|
|
147
|
+
# Destroys preserved job and batch records.
|
|
147
148
|
# By default, GoodJob destroys job records when the job is performed and this
|
|
148
149
|
# method is not necessary. However, when `GoodJob.preserve_job_records = true`,
|
|
149
150
|
# the jobs will be preserved in the database. This is useful when wanting to
|
|
@@ -151,7 +152,7 @@ module GoodJob
|
|
|
151
152
|
# If you are preserving job records this way, use this method regularly to
|
|
152
153
|
# destroy old records and preserve space in your database.
|
|
153
154
|
# @params older_than [nil,Numeric,ActiveSupport::Duration] Jobs older than this will be destroyed (default: +86400+).
|
|
154
|
-
# @return [Integer] Number of
|
|
155
|
+
# @return [Integer] Number of job execution records and batches that were destroyed.
|
|
155
156
|
def self.cleanup_preserved_jobs(older_than: nil)
|
|
156
157
|
older_than ||= GoodJob.configuration.cleanup_preserved_jobs_before_seconds_ago
|
|
157
158
|
timestamp = Time.current - older_than
|
|
@@ -160,10 +161,19 @@ module GoodJob
|
|
|
160
161
|
ActiveSupport::Notifications.instrument("cleanup_preserved_jobs.good_job", { older_than: older_than, timestamp: timestamp }) do |payload|
|
|
161
162
|
old_jobs = GoodJob::Job.where('finished_at <= ?', timestamp)
|
|
162
163
|
old_jobs = old_jobs.succeeded unless include_discarded
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
GoodJob::
|
|
166
|
-
|
|
164
|
+
deleted_executions_count = GoodJob::Execution.where(job: old_jobs).delete_all
|
|
165
|
+
|
|
166
|
+
if GoodJob::BatchRecord.migrated?
|
|
167
|
+
old_batches = GoodJob::BatchRecord.where('finished_at <= ?', timestamp)
|
|
168
|
+
old_batches = old_batches.succeeded unless include_discarded
|
|
169
|
+
deleted_batches_count = old_batches.delete_all
|
|
170
|
+
else
|
|
171
|
+
deleted_batches_count = 0
|
|
172
|
+
end
|
|
173
|
+
|
|
174
|
+
payload[:destroyed_executions_count] = deleted_executions_count
|
|
175
|
+
payload[:destroyed_batches_count] = deleted_batches_count
|
|
176
|
+
payload[:destroyed_records_count] = deleted_executions_count + deleted_batches_count
|
|
167
177
|
end
|
|
168
178
|
end
|
|
169
179
|
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: good_job
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 3.
|
|
4
|
+
version: 3.10.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ben Sheldon
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2023-
|
|
11
|
+
date: 2023-02-04 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: activejob
|
|
@@ -334,16 +334,20 @@ files:
|
|
|
334
334
|
- app/charts/good_job/scheduled_by_queue_chart.rb
|
|
335
335
|
- app/controllers/good_job/application_controller.rb
|
|
336
336
|
- app/controllers/good_job/assets_controller.rb
|
|
337
|
+
- app/controllers/good_job/batches_controller.rb
|
|
337
338
|
- app/controllers/good_job/cron_entries_controller.rb
|
|
338
339
|
- app/controllers/good_job/jobs_controller.rb
|
|
339
340
|
- app/controllers/good_job/processes_controller.rb
|
|
340
341
|
- app/filters/good_job/base_filter.rb
|
|
342
|
+
- app/filters/good_job/batches_filter.rb
|
|
341
343
|
- app/filters/good_job/jobs_filter.rb
|
|
342
344
|
- app/helpers/good_job/application_helper.rb
|
|
343
345
|
- app/models/concerns/good_job/filterable.rb
|
|
344
346
|
- app/models/concerns/good_job/reportable.rb
|
|
345
347
|
- app/models/good_job/active_job_job.rb
|
|
346
348
|
- app/models/good_job/base_record.rb
|
|
349
|
+
- app/models/good_job/batch.rb
|
|
350
|
+
- app/models/good_job/batch_record.rb
|
|
347
351
|
- app/models/good_job/cron_entry.rb
|
|
348
352
|
- app/models/good_job/execution.rb
|
|
349
353
|
- app/models/good_job/execution_result.rb
|
|
@@ -351,6 +355,10 @@ files:
|
|
|
351
355
|
- app/models/good_job/lockable.rb
|
|
352
356
|
- app/models/good_job/process.rb
|
|
353
357
|
- app/models/good_job/setting.rb
|
|
358
|
+
- app/views/good_job/batches/_jobs.erb
|
|
359
|
+
- app/views/good_job/batches/_table.erb
|
|
360
|
+
- app/views/good_job/batches/index.html.erb
|
|
361
|
+
- app/views/good_job/batches/show.html.erb
|
|
354
362
|
- app/views/good_job/cron_entries/index.html.erb
|
|
355
363
|
- app/views/good_job/cron_entries/show.html.erb
|
|
356
364
|
- app/views/good_job/jobs/_executions.erb
|
|
@@ -390,8 +398,10 @@ files:
|
|
|
390
398
|
- lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb.erb
|
|
391
399
|
- lib/generators/good_job/templates/update/migrations/02_create_good_job_settings.rb.erb
|
|
392
400
|
- lib/generators/good_job/templates/update/migrations/03_create_index_good_jobs_jobs_on_priority_created_at_when_unfinished.rb.erb
|
|
401
|
+
- lib/generators/good_job/templates/update/migrations/04_create_good_job_batches.rb.erb
|
|
393
402
|
- lib/generators/good_job/update_generator.rb
|
|
394
403
|
- lib/good_job.rb
|
|
404
|
+
- lib/good_job/active_job_extensions/batches.rb
|
|
395
405
|
- lib/good_job/active_job_extensions/concurrency.rb
|
|
396
406
|
- lib/good_job/adapter.rb
|
|
397
407
|
- lib/good_job/assignable_connection.rb
|