good_job 3.9.0 → 3.10.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +31 -0
- data/README.md +194 -23
- data/app/controllers/good_job/batches_controller.rb +12 -0
- data/app/filters/good_job/batches_filter.rb +21 -0
- data/app/models/good_job/batch.rb +149 -0
- data/app/models/good_job/batch_record.rb +95 -0
- data/app/models/good_job/execution.rb +47 -21
- data/app/models/good_job/job.rb +1 -0
- data/app/views/good_job/batches/_jobs.erb +108 -0
- data/app/views/good_job/batches/_table.erb +61 -0
- data/app/views/good_job/batches/index.html.erb +16 -0
- data/app/views/good_job/batches/show.html.erb +30 -0
- data/app/views/good_job/shared/_navbar.erb +7 -0
- data/config/routes.rb +2 -0
- data/lib/generators/good_job/templates/install/migrations/create_good_jobs.rb.erb +19 -0
- data/lib/generators/good_job/templates/update/migrations/04_create_good_job_batches.rb.erb +34 -0
- data/lib/good_job/active_job_extensions/batches.rb +13 -0
- data/lib/good_job/bulk.rb +25 -21
- data/lib/good_job/log_subscriber.rb +1 -1
- data/lib/good_job/version.rb +1 -1
- data/lib/good_job.rb +16 -6
- metadata +12 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 3e632ae76e69e8601479181fb0093e9c6777510ac0743184a8f578b3c4dacee8
|
4
|
+
data.tar.gz: 577f36ad182e6f4ff841abcd073925f4477f1086b1a0c7a454c12037f781ace7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f9211423d43cee468602aa1476d0d4b71a3831448157dc7dc540f98e0d1cb476e60bed884679ef10733af21c21e1c595dac14dc2d1c0b25d4b49b8d246f7be98
|
7
|
+
data.tar.gz: 25a51a4196ea6618ca44ac73ed6cb9f97b63d99596bd144a9d51f439033a22d15ab16a73cd9d48954dc9b447b19cc9f03a21a9e5667d5021c85b5510ad7ca577
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,36 @@
|
|
1
1
|
# Changelog
|
2
2
|
|
3
|
+
## [v3.10.1](https://github.com/bensheldon/good_job/tree/v3.10.1) (2023-02-06)
|
4
|
+
|
5
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.10.0...v3.10.1)
|
6
|
+
|
7
|
+
**Fixed bugs:**
|
8
|
+
|
9
|
+
- Ensure batch is reloaded before updating on multiple enqueues [\#824](https://github.com/bensheldon/good_job/pull/824) ([bensheldon](https://github.com/bensheldon))
|
10
|
+
|
11
|
+
**Closed issues:**
|
12
|
+
|
13
|
+
- Can't batch.enqueue the callback after retrying a job within the batch [\#822](https://github.com/bensheldon/good_job/issues/822)
|
14
|
+
|
15
|
+
**Merged pull requests:**
|
16
|
+
|
17
|
+
- In tests, retry when connecting to Puma returns Net::ReadTimeout [\#825](https://github.com/bensheldon/good_job/pull/825) ([bensheldon](https://github.com/bensheldon))
|
18
|
+
- Add Batch enqueue example to Demo's cron configuration [\#823](https://github.com/bensheldon/good_job/pull/823) ([bensheldon](https://github.com/bensheldon))
|
19
|
+
|
20
|
+
## [v3.10.0](https://github.com/bensheldon/good_job/tree/v3.10.0) (2023-02-04)
|
21
|
+
|
22
|
+
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.9.0...v3.10.0)
|
23
|
+
|
24
|
+
**Implemented enhancements:**
|
25
|
+
|
26
|
+
- Implement `GoodJob::Batch` [\#712](https://github.com/bensheldon/good_job/pull/712) ([bensheldon](https://github.com/bensheldon))
|
27
|
+
|
28
|
+
**Closed issues:**
|
29
|
+
|
30
|
+
- Support for Ruby 3.2 [\#785](https://github.com/bensheldon/good_job/issues/785)
|
31
|
+
- Custom table names [\#748](https://github.com/bensheldon/good_job/issues/748)
|
32
|
+
- Health check issue with cron scheduler job [\#741](https://github.com/bensheldon/good_job/issues/741)
|
33
|
+
|
3
34
|
## [v3.9.0](https://github.com/bensheldon/good_job/tree/v3.9.0) (2023-01-31)
|
4
35
|
|
5
36
|
[Full Changelog](https://github.com/bensheldon/good_job/compare/v3.8.0...v3.9.0)
|
data/README.md
CHANGED
@@ -41,9 +41,11 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
41
41
|
- [Dashboard](#dashboard)
|
42
42
|
- [API-only Rails applications](#api-only-rails-applications)
|
43
43
|
- [Live Polling](#live-polling)
|
44
|
-
- [
|
44
|
+
- [Concurrency controls](#concurrency-controls)
|
45
45
|
- [How concurrency controls work](#how-concurrency-controls-work)
|
46
46
|
- [Cron-style repeating/recurring jobs](#cron-style-repeatingrecurring-jobs)
|
47
|
+
- [Bulk enqueue](#bulk-enqueue)
|
48
|
+
- [Batches](#batches)
|
47
49
|
- [Updating](#updating)
|
48
50
|
- [Upgrading minor versions](#upgrading-minor-versions)
|
49
51
|
- [Upgrading v2 to v3](#upgrading-v2-to-v3)
|
@@ -58,7 +60,6 @@ For more of the story of GoodJob, read the [introductory blog post](https://isla
|
|
58
60
|
- [Database connections](#database-connections)
|
59
61
|
- [Production setup](#production-setup)
|
60
62
|
- [Queue performance with Queue Select Limit](#queue-performance-with-queue-select-limit)
|
61
|
-
- [Bulk enqueue](#bulk-enqueue)
|
62
63
|
- [Execute jobs async / in-process](#execute-jobs-async--in-process)
|
63
64
|
- [Migrate to GoodJob from a different ActiveJob backend](#migrate-to-goodjob-from-a-different-activejob-backend)
|
64
65
|
- [Monitor and preserve worked jobs](#monitor-and-preserve-worked-jobs)
|
@@ -392,7 +393,7 @@ end
|
|
392
393
|
|
393
394
|
The Dashboard can be set to automatically refresh by checking "Live Poll" in the Dashboard header, or by setting `?poll=10` with the interval in seconds (default 30 seconds).
|
394
395
|
|
395
|
-
###
|
396
|
+
### Concurrency controls
|
396
397
|
|
397
398
|
GoodJob can extend ActiveJob to provide limits on concurrently running jobs, either at time of _enqueue_ or at _perform_. Limiting concurrency can help prevent duplicate, double or unnecessary jobs from being enqueued, or race conditions when performing, for example when interacting with 3rd-party APIs.
|
398
399
|
|
@@ -485,6 +486,196 @@ config.good_job.cron = {
|
|
485
486
|
}
|
486
487
|
```
|
487
488
|
|
489
|
+
### Bulk enqueue
|
490
|
+
|
491
|
+
GoodJob's Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can more performant when enqueuing a large number of jobs.
|
492
|
+
|
493
|
+
```ruby
|
494
|
+
# Capture jobs using `.perform_later`:
|
495
|
+
active_jobs = GoodJob::Bulk.enqueue do
|
496
|
+
MyJob.perform_later
|
497
|
+
AnotherJob.perform_later
|
498
|
+
# If an exception is raised within this block, no jobs will be inserted.
|
499
|
+
end
|
500
|
+
|
501
|
+
# All ActiveJob instances are returned from GoodJob::Bulk.enqueue.
|
502
|
+
# Jobs that have been successfully enqueued have a `provider_job_id` set.
|
503
|
+
active_jobs.all?(&:provider_job_id)
|
504
|
+
|
505
|
+
# Bulk enqueue ActiveJob instances directly without using `.perform_later`:
|
506
|
+
GoodJob::Bulk.enqueue(MyJob.new, AnotherJob.new)
|
507
|
+
```
|
508
|
+
|
509
|
+
### Batches
|
510
|
+
|
511
|
+
Batches track a set of jobs, and enqueue an optional callback job when all of the jobs have finished (succeeded or discarded).
|
512
|
+
|
513
|
+
- A simple example that enqueues your `MyBatchCallbackJob` after the two jobs have finished, and passes along the current user as a batch property:
|
514
|
+
|
515
|
+
```ruby
|
516
|
+
GoodJob::Batch.enqueue(on_finish: MyBatchCallbackJob, user: current_user) do
|
517
|
+
MyJob.perform_later
|
518
|
+
OtherJob.perform_later
|
519
|
+
end
|
520
|
+
|
521
|
+
# When these jobs have finished, it will enqueue your `MyBatchCallbackJob.perform_later(batch, options)`
|
522
|
+
class MyBatchCallbackJob < ApplicationJob
|
523
|
+
# Callback jobs must accept a `batch` and `options` argument
|
524
|
+
def perform(batch, params)
|
525
|
+
# The batch object will contain the Batch's properties, which are mutable
|
526
|
+
batch.properties[:user] # => <User id: 1, ...>
|
527
|
+
|
528
|
+
# Params is a hash containing additional context (more may be added in the future)
|
529
|
+
params[:event] # => :finish, :success, :discard
|
530
|
+
end
|
531
|
+
end
|
532
|
+
```
|
533
|
+
|
534
|
+
- Jobs can be added to an existing batch. Jobs in a batch are enqueued and performed immediately/asynchronously. The final callback job will not be enqueued until `GoodJob::Batch#enqueue` is called.
|
535
|
+
|
536
|
+
```ruby
|
537
|
+
batch = GoodJob::Batch.add do
|
538
|
+
10.times { MyJob.perform_later }
|
539
|
+
end
|
540
|
+
batch.add do
|
541
|
+
10.times { OtherJob.perform_later }
|
542
|
+
end
|
543
|
+
batch.enqueue(on_finish: MyBatchCallbackJob, age: 42)
|
544
|
+
```
|
545
|
+
|
546
|
+
- If you need to access the batch within a job that is part of the batch, include [`GoodJob::ActiveJobExtensions::Batches`](lib/good_job/active_job_extensions/batches.rb) in your job class:
|
547
|
+
|
548
|
+
```ruby
|
549
|
+
class MyJob < ApplicationJob
|
550
|
+
include GoodJob::ActiveJobExtensions::Batches
|
551
|
+
|
552
|
+
def perform
|
553
|
+
self.batch # => <GoodJob::Batch id: 1, ...>
|
554
|
+
end
|
555
|
+
end
|
556
|
+
```
|
557
|
+
|
558
|
+
- [`GoodJob::Batch`](app/models/good_job/batch.rb) has a number of assignable attributes and methods:
|
559
|
+
|
560
|
+
```ruby
|
561
|
+
batch = GoodJob::Batch.new
|
562
|
+
batch.description = "My batch"
|
563
|
+
batch.on_finish = "MyBatchCallbackJob" # Callback job when all jobs have finished
|
564
|
+
batch.on_success = "MyBatchCallbackJob" # Callback job when/if all jobs have succeeded
|
565
|
+
batch.on_discard = "MyBatchCallbackJob" # Callback job when the first job in the batch is discarded
|
566
|
+
batch.callback_queue_name = "special_queue" # Optional queue for callback jobs, otherwise will defer to job class
|
567
|
+
batch.callback_priority = 10 # Optional priority name for callback jobs, otherwise will defer to job class
|
568
|
+
batch.properties = { age: 42 } # Custom data and state to attach to the batch
|
569
|
+
batch.add do
|
570
|
+
MyJob.perform_later
|
571
|
+
end
|
572
|
+
batch.enqueue
|
573
|
+
|
574
|
+
batch.discarded? # => Boolean
|
575
|
+
batch.discarded_at # => <DateTime>
|
576
|
+
batch.finished? # => Boolean
|
577
|
+
batch.finished_at # => <DateTime>
|
578
|
+
batch.succeeded? # => Boolean
|
579
|
+
batch.active_jobs # => Array of ActiveJob::Base-inherited jobs that are part of the batch
|
580
|
+
|
581
|
+
batch = GoodJob::Batch.find(batch.id)
|
582
|
+
batch.description = "Updated batch description"
|
583
|
+
batch.save
|
584
|
+
batch.reload
|
585
|
+
```
|
586
|
+
|
587
|
+
### Batch callback jobs
|
588
|
+
|
589
|
+
Batch callbacks are Active Job jobs that are enqueued at certain events during the execution of jobs within the batch:
|
590
|
+
|
591
|
+
- `:finish` - Enqueued when all jobs in the batch have finished, after all retries. Jobs will either be discarded or succeeded.
|
592
|
+
- `:success` - Enqueued only when all jobs in the batch have finished and succeeded.
|
593
|
+
- `:discard` - Enqueued immediately the first time a job in the batch is discarded.
|
594
|
+
|
595
|
+
Callback jobs must accept a `batch` and `params` argument in their `perform` method:
|
596
|
+
|
597
|
+
```ruby
|
598
|
+
class MyBatchCallbackJob < ApplicationJob
|
599
|
+
def perform(batch, params)
|
600
|
+
# The batch object will contain the Batch's properties
|
601
|
+
batch.properties[:user] # => <User id: 1, ...>
|
602
|
+
# Batches are mutable
|
603
|
+
batch.properties[:user] = User.find(2)
|
604
|
+
batch.save
|
605
|
+
|
606
|
+
# Params is a hash containing additional context (more may be added in the future)
|
607
|
+
params[:event] # => :finish, :success, :discard
|
608
|
+
end
|
609
|
+
end
|
610
|
+
```
|
611
|
+
|
612
|
+
#### Complex batches
|
613
|
+
|
614
|
+
Consider a multi-stage batch with both parallel and serial job steps:
|
615
|
+
|
616
|
+
```mermaid
|
617
|
+
graph TD
|
618
|
+
0{"BatchJob\n{ stage: nil }"}
|
619
|
+
0 --> a["WorkJob]\n{ step: a }"]
|
620
|
+
0 --> b["WorkJob]\n{ step: b }"]
|
621
|
+
0 --> c["WorkJob]\n{ step: c }"]
|
622
|
+
a --> 1
|
623
|
+
b --> 1
|
624
|
+
c --> 1
|
625
|
+
1{"BatchJob\n{ stage: 1 }"}
|
626
|
+
1 --> d["WorkJob]\n{ step: d }"]
|
627
|
+
1 --> e["WorkJob]\n{ step: e }"]
|
628
|
+
e --> f["WorkJob]\n{ step: f }"]
|
629
|
+
d --> 2
|
630
|
+
f --> 2
|
631
|
+
2{"BatchJob\n{ stage: 2 }"}
|
632
|
+
```
|
633
|
+
|
634
|
+
This can be implemented with a single, mutable batch job:
|
635
|
+
|
636
|
+
```ruby
|
637
|
+
class WorkJob < ApplicationJob
|
638
|
+
include GoodJob::ActiveJobExtensions::Batches
|
639
|
+
|
640
|
+
def perform(step)
|
641
|
+
# ...
|
642
|
+
if step == 'e'
|
643
|
+
batch.add { WorkJob.perform_later('f') }
|
644
|
+
end
|
645
|
+
end
|
646
|
+
end
|
647
|
+
|
648
|
+
class BatchJob < ApplicationJob
|
649
|
+
def perform(batch, options)
|
650
|
+
if batch.properties[:stage].nil?
|
651
|
+
batch.enqueue(stage: 1) do
|
652
|
+
WorkJob.perform_later('a')
|
653
|
+
WorkJob.perform_later('b')
|
654
|
+
WorkJob.perform_later('c')
|
655
|
+
end
|
656
|
+
elsif batch.properties[:stage] == 1
|
657
|
+
batch.enqueue(stage: 2) do
|
658
|
+
WorkJob.perform_later('d')
|
659
|
+
WorkJob.perform_later('e')
|
660
|
+
end
|
661
|
+
elsif batch.properties[:stage] == 2
|
662
|
+
# ...
|
663
|
+
end
|
664
|
+
end
|
665
|
+
end
|
666
|
+
|
667
|
+
GoodJob::Batch.enqueue(on_finish: BatchJob)
|
668
|
+
```
|
669
|
+
|
670
|
+
#### Other batch details
|
671
|
+
|
672
|
+
- Whether to enqueue a callback job is evaluated once the batch is in an `enqueued?`-state by using `GoodJob::Batch.enqueue` or `batch.enqueue`.
|
673
|
+
- Callback job enqueueing will be re-triggered if additional jobs are `enqueue`'d to the batch; use `add` to add jobs to the batch without retriggering callback jobs.
|
674
|
+
- Callback jobs will be enqueued even if the batch contains no jobs.
|
675
|
+
- Callback jobs perform asynchronously. It's possible that `:finish` and `:success` or `:discard` callback jobs perform at the same time. Keep this in mind when updating batch properties.
|
676
|
+
- Batch properties are serialized using Active Job serialization. This is flexible, but can lead to deserialization errors if a GlobalID record is directly referenced but is subsequently deleted and thus unloadable.
|
677
|
+
- 🚧Batches are a work in progress. Please let us know what would be helpful to improve their functionality and usefulness.
|
678
|
+
|
488
679
|
### Updating
|
489
680
|
|
490
681
|
GoodJob follows semantic versioning, though updates may be encouraged through deprecation warnings in minor versions.
|
@@ -794,26 +985,6 @@ To explain where this value is used, here is the pseudo-query that GoodJob uses
|
|
794
985
|
)
|
795
986
|
```
|
796
987
|
|
797
|
-
### Bulk enqueue
|
798
|
-
|
799
|
-
GoodJob's Bulk-enqueue functionality can buffer and enqueue multiple jobs at once, using a single INSERT statement. This can more performant when enqueuing a large number of jobs.
|
800
|
-
|
801
|
-
```ruby
|
802
|
-
# Capture jobs using `.perform_later`:
|
803
|
-
active_jobs = GoodJob::Bulk.enqueue do
|
804
|
-
MyJob.perform_later
|
805
|
-
AnotherJob.perform_later
|
806
|
-
# If an exception is raised within this block, no jobs will be inserted.
|
807
|
-
end
|
808
|
-
|
809
|
-
# All ActiveJob instances are returned from GoodJob::Bulk.enqueue.
|
810
|
-
# Jobs that have been successfully enqueued have a `provider_job_id` set.
|
811
|
-
active_jobs.all?(&:provider_job_id)
|
812
|
-
|
813
|
-
# Bulk enqueue ActiveJob instances directly without using `.perform_later`:
|
814
|
-
GoodJob::Bulk.enqueue(MyJob.new, AnotherJob.new)
|
815
|
-
```
|
816
|
-
|
817
988
|
### Execute jobs async / in-process
|
818
989
|
|
819
990
|
GoodJob can execute jobs "async" in the same process as the web server (e.g. `bin/rails s`). GoodJob's async execution mode offers benefits of economy by not requiring a separate job worker process, but with the tradeoff of increased complexity. Async mode can be configured in two ways:
|
@@ -0,0 +1,21 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
module GoodJob
|
3
|
+
class BatchesFilter < BaseFilter
|
4
|
+
def records
|
5
|
+
after_created_at = params[:after_created_at].present? ? Time.zone.parse(params[:after_created_at]) : nil
|
6
|
+
|
7
|
+
filtered_query.display_all(
|
8
|
+
after_created_at: after_created_at,
|
9
|
+
after_id: params[:after_id]
|
10
|
+
).limit(params.fetch(:limit, DEFAULT_LIMIT))
|
11
|
+
end
|
12
|
+
|
13
|
+
def filtered_query(_filtered_params = params)
|
14
|
+
base_query
|
15
|
+
end
|
16
|
+
|
17
|
+
def default_base_query
|
18
|
+
GoodJob::BatchRecord.all.includes(:jobs)
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,149 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module GoodJob
|
4
|
+
# NOTE: This class delegates to {GoodJob::BatchRecord} and is intended to be the public interface for Batches.
|
5
|
+
class Batch
|
6
|
+
include GlobalID::Identification
|
7
|
+
|
8
|
+
thread_cattr_accessor :current_batch_id
|
9
|
+
thread_cattr_accessor :current_batch_callback_id
|
10
|
+
|
11
|
+
PROTECTED_PROPERTIES = %i[
|
12
|
+
on_finish
|
13
|
+
on_success
|
14
|
+
on_discard
|
15
|
+
callback_queue_name
|
16
|
+
callback_priority
|
17
|
+
description
|
18
|
+
properties
|
19
|
+
].freeze
|
20
|
+
|
21
|
+
delegate(
|
22
|
+
:id,
|
23
|
+
:created_at,
|
24
|
+
:updated_at,
|
25
|
+
:persisted?,
|
26
|
+
:enqueued_at,
|
27
|
+
:finished_at,
|
28
|
+
:discarded_at,
|
29
|
+
:enqueued?,
|
30
|
+
:finished?,
|
31
|
+
:succeeded?,
|
32
|
+
:discarded?,
|
33
|
+
:description,
|
34
|
+
:description=,
|
35
|
+
:on_finish,
|
36
|
+
:on_finish=,
|
37
|
+
:on_success,
|
38
|
+
:on_success=,
|
39
|
+
:on_discard,
|
40
|
+
:on_discard=,
|
41
|
+
:callback_queue_name,
|
42
|
+
:callback_queue_name=,
|
43
|
+
:callback_priority,
|
44
|
+
:callback_priority=,
|
45
|
+
:properties,
|
46
|
+
:properties=,
|
47
|
+
:save,
|
48
|
+
:reload,
|
49
|
+
to: :record
|
50
|
+
)
|
51
|
+
|
52
|
+
# Create a new batch and enqueue it
|
53
|
+
# @param on_finish [String, Object] The class name of the callback job to be enqueued after the batch is finished
|
54
|
+
# @param properties [Hash] Additional properties to be stored on the batch
|
55
|
+
# @param block [Proc] Enqueue jobs within the block to add them to the batch
|
56
|
+
# @return [GoodJob::BatchRecord]
|
57
|
+
def self.enqueue(active_jobs = [], **properties, &block)
|
58
|
+
new.tap do |batch|
|
59
|
+
batch.enqueue(active_jobs, **properties, &block)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
def self.find(id)
|
64
|
+
new _record: BatchRecord.find(id)
|
65
|
+
end
|
66
|
+
|
67
|
+
# Helper method to enqueue jobs and assign them to a batch
|
68
|
+
def self.within_thread(batch_id: nil, batch_callback_id: nil)
|
69
|
+
original_batch_id = current_batch_id
|
70
|
+
original_batch_callback_id = current_batch_callback_id
|
71
|
+
|
72
|
+
self.current_batch_id = batch_id
|
73
|
+
self.current_batch_callback_id = batch_callback_id
|
74
|
+
|
75
|
+
yield
|
76
|
+
ensure
|
77
|
+
self.current_batch_id = original_batch_id
|
78
|
+
self.current_batch_callback_id = original_batch_callback_id
|
79
|
+
end
|
80
|
+
|
81
|
+
def initialize(_record: nil, **properties) # rubocop:disable Lint/UnderscorePrefixedVariableName
|
82
|
+
self.record = _record || BatchRecord.new
|
83
|
+
assign_properties(properties)
|
84
|
+
end
|
85
|
+
|
86
|
+
# @return [Array<ActiveJob::Base>] Active jobs added to the batch
|
87
|
+
def enqueue(active_jobs = [], **properties, &block)
|
88
|
+
assign_properties(properties)
|
89
|
+
if record.new_record?
|
90
|
+
record.save!
|
91
|
+
else
|
92
|
+
record.with_advisory_lock(function: "pg_advisory_lock") do
|
93
|
+
record.enqueued_at_will_change!
|
94
|
+
record.finished_at_will_change!
|
95
|
+
record.update!(enqueued_at: nil, finished_at: nil)
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
active_jobs = add(active_jobs, &block)
|
100
|
+
|
101
|
+
record.with_advisory_lock(function: "pg_advisory_lock") do
|
102
|
+
record.update!(enqueued_at: Time.current)
|
103
|
+
record._continue_discard_or_finish(lock: false)
|
104
|
+
end
|
105
|
+
|
106
|
+
active_jobs
|
107
|
+
end
|
108
|
+
|
109
|
+
# Enqueue jobs and add them to the batch
|
110
|
+
# @param block [Proc] Enqueue jobs within the block to add them to the batch
|
111
|
+
# @return [Array<ActiveJob::Base>] Active jobs added to the batch
|
112
|
+
def add(active_jobs = nil, &block)
|
113
|
+
record.save if record.new_record?
|
114
|
+
|
115
|
+
buffer = Bulk::Buffer.new
|
116
|
+
buffer.add(active_jobs)
|
117
|
+
buffer.capture(&block) if block
|
118
|
+
|
119
|
+
self.class.within_thread(batch_id: id) do
|
120
|
+
buffer.enqueue
|
121
|
+
end
|
122
|
+
|
123
|
+
buffer.active_jobs
|
124
|
+
end
|
125
|
+
|
126
|
+
def active_jobs
|
127
|
+
record.jobs.map(&:head_execution).map(&:active_job)
|
128
|
+
end
|
129
|
+
|
130
|
+
def callback_active_jobs
|
131
|
+
record.callback_jobs.map(&:head_execution).map(&:active_job)
|
132
|
+
end
|
133
|
+
|
134
|
+
def assign_properties(properties)
|
135
|
+
properties = properties.dup
|
136
|
+
batch_attrs = PROTECTED_PROPERTIES.index_with { |key| properties.delete(key) }.compact
|
137
|
+
record.assign_attributes(batch_attrs)
|
138
|
+
self.properties.merge!(properties)
|
139
|
+
end
|
140
|
+
|
141
|
+
def _record
|
142
|
+
record
|
143
|
+
end
|
144
|
+
|
145
|
+
private
|
146
|
+
|
147
|
+
attr_accessor :record
|
148
|
+
end
|
149
|
+
end
|
@@ -0,0 +1,95 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module GoodJob
|
4
|
+
class BatchRecord < BaseRecord
|
5
|
+
include Lockable
|
6
|
+
|
7
|
+
self.table_name = 'good_job_batches'
|
8
|
+
|
9
|
+
has_many :jobs, class_name: 'GoodJob::Job', inverse_of: :batch, foreign_key: :batch_id, dependent: nil
|
10
|
+
has_many :executions, class_name: 'GoodJob::Execution', foreign_key: :batch_id, inverse_of: :batch, dependent: nil
|
11
|
+
has_many :callback_jobs, class_name: 'GoodJob::Job', foreign_key: :batch_callback_id, dependent: nil # rubocop:disable Rails/InverseOf
|
12
|
+
|
13
|
+
scope :finished, -> { where.not(finished_at: nil) }
|
14
|
+
scope :discarded, -> { where.not(discarded_at: nil) }
|
15
|
+
scope :not_discarded, -> { where(discarded_at: nil) }
|
16
|
+
scope :succeeded, -> { finished.not_discarded }
|
17
|
+
|
18
|
+
alias_attribute :enqueued?, :enqueued_at
|
19
|
+
alias_attribute :discarded?, :discarded_at
|
20
|
+
alias_attribute :finished?, :finished_at
|
21
|
+
|
22
|
+
scope :display_all, (lambda do |after_created_at: nil, after_id: nil|
|
23
|
+
query = order(created_at: :desc, id: :desc)
|
24
|
+
if after_created_at.present? && after_id.present?
|
25
|
+
query = query.where(Arel.sql('(created_at, id) < (:after_created_at, :after_id)'), after_created_at: after_created_at, after_id: after_id)
|
26
|
+
elsif after_created_at.present?
|
27
|
+
query = query.where(Arel.sql('(after_created_at) < (:after_created_at)'), after_created_at: after_created_at)
|
28
|
+
end
|
29
|
+
query
|
30
|
+
end)
|
31
|
+
|
32
|
+
# Whether the batch has finished and no jobs were discarded
|
33
|
+
# @return [Boolean]
|
34
|
+
def succeeded?
|
35
|
+
!discarded? && finished?
|
36
|
+
end
|
37
|
+
|
38
|
+
def to_batch
|
39
|
+
Batch.new(_record: self)
|
40
|
+
end
|
41
|
+
|
42
|
+
def display_attributes
|
43
|
+
attributes.except('serialized_properties').merge(properties: properties)
|
44
|
+
end
|
45
|
+
|
46
|
+
def _continue_discard_or_finish(execution = nil, lock: true)
|
47
|
+
execution_discarded = execution && execution.error.present? && execution.retried_good_job_id.nil?
|
48
|
+
take_advisory_lock(lock) do
|
49
|
+
Batch.within_thread(batch_id: nil, batch_callback_id: id) do
|
50
|
+
reload
|
51
|
+
if execution_discarded && !discarded_at
|
52
|
+
update(discarded_at: Time.current)
|
53
|
+
on_discard.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :discard }) if on_discard.present?
|
54
|
+
end
|
55
|
+
|
56
|
+
if enqueued_at && !finished_at && jobs.where(finished_at: nil).count.zero?
|
57
|
+
update(finished_at: Time.current)
|
58
|
+
on_success.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :success }) if !discarded_at && on_success.present?
|
59
|
+
on_finish.constantize.set(priority: callback_priority, queue: callback_queue_name).perform_later(to_batch, { event: :finish }) if on_finish.present?
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
|
65
|
+
class PropertySerializer
|
66
|
+
def self.dump(value)
|
67
|
+
ActiveJob::Arguments.serialize([value]).first
|
68
|
+
end
|
69
|
+
|
70
|
+
def self.load(value)
|
71
|
+
ActiveJob::Arguments.deserialize([value]).first
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
75
|
+
attribute :serialized_properties, :json, default: -> { {} } # Can be set as default value in `serialize` as of Rails v6.1
|
76
|
+
serialize :serialized_properties, PropertySerializer
|
77
|
+
alias_attribute :properties, :serialized_properties
|
78
|
+
|
79
|
+
def properties=(value)
|
80
|
+
raise ArgumentError, "Properties must be a Hash" unless value.is_a?(Hash)
|
81
|
+
|
82
|
+
self.serialized_properties = value
|
83
|
+
end
|
84
|
+
|
85
|
+
private
|
86
|
+
|
87
|
+
def take_advisory_lock(value, &block)
|
88
|
+
if value
|
89
|
+
with_advisory_lock(function: "pg_advisory_lock", &block)
|
90
|
+
else
|
91
|
+
yield
|
92
|
+
end
|
93
|
+
end
|
94
|
+
end
|
95
|
+
end
|
@@ -20,8 +20,12 @@ module GoodJob
|
|
20
20
|
self.table_name = 'good_jobs'
|
21
21
|
self.advisory_lockable_column = 'active_job_id'
|
22
22
|
|
23
|
+
define_model_callbacks :perform
|
23
24
|
define_model_callbacks :perform_unlocked, only: :after
|
24
25
|
|
26
|
+
set_callback :perform, :around, :reset_batch_values
|
27
|
+
set_callback :perform_unlocked, :after, :continue_discard_or_finish_batch
|
28
|
+
|
25
29
|
# Parse a string representing a group of queues into a more readable data
|
26
30
|
# structure.
|
27
31
|
# @param string [String] Queue string
|
@@ -65,6 +69,9 @@ module GoodJob
|
|
65
69
|
end
|
66
70
|
end
|
67
71
|
|
72
|
+
belongs_to :batch, class_name: 'GoodJob::BatchRecord', optional: true, inverse_of: :executions
|
73
|
+
belongs_to :batch_callback, class_name: 'GoodJob::Batch', optional: true
|
74
|
+
|
68
75
|
belongs_to :job, class_name: 'GoodJob::Job', foreign_key: 'active_job_id', primary_key: 'active_job_id', optional: true, inverse_of: :executions
|
69
76
|
after_destroy -> { self.class.active_job_id(active_job_id).delete_all }, if: -> { @_destroy_job }
|
70
77
|
|
@@ -206,14 +213,24 @@ module GoodJob
|
|
206
213
|
serialized_params: active_job.serialize,
|
207
214
|
scheduled_at: active_job.scheduled_at,
|
208
215
|
}
|
209
|
-
|
210
216
|
execution_args[:concurrency_key] = active_job.good_job_concurrency_key if active_job.respond_to?(:good_job_concurrency_key)
|
211
217
|
|
212
|
-
|
218
|
+
reenqueued_current_execution = CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
219
|
+
current_execution = CurrentThread.execution
|
220
|
+
|
221
|
+
if reenqueued_current_execution
|
222
|
+
if GoodJob::BatchRecord.migrated?
|
223
|
+
execution_args[:batch_id] = current_execution.batch_id
|
224
|
+
execution_args[:batch_callback_id] = current_execution.batch_callback_id
|
225
|
+
end
|
226
|
+
execution_args[:cron_key] = current_execution.cron_key
|
227
|
+
else
|
228
|
+
if GoodJob::BatchRecord.migrated?
|
229
|
+
execution_args[:batch_id] = GoodJob::Batch.current_batch_id
|
230
|
+
execution_args[:batch_callback_id] = GoodJob::Batch.current_batch_callback_id
|
231
|
+
end
|
213
232
|
execution_args[:cron_key] = CurrentThread.cron_key
|
214
233
|
execution_args[:cron_at] = CurrentThread.cron_at
|
215
|
-
elsif CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
216
|
-
execution_args[:cron_key] = CurrentThread.execution.cron_key
|
217
234
|
end
|
218
235
|
|
219
236
|
new(**execution_args.merge(overrides))
|
@@ -283,7 +300,6 @@ module GoodJob
|
|
283
300
|
|
284
301
|
execution.save!
|
285
302
|
active_job.provider_job_id = execution.id
|
286
|
-
|
287
303
|
CurrentThread.execution.retried_good_job_id = execution.id if CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
|
288
304
|
|
289
305
|
execution
|
@@ -296,27 +312,29 @@ module GoodJob
|
|
296
312
|
# exception raised by the job, if any. If the job completed successfully,
|
297
313
|
# the second array entry (the exception) will be +nil+ and vice versa.
|
298
314
|
def perform
|
299
|
-
|
315
|
+
run_callbacks(:perform) do
|
316
|
+
raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
|
300
317
|
|
301
|
-
|
302
|
-
|
318
|
+
self.performed_at = Time.current
|
319
|
+
save! if GoodJob.preserve_job_records
|
303
320
|
|
304
|
-
|
321
|
+
result = execute
|
305
322
|
|
306
|
-
|
307
|
-
|
323
|
+
job_error = result.handled_error || result.unhandled_error
|
324
|
+
self.error = [job_error.class, ERROR_MESSAGE_SEPARATOR, job_error.message].join if job_error
|
308
325
|
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
313
|
-
|
314
|
-
|
315
|
-
|
316
|
-
|
317
|
-
|
326
|
+
reenqueued = result.retried? || retried_good_job_id.present?
|
327
|
+
if result.unhandled_error && GoodJob.retry_on_unhandled_error
|
328
|
+
save!
|
329
|
+
elsif GoodJob.preserve_job_records == true || reenqueued || (result.unhandled_error && GoodJob.preserve_job_records == :on_unhandled_error) || cron_key.present?
|
330
|
+
self.finished_at = Time.current
|
331
|
+
save!
|
332
|
+
else
|
333
|
+
destroy_job
|
334
|
+
end
|
318
335
|
|
319
|
-
|
336
|
+
result
|
337
|
+
end
|
320
338
|
end
|
321
339
|
|
322
340
|
# Tests whether this job is safe to be executed by this thread.
|
@@ -411,5 +429,13 @@ module GoodJob
|
|
411
429
|
end
|
412
430
|
end
|
413
431
|
end
|
432
|
+
|
433
|
+
def reset_batch_values(&block)
|
434
|
+
GoodJob::Batch.within_thread(batch_id: nil, batch_callback_id: nil, &block)
|
435
|
+
end
|
436
|
+
|
437
|
+
def continue_discard_or_finish_batch
|
438
|
+
batch._continue_discard_or_finish(self) if GoodJob::BatchRecord.migrated? && batch.present?
|
439
|
+
end
|
414
440
|
end
|
415
441
|
end
|
data/app/models/good_job/job.rb
CHANGED
@@ -28,6 +28,7 @@ module GoodJob
|
|
28
28
|
self.primary_key = 'active_job_id'
|
29
29
|
self.advisory_lockable_column = 'active_job_id'
|
30
30
|
|
31
|
+
belongs_to :batch, class_name: 'GoodJob::BatchRecord', inverse_of: :jobs, optional: true
|
31
32
|
has_many :executions, -> { order(created_at: :asc) }, class_name: 'GoodJob::Execution', foreign_key: 'active_job_id', inverse_of: :job # rubocop:disable Rails/HasManyOrHasOneDependent
|
32
33
|
|
33
34
|
# Only the most-recent unretried execution represents a "Job"
|
@@ -0,0 +1,108 @@
|
|
1
|
+
<div class="my-3 card" data-gj-poll-replace id="jobs-table">
|
2
|
+
<div class="list-group list-group-flush text-nowrap table-jobs" role="table">
|
3
|
+
<header class="list-group-item bg-light">
|
4
|
+
<div class="row small text-muted text-uppercase align-items-center">
|
5
|
+
<div class="col-4">Jobs</div>
|
6
|
+
<div class="col-1">Queue</div>
|
7
|
+
<div class="col-1">Priority</div>
|
8
|
+
<div class="col-1 text-end">Attempts</div>
|
9
|
+
<div class="col text-end">
|
10
|
+
<%= tag.button type: "button", class: "btn btn-sm text-muted", role: "button",
|
11
|
+
data: { bs_toggle: "collapse", bs_target: ".job-params" },
|
12
|
+
aria: { expanded: false, controls: jobs.map { |job| "##{dom_id(job, "params")}" }.join(" ") } do %>
|
13
|
+
<%= render_icon "info" %>
|
14
|
+
<span class="visually-hidden">Inspect</span>
|
15
|
+
<% end %>
|
16
|
+
</div>
|
17
|
+
</div>
|
18
|
+
</header>
|
19
|
+
|
20
|
+
<% if jobs.present? %>
|
21
|
+
<% jobs.each do |job| %>
|
22
|
+
<div role="row" class="list-group-item list-group-item-action py-3">
|
23
|
+
<div class="row align-items-center">
|
24
|
+
<div class="col-4">
|
25
|
+
<%= tag.code link_to(job.id, job_path(job), class: "small text-muted text-decoration-none") %>
|
26
|
+
<%= tag.h5 tag.code(link_to(job.job_class, job_path(job), class: "text-reset text-decoration-none")), class: "text-reset mb-0" %>
|
27
|
+
</div>
|
28
|
+
<div class="col-1">
|
29
|
+
<span class="badge bg-primary bg-opacity-25 text-dark font-monospace"><%= job.queue_name %></span>
|
30
|
+
</div>
|
31
|
+
<div class="col-1 small text-center">
|
32
|
+
<span class="font-monospace fw-bold"><%= job.priority %></span>
|
33
|
+
</div>
|
34
|
+
<div class="col-1 text-center">
|
35
|
+
<% if job.executions_count > 0 && job.status != :finished %>
|
36
|
+
<%= tag.span job.executions_count, class: "badge rounded-pill bg-danger", data: {
|
37
|
+
bs_toggle: "popover",
|
38
|
+
bs_trigger: "hover focus click",
|
39
|
+
bs_placement: "bottom",
|
40
|
+
bs_content: job.recent_error
|
41
|
+
} %>
|
42
|
+
<% else %>
|
43
|
+
<span class="badge bg-secondary bg-opacity-50 rounded-pill"><%= job.executions_count %></span>
|
44
|
+
<% end %>
|
45
|
+
</div>
|
46
|
+
<div class="col d-flex gap-3 align-items-center justify-content-end">
|
47
|
+
<%= tag.span relative_time(job.last_status_at), class: "small" %>
|
48
|
+
<%= status_badge job.status %>
|
49
|
+
</div>
|
50
|
+
<div class="col-auto">
|
51
|
+
<div class="dropdown float-end">
|
52
|
+
<button class="d-flex align-items-center btn btn-sm" type="button" id="<%= dom_id(job, :actions) %>" data-bs-toggle="dropdown" aria-expanded="false">
|
53
|
+
<%= render "good_job/shared/icons/dots" %>
|
54
|
+
<span class="visually-hidden">Actions</span>
|
55
|
+
</button>
|
56
|
+
<ul class="dropdown-menu shadow" aria-labelledby="<%= dom_id(job, :actions) %>">
|
57
|
+
<li>
|
58
|
+
<% job_reschedulable = job.status.in? [:scheduled, :retried, :queued] %>
|
59
|
+
<%= link_to reschedule_job_path(job.id), method: :put, class: "dropdown-item #{'disabled' unless job_reschedulable}", title: "Reschedule job", data: { confirm: "Confirm reschedule", disable: true } do %>
|
60
|
+
<%= render "good_job/shared/icons/skip_forward" %>
|
61
|
+
Reschedule
|
62
|
+
<% end %>
|
63
|
+
</li>
|
64
|
+
<li>
|
65
|
+
<% job_discardable = job.status.in? [:scheduled, :retried, :queued] %>
|
66
|
+
<%= link_to discard_job_path(job.id), method: :put, class: "dropdown-item #{'disabled' unless job_discardable}", title: "Discard job", data: { confirm: "Confirm discard", disable: true } do %>
|
67
|
+
<%= render "good_job/shared/icons/stop" %>
|
68
|
+
Discard
|
69
|
+
<% end %>
|
70
|
+
</li>
|
71
|
+
<li>
|
72
|
+
<%= link_to retry_job_path(job.id), method: :put, class: "dropdown-item #{'disabled' unless job.status == :discarded}", title: "Retry job", data: { confirm: "Confirm retry", disable: true } do %>
|
73
|
+
<%= render "good_job/shared/icons/arrow_clockwise" %>
|
74
|
+
Retry
|
75
|
+
<% end %>
|
76
|
+
</li>
|
77
|
+
<li>
|
78
|
+
<%= link_to job_path(job.id), method: :delete, class: "dropdown-item #{'disabled' unless job.status.in? [:discarded, :finished]}", title: "Destroy job", data: { confirm: "Confirm destroy", disable: true } do %>
|
79
|
+
<%= render_icon "trash" %>
|
80
|
+
Destroy
|
81
|
+
<% end %>
|
82
|
+
</li>
|
83
|
+
|
84
|
+
<li>
|
85
|
+
<%= link_to "##{dom_id(job, 'params')}",
|
86
|
+
class: "dropdown-item",
|
87
|
+
data: { bs_toggle: "collapse" },
|
88
|
+
aria: { expanded: false, controls: dom_id(job, "params") } do %>
|
89
|
+
<%= render_icon "info" %>
|
90
|
+
Inspect
|
91
|
+
<% end %>
|
92
|
+
</li>
|
93
|
+
</ul>
|
94
|
+
</div>
|
95
|
+
</div>
|
96
|
+
</div>
|
97
|
+
</div>
|
98
|
+
<%= tag.div id: dom_id(job, "params"), class: "job-params list-group-item collapse small bg-dark text-light" do %>
|
99
|
+
<%= tag.pre JSON.pretty_generate(job.display_serialized_params) %>
|
100
|
+
<% end %>
|
101
|
+
<% end %>
|
102
|
+
<% else %>
|
103
|
+
<div class="list-group-item py-4 text-center text-muted">
|
104
|
+
No jobs found.
|
105
|
+
</div>
|
106
|
+
<% end %>
|
107
|
+
</>
|
108
|
+
</div>
|
@@ -0,0 +1,61 @@
|
|
1
|
+
<div class="my-3 card" data-gj-poll-replace id="batches-table">
|
2
|
+
<div class="list-group list-group-flush text-nowrap table-batches" role="table">
|
3
|
+
<header class="list-group-item bg-light">
|
4
|
+
<div class="row small text-muted text-uppercase align-items-center">
|
5
|
+
<div class="col-4">Name</div>
|
6
|
+
<div class="col-1">Created</div>
|
7
|
+
<div class="col-1">Enqueued</div>
|
8
|
+
<div class="col-1">Discarded</div>
|
9
|
+
<div class="col-1">Finished</div>
|
10
|
+
<div class="col">Jobs</div>
|
11
|
+
<div class="col text-end">
|
12
|
+
<%= tag.button type: "button", class: "btn btn-sm text-muted", role: "button",
|
13
|
+
data: { bs_toggle: "collapse", bs_target: ".batch-properties" },
|
14
|
+
aria: { expanded: false, controls: batches.map { |batch| "##{dom_id(batch, "params")}" }.join(" ") } do %>
|
15
|
+
<%= render_icon "info" %>
|
16
|
+
<span class="visually-hidden">Inspect</span>
|
17
|
+
<% end %>
|
18
|
+
</div>
|
19
|
+
</div>
|
20
|
+
</header>
|
21
|
+
|
22
|
+
<% if batches.present? %>
|
23
|
+
<% batches.each do |batch| %>
|
24
|
+
<div id="<%= dom_id(batch) %>" class="list-group-item py-3" role="row">
|
25
|
+
<div class="row align-items-center">
|
26
|
+
<div class="col-4">
|
27
|
+
<%= link_to batch_path(batch), class: "text-decoration-none" do %>
|
28
|
+
<code class="small text-muted">
|
29
|
+
<%= batch.id %>
|
30
|
+
</code>
|
31
|
+
<h5 class=""><code><%= batch.on_finish %></code></h5>
|
32
|
+
<div class="text-muted"><%= batch.description %></div>
|
33
|
+
<% end %>
|
34
|
+
</div>
|
35
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.created_at) %></div>
|
36
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.enqueued_at) if batch.enqueued_at %></div>
|
37
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.discarded_at) if batch.discarded_at %></div>
|
38
|
+
<div class="col-1 text-wrap"><%= relative_time(batch.finished_at) if batch.finished_at %></div>
|
39
|
+
<div class="col"><%= batch.jobs.count %></div>
|
40
|
+
<div class="col text-end">
|
41
|
+
<%= tag.button type: "button", class: "btn btn-sm text-muted ms-auto", role: "button",
|
42
|
+
title: "Inspect",
|
43
|
+
data: { bs_toggle: "collapse", bs_target: "##{dom_id(batch, 'properties')}" },
|
44
|
+
aria: { expanded: false, controls: dom_id(batch, "state") } do %>
|
45
|
+
<%= render_icon "info" %>
|
46
|
+
<span class="visually-hidden">Inspect</span>
|
47
|
+
<% end %>
|
48
|
+
</div>
|
49
|
+
</div>
|
50
|
+
</div>
|
51
|
+
<%= tag.div id: dom_id(batch, "properties"), class: "batch-properties list-group-item collapse small bg-dark text-light" do %>
|
52
|
+
<%= tag.pre JSON.pretty_generate(batch.properties) %>
|
53
|
+
<% end %>
|
54
|
+
<% end %>
|
55
|
+
<% else %>
|
56
|
+
<div class="list-group-item py-4 text-center text-muted">
|
57
|
+
No batches found.
|
58
|
+
</div>
|
59
|
+
<% end %>
|
60
|
+
</div>
|
61
|
+
</div>
|
@@ -0,0 +1,16 @@
|
|
1
|
+
<% if GoodJob::BatchRecord.migrated? %>
|
2
|
+
<%= render 'good_job/batches/table', batches: @filter.records, filter: @filter %>
|
3
|
+
<% if @filter.records.present? %>
|
4
|
+
<nav aria-label="Batch pagination" class="mt-3">
|
5
|
+
<ul class="pagination">
|
6
|
+
<li class="page-item">
|
7
|
+
<%= link_to(@filter.to_params(after_created_at: @filter.last.created_at, after_id: @filter.last.id), class: "page-link") do %>
|
8
|
+
Older batches <span aria-hidden="true">»</span>
|
9
|
+
<% end %>
|
10
|
+
</li>
|
11
|
+
</ul>
|
12
|
+
</nav>
|
13
|
+
<% end %>
|
14
|
+
<% else %>
|
15
|
+
<h3 class="text-center my-5">GoodJob has pending database migrations.</h3>
|
16
|
+
<% end %>
|
@@ -0,0 +1,30 @@
|
|
1
|
+
<div class="break-out bg-light border-bottom py-2 mb-3">
|
2
|
+
<div class="container-fluid pt-2">
|
3
|
+
<div class="row align-items-center">
|
4
|
+
<div class="col-5">
|
5
|
+
<nav aria-label="breadcrumb">
|
6
|
+
<ol class="breadcrumb small mb-0">
|
7
|
+
<li class="breadcrumb-item"><%= link_to "Batches", batches_path %></li>
|
8
|
+
<li class="breadcrumb-item active" aria-current="page"><%= tag.code @batch.id, class: "text-muted" %></li>
|
9
|
+
</ol>
|
10
|
+
<h2 class="h5 mt-2"><%= @batch.description %></h2>
|
11
|
+
</nav>
|
12
|
+
</div>
|
13
|
+
</div>
|
14
|
+
</div>
|
15
|
+
</div>
|
16
|
+
|
17
|
+
<div class="my-4">
|
18
|
+
<h5>Attributes</h5>
|
19
|
+
<%= tag.pre JSON.pretty_generate @batch.display_attributes, class: 'text-wrap text-break' %>
|
20
|
+
</div>
|
21
|
+
|
22
|
+
<div class="my-4">
|
23
|
+
<h5>Callback Jobs</h5>
|
24
|
+
<%= render 'jobs', jobs: @batch.callback_jobs.reverse %>
|
25
|
+
</div>
|
26
|
+
|
27
|
+
<div class="my-4">
|
28
|
+
<h5>Batched Jobs</h5>
|
29
|
+
<%= render 'jobs', jobs: @batch.jobs.reverse %>
|
30
|
+
</div>
|
@@ -14,6 +14,13 @@
|
|
14
14
|
<span class="badge bg-secondary rounded-pill"><%= number_to_human(jobs_count) %></span>
|
15
15
|
<% end %>
|
16
16
|
</li>
|
17
|
+
<li class="nav-item">
|
18
|
+
<%= link_to batches_path, class: ["nav-link", ("active" if controller_name == 'batches')] do %>
|
19
|
+
<%= "Batches" %>
|
20
|
+
<% batches_count = GoodJob::BatchRecord.migrated? ? GoodJob::BatchRecord.all.size : 0 %>
|
21
|
+
<span class="badge bg-secondary rounded-pill"><%= batches_count %></span>
|
22
|
+
<% end %>
|
23
|
+
</li>
|
17
24
|
<li class="nav-item">
|
18
25
|
<%= link_to cron_entries_path, class: ["nav-link", ("active" if controller_name == 'cron_entries')] do %>
|
19
26
|
<%= t(".cron_schedules") %>
|
data/config/routes.rb
CHANGED
@@ -19,6 +19,23 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
|
|
19
19
|
t.text :cron_key
|
20
20
|
t.uuid :retried_good_job_id
|
21
21
|
t.datetime :cron_at
|
22
|
+
|
23
|
+
t.uuid :batch_id
|
24
|
+
t.uuid :batch_callback_id
|
25
|
+
end
|
26
|
+
|
27
|
+
create_table :good_job_batches, id: :uuid do |t|
|
28
|
+
t.timestamps
|
29
|
+
t.text :description
|
30
|
+
t.jsonb :serialized_properties
|
31
|
+
t.text :on_finish
|
32
|
+
t.text :on_success
|
33
|
+
t.text :on_discard
|
34
|
+
t.text :callback_queue_name
|
35
|
+
t.integer :callback_priority
|
36
|
+
t.datetime :enqueued_at
|
37
|
+
t.datetime :discarded_at
|
38
|
+
t.datetime :finished_at
|
22
39
|
end
|
23
40
|
|
24
41
|
create_table :good_job_processes, id: :uuid do |t|
|
@@ -43,5 +60,7 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
|
|
43
60
|
add_index :good_jobs, [:finished_at], where: "retried_good_job_id IS NULL AND finished_at IS NOT NULL", name: :index_good_jobs_jobs_on_finished_at
|
44
61
|
add_index :good_jobs, [:priority, :created_at], order: { priority: "DESC NULLS LAST", created_at: :asc },
|
45
62
|
where: "finished_at IS NULL", name: :index_good_jobs_jobs_on_priority_created_at_when_unfinished
|
63
|
+
add_index :good_jobs, [:batch_id], where: "batch_id IS NOT NULL"
|
64
|
+
add_index :good_jobs, [:batch_callback_id], where: "batch_callback_id IS NOT NULL"
|
46
65
|
end
|
47
66
|
end
|
@@ -0,0 +1,34 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
class CreateGoodJobBatches < ActiveRecord::Migration<%= migration_version %>
|
3
|
+
def change
|
4
|
+
reversible do |dir|
|
5
|
+
dir.up do
|
6
|
+
# Ensure this incremental update migration is idempotent
|
7
|
+
# with monolithic install migration.
|
8
|
+
return if connection.table_exists?(:good_job_batches)
|
9
|
+
end
|
10
|
+
end
|
11
|
+
|
12
|
+
create_table :good_job_batches, id: :uuid do |t|
|
13
|
+
t.timestamps
|
14
|
+
t.text :description
|
15
|
+
t.jsonb :serialized_properties
|
16
|
+
t.text :on_finish
|
17
|
+
t.text :on_success
|
18
|
+
t.text :on_discard
|
19
|
+
t.text :callback_queue_name
|
20
|
+
t.integer :callback_priority
|
21
|
+
t.datetime :enqueued_at
|
22
|
+
t.datetime :discarded_at
|
23
|
+
t.datetime :finished_at
|
24
|
+
end
|
25
|
+
|
26
|
+
change_table :good_jobs do |t|
|
27
|
+
t.uuid :batch_id
|
28
|
+
t.uuid :batch_callback_id
|
29
|
+
|
30
|
+
t.index :batch_id, where: "batch_id IS NOT NULL"
|
31
|
+
t.index :batch_callback_id, where: "batch_callback_id IS NOT NULL"
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
data/lib/good_job/bulk.rb
CHANGED
@@ -15,19 +15,14 @@ module GoodJob
|
|
15
15
|
# @param active_jobs [Array<ActiveJob::Base>] Active Jobs to be buffered.
|
16
16
|
# @param queue_adapter Override the jobs implict queue adapter with an explicit one.
|
17
17
|
# @return [nil, Array<ActiveJob::Base>] The ActiveJob instances that have been buffered; nil if no active buffer
|
18
|
-
def self.capture(active_jobs = nil, queue_adapter: nil)
|
19
|
-
raise(ArgumentError, "Use either the block form or the argument form, not both") if
|
20
|
-
|
21
|
-
if
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
current_buffer.active_jobs
|
27
|
-
ensure
|
28
|
-
self.current_buffer = original_buffer
|
29
|
-
end
|
30
|
-
else
|
18
|
+
def self.capture(active_jobs = nil, queue_adapter: nil, &block)
|
19
|
+
raise(ArgumentError, "Use either the block form or the argument form, not both") if block && active_jobs
|
20
|
+
|
21
|
+
if block
|
22
|
+
buffer = Buffer.new
|
23
|
+
buffer.capture(&block)
|
24
|
+
buffer.active_jobs
|
25
|
+
elsif current_buffer
|
31
26
|
current_buffer&.add(active_jobs, queue_adapter: queue_adapter)
|
32
27
|
end
|
33
28
|
end
|
@@ -35,16 +30,15 @@ module GoodJob
|
|
35
30
|
# Capture jobs to a buffer and enqueue them all at once; or enqueue the current buffer.
|
36
31
|
# @param active_jobs [Array<ActiveJob::Base>] Active Jobs to be enqueued.
|
37
32
|
# @return [Array<ActiveJob::Base>] The ActiveJob instances that have been captured; check provider_job_id to confirm enqueued.
|
38
|
-
def self.enqueue(active_jobs = nil)
|
39
|
-
raise(ArgumentError, "Use either the block form or the argument form, not both") if
|
33
|
+
def self.enqueue(active_jobs = nil, &block)
|
34
|
+
raise(ArgumentError, "Use either the block form or the argument form, not both") if block && active_jobs
|
40
35
|
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
36
|
+
buffer = Buffer.new
|
37
|
+
if block
|
38
|
+
buffer.capture(&block)
|
39
|
+
buffer.enqueue
|
40
|
+
buffer.active_jobs
|
46
41
|
elsif active_jobs.present?
|
47
|
-
buffer = Buffer.new
|
48
42
|
buffer.add(active_jobs)
|
49
43
|
buffer.enqueue
|
50
44
|
buffer.active_jobs
|
@@ -69,8 +63,18 @@ module GoodJob
|
|
69
63
|
@values = []
|
70
64
|
end
|
71
65
|
|
66
|
+
def capture
|
67
|
+
original_buffer = Bulk.current_buffer
|
68
|
+
Bulk.current_buffer = self
|
69
|
+
yield
|
70
|
+
ensure
|
71
|
+
Bulk.current_buffer = original_buffer
|
72
|
+
end
|
73
|
+
|
72
74
|
def add(active_jobs, queue_adapter: nil)
|
73
75
|
new_pairs = Array(active_jobs).map do |active_job|
|
76
|
+
raise ArgumentError, "Expected an ActiveJob::Base instance, got #{active_job.class}" unless active_job.is_a?(ActiveJob::Base)
|
77
|
+
|
74
78
|
adapter = queue_adapter || active_job.class.queue_adapter
|
75
79
|
raise Error, "Jobs must have a Queue Adapter" unless adapter
|
76
80
|
|
@@ -156,7 +156,7 @@ module GoodJob
|
|
156
156
|
destroyed_records_count = event.payload[:destroyed_records_count]
|
157
157
|
|
158
158
|
info do
|
159
|
-
"GoodJob destroyed #{destroyed_records_count} preserved #{'
|
159
|
+
"GoodJob destroyed #{destroyed_records_count} preserved job execution #{'records'.pluralize(destroyed_records_count)} finished before #{timestamp}."
|
160
160
|
end
|
161
161
|
end
|
162
162
|
|
data/lib/good_job/version.rb
CHANGED
data/lib/good_job.rb
CHANGED
@@ -8,6 +8,7 @@ require "good_job/engine"
|
|
8
8
|
require "good_job/adapter"
|
9
9
|
require "active_job/queue_adapters/good_job_adapter"
|
10
10
|
require "good_job/active_job_extensions/concurrency"
|
11
|
+
require "good_job/active_job_extensions/batches"
|
11
12
|
|
12
13
|
require "good_job/assignable_connection"
|
13
14
|
require "good_job/bulk"
|
@@ -143,7 +144,7 @@ module GoodJob
|
|
143
144
|
end
|
144
145
|
end
|
145
146
|
|
146
|
-
# Destroys preserved job records.
|
147
|
+
# Destroys preserved job and batch records.
|
147
148
|
# By default, GoodJob destroys job records when the job is performed and this
|
148
149
|
# method is not necessary. However, when `GoodJob.preserve_job_records = true`,
|
149
150
|
# the jobs will be preserved in the database. This is useful when wanting to
|
@@ -151,7 +152,7 @@ module GoodJob
|
|
151
152
|
# If you are preserving job records this way, use this method regularly to
|
152
153
|
# destroy old records and preserve space in your database.
|
153
154
|
# @params older_than [nil,Numeric,ActiveSupport::Duration] Jobs older than this will be destroyed (default: +86400+).
|
154
|
-
# @return [Integer] Number of
|
155
|
+
# @return [Integer] Number of job execution records and batches that were destroyed.
|
155
156
|
def self.cleanup_preserved_jobs(older_than: nil)
|
156
157
|
older_than ||= GoodJob.configuration.cleanup_preserved_jobs_before_seconds_ago
|
157
158
|
timestamp = Time.current - older_than
|
@@ -160,10 +161,19 @@ module GoodJob
|
|
160
161
|
ActiveSupport::Notifications.instrument("cleanup_preserved_jobs.good_job", { older_than: older_than, timestamp: timestamp }) do |payload|
|
161
162
|
old_jobs = GoodJob::Job.where('finished_at <= ?', timestamp)
|
162
163
|
old_jobs = old_jobs.succeeded unless include_discarded
|
163
|
-
|
164
|
-
|
165
|
-
GoodJob::
|
166
|
-
|
164
|
+
deleted_executions_count = GoodJob::Execution.where(job: old_jobs).delete_all
|
165
|
+
|
166
|
+
if GoodJob::BatchRecord.migrated?
|
167
|
+
old_batches = GoodJob::BatchRecord.where('finished_at <= ?', timestamp)
|
168
|
+
old_batches = old_batches.succeeded unless include_discarded
|
169
|
+
deleted_batches_count = old_batches.delete_all
|
170
|
+
else
|
171
|
+
deleted_batches_count = 0
|
172
|
+
end
|
173
|
+
|
174
|
+
payload[:destroyed_executions_count] = deleted_executions_count
|
175
|
+
payload[:destroyed_batches_count] = deleted_batches_count
|
176
|
+
payload[:destroyed_records_count] = deleted_executions_count + deleted_batches_count
|
167
177
|
end
|
168
178
|
end
|
169
179
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: good_job
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 3.
|
4
|
+
version: 3.10.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Ben Sheldon
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-02-06 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activejob
|
@@ -334,16 +334,20 @@ files:
|
|
334
334
|
- app/charts/good_job/scheduled_by_queue_chart.rb
|
335
335
|
- app/controllers/good_job/application_controller.rb
|
336
336
|
- app/controllers/good_job/assets_controller.rb
|
337
|
+
- app/controllers/good_job/batches_controller.rb
|
337
338
|
- app/controllers/good_job/cron_entries_controller.rb
|
338
339
|
- app/controllers/good_job/jobs_controller.rb
|
339
340
|
- app/controllers/good_job/processes_controller.rb
|
340
341
|
- app/filters/good_job/base_filter.rb
|
342
|
+
- app/filters/good_job/batches_filter.rb
|
341
343
|
- app/filters/good_job/jobs_filter.rb
|
342
344
|
- app/helpers/good_job/application_helper.rb
|
343
345
|
- app/models/concerns/good_job/filterable.rb
|
344
346
|
- app/models/concerns/good_job/reportable.rb
|
345
347
|
- app/models/good_job/active_job_job.rb
|
346
348
|
- app/models/good_job/base_record.rb
|
349
|
+
- app/models/good_job/batch.rb
|
350
|
+
- app/models/good_job/batch_record.rb
|
347
351
|
- app/models/good_job/cron_entry.rb
|
348
352
|
- app/models/good_job/execution.rb
|
349
353
|
- app/models/good_job/execution_result.rb
|
@@ -351,6 +355,10 @@ files:
|
|
351
355
|
- app/models/good_job/lockable.rb
|
352
356
|
- app/models/good_job/process.rb
|
353
357
|
- app/models/good_job/setting.rb
|
358
|
+
- app/views/good_job/batches/_jobs.erb
|
359
|
+
- app/views/good_job/batches/_table.erb
|
360
|
+
- app/views/good_job/batches/index.html.erb
|
361
|
+
- app/views/good_job/batches/show.html.erb
|
354
362
|
- app/views/good_job/cron_entries/index.html.erb
|
355
363
|
- app/views/good_job/cron_entries/show.html.erb
|
356
364
|
- app/views/good_job/jobs/_executions.erb
|
@@ -390,8 +398,10 @@ files:
|
|
390
398
|
- lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb.erb
|
391
399
|
- lib/generators/good_job/templates/update/migrations/02_create_good_job_settings.rb.erb
|
392
400
|
- lib/generators/good_job/templates/update/migrations/03_create_index_good_jobs_jobs_on_priority_created_at_when_unfinished.rb.erb
|
401
|
+
- lib/generators/good_job/templates/update/migrations/04_create_good_job_batches.rb.erb
|
393
402
|
- lib/generators/good_job/update_generator.rb
|
394
403
|
- lib/good_job.rb
|
404
|
+
- lib/good_job/active_job_extensions/batches.rb
|
395
405
|
- lib/good_job/active_job_extensions/concurrency.rb
|
396
406
|
- lib/good_job/adapter.rb
|
397
407
|
- lib/good_job/assignable_connection.rb
|