good_job 2.7.4 → 2.8.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: c43ea987ac7a83911835cc6d6232839eba41f614b888dcb9752ae085cb4e30a0
4
- data.tar.gz: eb7efe2336fa441d8c5b0e64fa43b017078c4306599bc30f829610ef24ca037a
3
+ metadata.gz: ae78ac0322802107488f4a8b55963665cf9b3f173eb05174382bf05afdcef5a6
4
+ data.tar.gz: 2f41dd00281bcf2d0a2c29f6124de36964dcc730436f1cbd2f4c740c873263ed
5
5
  SHA512:
6
- metadata.gz: 3695aaebdb614804fa6df0210c411cc45120880674663551e6f53036bd237d6b64b12e2728bf810fac18c76f1372eb40de57f4b4daae24eda5448177218ffe63
7
- data.tar.gz: d03963f9186b5c177fbd107107c4f2dcd646df4dbc96fbf9a08e91f91af01d01369a89b44ee7bbf97baa0d941c47550fa4b29e3eca326c36fdc72b21646c0796
6
+ metadata.gz: 0ec2d40fdd87f293f8372e27040c7b14f8668645f50bc968ce91ec758202a400bb7c4a9ecb2e31a5827d59717dc3069840d5402981e1dc9f98c113120e259258
7
+ data.tar.gz: 0dd784cee33ab996ad332d1bfc371c357ffe3fb9727629096cff7189cc391195c109400240569444221a2ebf3340e2ab0d2d1c7bbd2830ee1c5b2ada9117d995
data/CHANGELOG.md CHANGED
@@ -1,9 +1,34 @@
1
1
  # Changelog
2
2
 
3
+ ## [v2.8.0](https://github.com/bensheldon/good_job/tree/v2.8.0) (2021-12-31)
4
+
5
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.7.4...v2.8.0)
6
+
7
+ **Implemented enhancements:**
8
+
9
+ - GoodJob should automatically clean up after itself and delete old job records [\#412](https://github.com/bensheldon/good_job/issues/412)
10
+ - Track processes in the database and on the Dashboard [\#472](https://github.com/bensheldon/good_job/pull/472) ([bensheldon](https://github.com/bensheldon))
11
+ - Allow Scheduler to automatically clean up preserved jobs every N jobs or seconds [\#465](https://github.com/bensheldon/good_job/pull/465) ([bensheldon](https://github.com/bensheldon))
12
+
13
+ **Closed issues:**
14
+
15
+ - Is there a way to show how many worker/process is running currently [\#471](https://github.com/bensheldon/good_job/issues/471)
16
+ - Jobs stuck in the unfinished state [\#448](https://github.com/bensheldon/good_job/issues/448)
17
+
18
+ **Merged pull requests:**
19
+
20
+ - Doublequote Ruby 3.0 in testing matrix [\#473](https://github.com/bensheldon/good_job/pull/473) ([bensheldon](https://github.com/bensheldon))
21
+ - Have demo CleanupJob use GoodJob.cleanup\_preserved\_jobs [\#470](https://github.com/bensheldon/good_job/pull/470) ([bensheldon](https://github.com/bensheldon))
22
+ - Test with Rails 7.0.0 [\#469](https://github.com/bensheldon/good_job/pull/469) ([aried3r](https://github.com/aried3r))
23
+
3
24
  ## [v2.7.4](https://github.com/bensheldon/good_job/tree/v2.7.4) (2021-12-16)
4
25
 
5
26
  [Full Changelog](https://github.com/bensheldon/good_job/compare/v2.7.3...v2.7.4)
6
27
 
28
+ **Fixed bugs:**
29
+
30
+ - Add nonce: true to javascript\_include\_tag in dashboard [\#468](https://github.com/bensheldon/good_job/pull/468) ([bouk](https://github.com/bouk))
31
+
7
32
  **Closed issues:**
8
33
 
9
34
  - Add nonce: true to engine views [\#467](https://github.com/bensheldon/good_job/issues/467)
@@ -11,7 +36,6 @@
11
36
 
12
37
  **Merged pull requests:**
13
38
 
14
- - Add nonce: true to javascript\_include\_tag in dashboard [\#468](https://github.com/bensheldon/good_job/pull/468) ([bouk](https://github.com/bouk))
15
39
  - Update appraisal for Rails 7.0.0.rc1 [\#466](https://github.com/bensheldon/good_job/pull/466) ([bensheldon](https://github.com/bensheldon))
16
40
 
17
41
  ## [v2.7.3](https://github.com/bensheldon/good_job/tree/v2.7.3) (2021-11-30)
data/README.md CHANGED
@@ -271,6 +271,9 @@ Available configuration options are:
271
271
  - `shutdown_timeout` (float) number of seconds to wait for jobs to finish when shutting down before stopping the thread. Defaults to forever: `-1`. You can also set this with the environment variable `GOOD_JOB_SHUTDOWN_TIMEOUT`.
272
272
  - `enable_cron` (boolean) whether to run cron process. Defaults to `false`. You can also set this with the environment variable `GOOD_JOB_ENABLE_CRON`.
273
273
  - `cron` (hash) cron configuration. Defaults to `{}`. You can also set this as a JSON string with the environment variable `GOOD_JOB_CRON`
274
+ - `cleanup_preserved_jobs_before_seconds_ago` (integer) number of seconds to preserve jobs when using the `$ good_job cleanup_preserved_jobs` CLI command or calling `GoodJob.cleanup_preserved_jobs`. Defaults to `86400` (1 day). Can also be set with the environment variable `GOOD_JOB_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO`. _This configuration is only used when {GoodJob.preserve_job_records} is `true`._
275
+ - `cleanup_interval_jobs` (integer) Number of jobs a Scheduler will execute before cleaning up preserved jobs. Defaults to `nil`. Can also be set with the environment variable `GOOD_JOB_CLEANUP_INTERVAL_JOBS`.
276
+ - `cleanup_interval_seconds` (integer) Number of seconds a Scheduler will wait before cleaning up preserved jobs. Defaults to `nil`. Can also be set with the environment variable `GOOD_JOB_CLEANUP_INTERVAL_SECONDS`.
274
277
  - `logger` ([Rails Logger](https://api.rubyonrails.org/classes/ActiveSupport/Logger.html)) lets you set a custom logger for GoodJob. It should be an instance of a Rails `Logger` (Default: `Rails.logger`).
275
278
  - `preserve_job_records` (boolean) keeps job records in your database even after jobs are completed. (Default: `false`)
276
279
  - `retry_on_unhandled_error` (boolean) causes jobs to be re-queued and retried if they raise an instance of `StandardError`. Instances of `Exception`, like SIGINT, will *always* be retried, regardless of this attribute’s value. (Default: `true`)
@@ -0,0 +1,8 @@
1
+ # frozen_string_literal: true
2
+ module GoodJob
3
+ class ProcessesController < GoodJob::BaseController
4
+ def index
5
+ @processes = GoodJob::Process.active.order(created_at: :desc) if GoodJob::Process.migrated?
6
+ end
7
+ end
8
+ end
@@ -0,0 +1,40 @@
1
+ <% if !GoodJob::Process.migrated? %>
2
+ <div class="card my-3">
3
+ <div class="card-body">
4
+ <p class="card-text">
5
+ <em>Feature unavailable because of pending database migration.</em>
6
+ </p>
7
+ </div>
8
+ </div>
9
+ <% elsif @processes.present? %>
10
+ <div class="card my-3">
11
+ <div class="table-responsive">
12
+ <table class="table card-table table-bordered table-hover table-sm mb-0">
13
+ <thead>
14
+ <tr>
15
+ <th>Process UUID</th>
16
+ <th>Created At</th></th>
17
+ <th>State</th>
18
+ </tr>
19
+ </thead>
20
+ <tbody>
21
+ <% @processes.each do |process| %>
22
+ <tr class="<%= dom_class(process) %>" id="<%= dom_id(process) %>">
23
+ <td><%= process.id %></td>
24
+ <td><%= relative_time(process.created_at) %></td>
25
+ <td><%= tag.pre JSON.pretty_generate(process.state) %></td>
26
+ </tr>
27
+ <% end %>
28
+ </tbody>
29
+ </table>
30
+ </div>
31
+ </div>
32
+ <% else %>
33
+ <div class="card my-3">
34
+ <div class="card-body">
35
+ <p class="card-text">
36
+ <em>No GoodJob processes found.</em>
37
+ </p>
38
+ </div>
39
+ </div>
40
+ <% end %>
@@ -33,23 +33,14 @@
33
33
  <li class="nav-item">
34
34
  <%= link_to "Cron Schedules", cron_entries_path, class: ["nav-link", ("active" if current_page?(cron_entries_path))] %>
35
35
  </li>
36
+ <li class="nav-item">
37
+ <%= link_to "Processes", processes_path, class: ["nav-link", ("active" if current_page?(processes_path))] %>
38
+ </li>
36
39
  <li class="nav-item">
37
40
  <div class="nav-link">
38
41
  <span class="badge bg-secondary">More views coming soon</span>
39
42
  </div>
40
43
  </li>
41
-
42
- <!-- Coming Soon
43
- <li class="nav-item">
44
- <%= link_to "Upcoming Jobs", 'todo', class: ["nav-link", ("active" if current_page?('todo'))] %>
45
- </li>
46
- <li class="nav-item">
47
- <%= link_to "Finished Jobs", 'todo', class: ["nav-link", ("active" if current_page?('todo'))] %>
48
- </li>
49
- <li class="nav-item">
50
- <%= link_to "Errored Jobs", 'todo', class: ["nav-link", ("active" if current_page?('todo'))] %>
51
- </li>
52
- -->
53
44
  </ul>
54
45
  <div class="text-muted" title="Now is <%= Time.current %>">Times are displayed in <%= Time.current.zone %> timezone</div>
55
46
  </div>
@@ -2,11 +2,7 @@
2
2
  GoodJob::Engine.routes.draw do
3
3
  root to: 'executions#index'
4
4
 
5
- resources :cron_entries, only: %i[index show] do
6
- member do
7
- post :enqueue
8
- end
9
- end
5
+ resources :executions, only: %i[destroy]
10
6
 
11
7
  resources :jobs, only: %i[index show] do
12
8
  member do
@@ -15,7 +11,14 @@ GoodJob::Engine.routes.draw do
15
11
  put :retry
16
12
  end
17
13
  end
18
- resources :executions, only: %i[destroy]
14
+
15
+ resources :cron_entries, only: %i[index show] do
16
+ member do
17
+ post :enqueue
18
+ end
19
+ end
20
+
21
+ resources :processes, only: %i[index]
19
22
 
20
23
  scope controller: :assets do
21
24
  constraints(format: :css) do
@@ -21,6 +21,11 @@ class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
21
21
  t.timestamp :cron_at
22
22
  end
23
23
 
24
+ create_table :good_job_processes, id: :uuid do |t|
25
+ t.timestamps
26
+ t.jsonb :state
27
+ end
28
+
24
29
  add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)", name: "index_good_jobs_on_scheduled_at"
25
30
  add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)", name: :index_good_jobs_on_queue_name_and_scheduled_at
26
31
  add_index :good_jobs, [:active_job_id, :created_at], name: :index_good_jobs_on_active_job_id_and_created_at
@@ -0,0 +1,17 @@
1
+ # frozen_string_literal: true
2
+ class CreateGoodJobProcesses < ActiveRecord::Migration<%= migration_version %>
3
+ def change
4
+ reversible do |dir|
5
+ dir.up do
6
+ # Ensure this incremental update migration is idempotent
7
+ # with monolithic install migration.
8
+ return if connection.table_exists?(:good_job_processes)
9
+ end
10
+ end
11
+
12
+ create_table :good_job_processes, id: :uuid do |t|
13
+ t.timestamps
14
+ t.jsonb :state
15
+ end
16
+ end
17
+ end
@@ -4,10 +4,7 @@ module GoodJob
4
4
  # There is not a table in the database whose discrete rows represents "Jobs".
5
5
  # The +good_jobs+ table is a table of individual {GoodJob::Execution}s that share the same +active_job_id+.
6
6
  # A single row from the +good_jobs+ table of executions is fetched to represent an ActiveJobJob
7
- # Parent class can be configured with +GoodJob.active_record_parent_class+.
8
- # @!parse
9
- # class ActiveJob < ActiveRecord::Base; end
10
- class ActiveJobJob < Object.const_get(GoodJob.active_record_parent_class)
7
+ class ActiveJobJob < BaseRecord
11
8
  include Filterable
12
9
  include Lockable
13
10
 
@@ -121,7 +121,7 @@ module GoodJob
121
121
  end
122
122
 
123
123
  # Start async executors
124
- # @return void
124
+ # @return [void]
125
125
  def start_async
126
126
  return unless execute_async?
127
127
 
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+ module GoodJob # :nodoc:
3
+ # Extends an ActiveRecord odel to override the connection and use
4
+ # an explicit connection that has been removed from the pool.
5
+ module AssignableConnection
6
+ extend ActiveSupport::Concern
7
+
8
+ included do
9
+ thread_cattr_accessor :_connection
10
+ end
11
+
12
+ class_methods do
13
+ # Assigns a connection to the model.
14
+ # @param conn [ActiveRecord::ConnectionAdapters::AbstractAdapter]
15
+ # @return [void]
16
+ def connection=(conn)
17
+ self._connection = conn
18
+ end
19
+
20
+ # Overrides the existing connection method to use the assigned connection
21
+ # @return [ActiveRecord::ConnectionAdapters::AbstractAdapter]
22
+ def connection
23
+ _connection || super
24
+ end
25
+
26
+ # Block interface to assign the connection, yield, then unassign the connection.
27
+ # @param conn [ActiveRecord::ConnectionAdapters::AbstractAdapter]
28
+ # @return [void]
29
+ def with_connection(conn)
30
+ original_conn = _connection
31
+ self.connection = conn
32
+ yield
33
+ ensure
34
+ self._connection = original_conn
35
+ end
36
+ end
37
+ end
38
+ end
@@ -0,0 +1,20 @@
1
+ # frozen_string_literal: true
2
+ module GoodJob
3
+ # Base ActiveRecord class that all GoodJob models inherit from.
4
+ # Parent class can be configured with +GoodJob.active_record_parent_class+.
5
+ # @!parse
6
+ # class BaseRecord < ActiveRecord::Base; end
7
+ class BaseRecord < Object.const_get(GoodJob.active_record_parent_class)
8
+ self.abstract_class = true
9
+
10
+ def self.migration_pending_warning!
11
+ ActiveSupport::Deprecation.warn(<<~DEPRECATION)
12
+ GoodJob has pending database migrations. To create the migration files, run:
13
+ rails generate good_job:update
14
+ To apply the migration files, run:
15
+ rails db:migrate
16
+ DEPRECATION
17
+ nil
18
+ end
19
+ end
20
+ end
@@ -0,0 +1,38 @@
1
+ # frozen_string_literal: true
2
+ module GoodJob # :nodoc:
3
+ # Tracks thresholds for cleaning up old jobs.
4
+ class CleanupTracker
5
+ attr_accessor :cleanup_interval_seconds,
6
+ :cleanup_interval_jobs,
7
+ :job_count,
8
+ :last_at
9
+
10
+ def initialize(cleanup_interval_seconds: nil, cleanup_interval_jobs: nil)
11
+ self.cleanup_interval_seconds = cleanup_interval_seconds
12
+ self.cleanup_interval_jobs = cleanup_interval_jobs
13
+
14
+ reset
15
+ end
16
+
17
+ # Increments job count.
18
+ # @return [void]
19
+ def increment
20
+ self.job_count += 1
21
+ end
22
+
23
+ # Whether a cleanup should be run.
24
+ # @return [Boolean]
25
+ def cleanup?
26
+ (cleanup_interval_jobs && job_count > cleanup_interval_jobs) ||
27
+ (cleanup_interval_seconds && last_at < Time.current - cleanup_interval_seconds) ||
28
+ false
29
+ end
30
+
31
+ # Resets the counters.
32
+ # @return [void]
33
+ def reset
34
+ self.job_count = 0
35
+ self.last_at = Time.current
36
+ end
37
+ end
38
+ end
@@ -18,6 +18,10 @@ module GoodJob
18
18
  DEFAULT_MAX_CACHE = 10000
19
19
  # Default number of seconds to preserve jobs for {CLI#cleanup_preserved_jobs} and {GoodJob.cleanup_preserved_jobs}
20
20
  DEFAULT_CLEANUP_PRESERVED_JOBS_BEFORE_SECONDS_AGO = 24 * 60 * 60
21
+ # Default number of jobs to execute between preserved job cleanup runs
22
+ DEFAULT_CLEANUP_INTERVAL_JOBS = nil
23
+ # Default number of seconds to wait between preserved job cleanup runs
24
+ DEFAULT_CLEANUP_INTERVAL_SECONDS = nil
21
25
  # Default to always wait for jobs to finish for {Adapter#shutdown}
22
26
  DEFAULT_SHUTDOWN_TIMEOUT = -1
23
27
  # Default to not running cron
@@ -179,6 +183,28 @@ module GoodJob
179
183
  ).to_i
180
184
  end
181
185
 
186
+ # Number of jobs a {Scheduler} will execute before cleaning up preserved jobs.
187
+ # @return [Integer, nil]
188
+ def cleanup_interval_jobs
189
+ value = (
190
+ rails_config[:cleanup_interval_jobs] ||
191
+ env['GOOD_JOB_CLEANUP_INTERVAL_JOBS'] ||
192
+ DEFAULT_CLEANUP_INTERVAL_JOBS
193
+ )
194
+ value.present? ? value.to_i : nil
195
+ end
196
+
197
+ # Number of seconds a {Scheduler} will wait before cleaning up preserved jobs.
198
+ # @return [Integer, nil]
199
+ def cleanup_interval_seconds
200
+ value = (
201
+ rails_config[:cleanup_interval_seconds] ||
202
+ env['GOOD_JOB_CLEANUP_INTERVAL_SECONDS'] ||
203
+ DEFAULT_CLEANUP_INTERVAL_SECONDS
204
+ )
205
+ value.present? ? value.to_i : nil
206
+ end
207
+
182
208
  # Tests whether to daemonize the process.
183
209
  # @return [Boolean]
184
210
  def daemonize?
@@ -68,7 +68,7 @@ module GoodJob
68
68
 
69
69
  # @return [Integer] Current process ID
70
70
  def self.process_id
71
- Process.pid
71
+ ::Process.pid
72
72
  end
73
73
 
74
74
  # @return [String] Current thread name
@@ -17,7 +17,7 @@ module GoodJob
17
17
  # @return [void]
18
18
  def daemonize
19
19
  check_pid
20
- Process.daemon
20
+ ::Process.daemon
21
21
  write_pid
22
22
  end
23
23
 
@@ -25,7 +25,7 @@ module GoodJob
25
25
 
26
26
  # @return [void]
27
27
  def write_pid
28
- File.open(pidfile, ::File::CREAT | ::File::EXCL | ::File::WRONLY) { |f| f.write(Process.pid.to_s) }
28
+ File.open(pidfile, ::File::CREAT | ::File::EXCL | ::File::WRONLY) { |f| f.write(::Process.pid.to_s) }
29
29
  at_exit { File.delete(pidfile) if File.exist?(pidfile) }
30
30
  rescue Errno::EEXIST
31
31
  check_pid
@@ -55,7 +55,7 @@ module GoodJob
55
55
  pid = ::File.read(pidfile).to_i
56
56
  return :dead if pid.zero?
57
57
 
58
- Process.kill(0, pid) # check process status
58
+ ::Process.kill(0, pid) # check process status
59
59
  :running
60
60
  rescue Errno::ESRCH
61
61
  :dead
@@ -1,10 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
  module GoodJob
3
3
  # ActiveRecord model that represents an +ActiveJob+ job.
4
- # Parent class can be configured with +GoodJob.active_record_parent_class+.
5
- # @!parse
6
- # class Execution < ActiveRecord::Base; end
7
- class Execution < Object.const_get(GoodJob.active_record_parent_class)
4
+ class Execution < BaseRecord
8
5
  include Lockable
9
6
  include Filterable
10
7
 
@@ -54,16 +51,6 @@ module GoodJob
54
51
  end
55
52
  end
56
53
 
57
- def self._migration_pending_warning
58
- ActiveSupport::Deprecation.warn(<<~DEPRECATION)
59
- GoodJob has pending database migrations. To create the migration files, run:
60
- rails generate good_job:update
61
- To apply the migration files, run:
62
- rails db:migrate
63
- DEPRECATION
64
- nil
65
- end
66
-
67
54
  # Get Jobs with given ActiveJob ID
68
55
  # @!method active_job_id
69
56
  # @!scope class
@@ -224,7 +211,7 @@ module GoodJob
224
211
  if @cron_at_index
225
212
  execution_args[:cron_at] = CurrentThread.cron_at
226
213
  else
227
- _migration_pending_warning
214
+ migration_pending_warning!
228
215
  end
229
216
  elsif CurrentThread.active_job_id && CurrentThread.active_job_id == active_job.job_id
230
217
  execution_args[:cron_key] = CurrentThread.execution.cron_key
@@ -57,6 +57,12 @@ module GoodJob
57
57
  job_query.next_scheduled_at(after: after, limit: limit, now_limit: now_limit)
58
58
  end
59
59
 
60
+ # Delete expired preserved jobs
61
+ # @return [void]
62
+ def cleanup
63
+ GoodJob.cleanup_preserved_jobs
64
+ end
65
+
60
66
  private
61
67
 
62
68
  attr_reader :queue_string
@@ -0,0 +1,34 @@
1
+ # frozen_string_literal: true
2
+
3
+ module GoodJob # :nodoc:
4
+ class Notifier # :nodoc:
5
+ # Extends the Notifier to register the process in the database.
6
+ module ProcessRegistration
7
+ extend ActiveSupport::Concern
8
+
9
+ included do
10
+ set_callback :listen, :after, :register_process
11
+ set_callback :unlisten, :after, :deregister_process
12
+ end
13
+
14
+ # Registers the current process.
15
+ def register_process
16
+ GoodJob::Process.with_connection(connection) do
17
+ next unless Process.migrated?
18
+
19
+ GoodJob::Process.cleanup
20
+ @process = GoodJob::Process.register
21
+ end
22
+ end
23
+
24
+ # Deregisters the current process.
25
+ def deregister_process
26
+ GoodJob::Process.with_connection(connection) do
27
+ next unless Process.migrated?
28
+
29
+ @process&.deregister
30
+ end
31
+ end
32
+ end
33
+ end
34
+ end
@@ -1,4 +1,5 @@
1
1
  # frozen_string_literal: true
2
+ require 'active_support/core_ext/module/attribute_accessors_per_thread'
2
3
  require 'concurrent/atomic/atomic_boolean'
3
4
 
4
5
  module GoodJob # :nodoc:
@@ -10,6 +11,11 @@ module GoodJob # :nodoc:
10
11
  # When a message is received, the notifier passes the message to each of its recipients.
11
12
  #
12
13
  class Notifier
14
+ include ActiveSupport::Callbacks
15
+ define_callbacks :listen, :unlisten
16
+
17
+ include Notifier::ProcessRegistration
18
+
13
19
  # Raised if the Database adapter does not implement LISTEN.
14
20
  AdapterCannotListenError = Class.new(StandardError)
15
21
 
@@ -43,6 +49,12 @@ module GoodJob # :nodoc:
43
49
  # @return [Array<GoodJob::Notifier>, nil]
44
50
  cattr_reader :instances, default: [], instance_reader: false
45
51
 
52
+ # @!attribute [rw] connection
53
+ # @!scope class
54
+ # ActiveRecord Connection that has been established for the Notifier.
55
+ # @return [ActiveRecord::ConnectionAdapters::AbstractAdapter, nil]
56
+ thread_cattr_accessor :connection
57
+
46
58
  # Send a message via Postgres NOTIFY
47
59
  # @param message [#to_json]
48
60
  def self.notify(message)
@@ -146,30 +158,36 @@ module GoodJob # :nodoc:
146
158
 
147
159
  def listen(delay: 0)
148
160
  future = Concurrent::ScheduledTask.new(delay, args: [@recipients, executor, @listening], executor: @executor) do |thr_recipients, thr_executor, thr_listening|
149
- with_listen_connection do |conn|
150
- ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
151
- conn.async_exec("LISTEN #{CHANNEL}").clear
152
- end
161
+ with_connection do
162
+ begin
163
+ run_callbacks :listen do
164
+ ActiveSupport::Notifications.instrument("notifier_listen.good_job") do
165
+ connection.execute("LISTEN #{CHANNEL}")
166
+ end
167
+ thr_listening.make_true
168
+ end
153
169
 
154
- ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
155
- thr_listening.make_true
156
- while thr_executor.running?
157
- conn.wait_for_notify(WAIT_INTERVAL) do |channel, _pid, payload|
158
- next unless channel == CHANNEL
159
-
160
- ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
161
- parsed_payload = JSON.parse(payload, symbolize_names: true)
162
- thr_recipients.each do |recipient|
163
- target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
164
- target.send(method_name, parsed_payload)
170
+ ActiveSupport::Dependencies.interlock.permit_concurrent_loads do
171
+ while thr_executor.running?
172
+ wait_for_notify do |channel, payload|
173
+ next unless channel == CHANNEL
174
+
175
+ ActiveSupport::Notifications.instrument("notifier_notified.good_job", { payload: payload })
176
+ parsed_payload = JSON.parse(payload, symbolize_names: true)
177
+ thr_recipients.each do |recipient|
178
+ target, method_name = recipient.is_a?(Array) ? recipient : [recipient, :call]
179
+ target.send(method_name, parsed_payload)
180
+ end
165
181
  end
166
182
  end
167
183
  end
168
184
  end
169
185
  ensure
170
- thr_listening.make_false
171
- ActiveSupport::Notifications.instrument("notifier_unlisten.good_job") do
172
- conn.async_exec("UNLISTEN *").clear
186
+ run_callbacks :unlisten do
187
+ thr_listening.make_false
188
+ ActiveSupport::Notifications.instrument("notifier_unlisten.good_job") do
189
+ connection.execute("UNLISTEN *")
190
+ end
173
191
  end
174
192
  end
175
193
  end
@@ -178,17 +196,27 @@ module GoodJob # :nodoc:
178
196
  future.execute
179
197
  end
180
198
 
181
- def with_listen_connection
182
- ar_conn = Execution.connection_pool.checkout.tap do |conn|
199
+ def with_connection
200
+ self.connection = Execution.connection_pool.checkout.tap do |conn|
183
201
  Execution.connection_pool.remove(conn)
184
202
  end
185
- pg_conn = ar_conn.raw_connection
186
- raise AdapterCannotListenError unless pg_conn.respond_to? :wait_for_notify
203
+ connection.execute("SET application_name = #{connection.quote(self.class.name)}")
187
204
 
188
- pg_conn.async_exec("SET application_name = #{pg_conn.escape_identifier(self.class.name)}").clear
189
- yield pg_conn
205
+ yield
190
206
  ensure
191
- ar_conn&.disconnect!
207
+ connection&.disconnect!
208
+ self.connection = nil
209
+ end
210
+
211
+ def wait_for_notify
212
+ raw_connection = connection.raw_connection
213
+ if raw_connection.respond_to?(:wait_for_notify)
214
+ raw_connection.wait_for_notify(WAIT_INTERVAL) do |channel, _pid, payload|
215
+ yield(channel, payload)
216
+ end
217
+ else
218
+ sleep WAIT_INTERVAL
219
+ end
192
220
  end
193
221
  end
194
222
  end
@@ -0,0 +1,82 @@
1
+ # frozen_string_literal: true
2
+ require 'socket'
3
+
4
+ module GoodJob # :nodoc:
5
+ # ActiveRecord model that represents an GoodJob process (either async or CLI).
6
+ class Process < BaseRecord
7
+ include AssignableConnection
8
+ include Lockable
9
+
10
+ self.table_name = 'good_job_processes'
11
+
12
+ cattr_reader :mutex, default: Mutex.new
13
+ cattr_accessor :_current_id, default: nil
14
+ cattr_accessor :_pid, default: nil
15
+
16
+ # Processes that are active and locked.
17
+ # @!method active
18
+ # @!scope class
19
+ # @return [ActiveRecord::Relation]
20
+ scope :active, -> { advisory_locked }
21
+
22
+ # Processes that are inactive and unlocked (e.g. SIGKILLed)
23
+ # @!method active
24
+ # @!scope class
25
+ # @return [ActiveRecord::Relation]
26
+ scope :inactive, -> { advisory_unlocked }
27
+
28
+ # Whether the +good_job_processes+ table exsists.
29
+ # @return [Boolean]
30
+ def self.migrated?
31
+ return true if connection.table_exists?(table_name)
32
+
33
+ migration_pending_warning!
34
+ false
35
+ end
36
+
37
+ # UUID that is unique to the current process and changes when forked.
38
+ # @return [String]
39
+ def self.current_id
40
+ mutex.synchronize do
41
+ if _current_id.nil? || _pid != ::Process.pid
42
+ self._current_id = SecureRandom.uuid
43
+ self._pid = ::Process.pid
44
+ end
45
+ _current_id
46
+ end
47
+ end
48
+
49
+ # Hash representing metadata about the current process.
50
+ # @return [Hash]
51
+ def self.current_state
52
+ {
53
+ id: current_id,
54
+ hostname: Socket.gethostname,
55
+ pid: ::Process.pid,
56
+ proctitle: $PROGRAM_NAME,
57
+ schedulers: GoodJob::Scheduler.instances.map(&:name),
58
+ }
59
+ end
60
+
61
+ # Deletes all inactive process records.
62
+ def self.cleanup
63
+ inactive.delete_all
64
+ end
65
+
66
+ # Registers the current process in the database
67
+ # @return [GoodJob::Process]
68
+ def self.register
69
+ create(id: current_id, state: current_state, create_with_advisory_lock: true)
70
+ rescue ActiveRecord::RecordNotUnique
71
+ nil
72
+ end
73
+
74
+ # Unregisters the instance.
75
+ def deregister
76
+ return unless owns_advisory_lock?
77
+
78
+ destroy!
79
+ advisory_unlock
80
+ end
81
+ end
82
+ end
@@ -48,7 +48,9 @@ module GoodJob # :nodoc:
48
48
  job_performer,
49
49
  max_threads: max_threads,
50
50
  max_cache: configuration.max_cache,
51
- warm_cache_on_initialize: warm_cache_on_initialize
51
+ warm_cache_on_initialize: warm_cache_on_initialize,
52
+ cleanup_interval_seconds: configuration.cleanup_interval_seconds,
53
+ cleanup_interval_jobs: configuration.cleanup_interval_jobs
52
54
  )
53
55
  end
54
56
 
@@ -59,11 +61,17 @@ module GoodJob # :nodoc:
59
61
  end
60
62
  end
61
63
 
64
+ # Human readable name of the scheduler that includes configuration values.
65
+ # @return [String]
66
+ attr_reader :name
67
+
62
68
  # @param performer [GoodJob::JobPerformer]
63
69
  # @param max_threads [Numeric, nil] number of seconds between polls for jobs
64
70
  # @param max_cache [Numeric, nil] maximum number of scheduled jobs to cache in memory
65
71
  # @param warm_cache_on_initialize [Boolean] whether to warm the cache immediately, or manually by calling +warm_cache+
66
- def initialize(performer, max_threads: nil, max_cache: nil, warm_cache_on_initialize: false)
72
+ # @param cleanup_interval_seconds [Numeric, nil] number of seconds between cleaning up job records
73
+ # @param cleanup_interval_jobs [Numeric, nil] number of executed jobs between cleaning up job records
74
+ def initialize(performer, max_threads: nil, max_cache: nil, warm_cache_on_initialize: false, cleanup_interval_seconds: nil, cleanup_interval_jobs: nil)
67
75
  raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
68
76
 
69
77
  self.class.instances << self
@@ -76,8 +84,10 @@ module GoodJob # :nodoc:
76
84
  @executor_options[:max_threads] = max_threads
77
85
  @executor_options[:max_queue] = max_threads
78
86
  end
79
- @executor_options[:name] = "GoodJob::Scheduler(queues=#{@performer.name} max_threads=#{@executor_options[:max_threads]})"
87
+ @name = "GoodJob::Scheduler(queues=#{@performer.name} max_threads=#{@executor_options[:max_threads]})"
88
+ @executor_options[:name] = name
80
89
 
90
+ @cleanup_tracker = CleanupTracker.new(cleanup_interval_seconds: cleanup_interval_seconds, cleanup_interval_jobs: cleanup_interval_jobs)
81
91
  create_executor
82
92
  warm_cache if warm_cache_on_initialize
83
93
  end
@@ -172,7 +182,14 @@ module GoodJob # :nodoc:
172
182
  GoodJob._on_thread_error(error) if error
173
183
 
174
184
  instrument("finished_job_task", { result: output, error: thread_error, time: time })
175
- create_task if output
185
+ return unless output
186
+
187
+ @cleanup_tracker.increment
188
+ if @cleanup_tracker.cleanup?
189
+ cleanup
190
+ else
191
+ create_task
192
+ end
176
193
  end
177
194
 
178
195
  # Information about the Scheduler
@@ -210,7 +227,25 @@ module GoodJob # :nodoc:
210
227
  create_task # If cache-warming exhausts the threads, ensure there isn't an executable task remaining
211
228
  end
212
229
  future.add_observer(observer, :call)
230
+ future.execute
231
+ end
232
+
233
+ # Preload existing runnable and future-scheduled jobs
234
+ # @return [void]
235
+ def cleanup
236
+ @cleanup_tracker.reset
213
237
 
238
+ future = Concurrent::Future.new(args: [self, @performer], executor: executor) do |_thr_scheduler, thr_performer|
239
+ Rails.application.executor.wrap do
240
+ thr_performer.cleanup
241
+ end
242
+ end
243
+
244
+ observer = lambda do |_time, _output, thread_error|
245
+ GoodJob._on_thread_error(thread_error) if thread_error
246
+ create_task
247
+ end
248
+ future.add_observer(observer, :call)
214
249
  future.execute
215
250
  end
216
251
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
  module GoodJob
3
3
  # GoodJob gem version.
4
- VERSION = '2.7.4'
4
+ VERSION = '2.8.0'
5
5
  end
data/lib/good_job.rb CHANGED
@@ -131,7 +131,7 @@ module GoodJob
131
131
  # analyze or inspect job performance.
132
132
  # If you are preserving job records this way, use this method regularly to
133
133
  # delete old records and preserve space in your database.
134
- # @params older_than [nil,Numeric,ActiveSupport::Duration] Jobs olders than this will be deleted (default: +86400+).
134
+ # @params older_than [nil,Numeric,ActiveSupport::Duration] Jobs older than this will be deleted (default: +86400+).
135
135
  # @return [Integer] Number of jobs that were deleted.
136
136
  def self.cleanup_preserved_jobs(older_than: nil)
137
137
  older_than ||= GoodJob::Configuration.new({}).cleanup_preserved_jobs_before_seconds_ago
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: good_job
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.7.4
4
+ version: 2.8.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ben Sheldon
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2021-12-16 00:00:00.000000000 Z
11
+ date: 2021-12-31 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activejob
@@ -372,6 +372,7 @@ files:
372
372
  - engine/app/controllers/good_job/cron_entries_controller.rb
373
373
  - engine/app/controllers/good_job/executions_controller.rb
374
374
  - engine/app/controllers/good_job/jobs_controller.rb
375
+ - engine/app/controllers/good_job/processes_controller.rb
375
376
  - engine/app/filters/good_job/base_filter.rb
376
377
  - engine/app/filters/good_job/executions_filter.rb
377
378
  - engine/app/filters/good_job/jobs_filter.rb
@@ -383,6 +384,7 @@ files:
383
384
  - engine/app/views/good_job/jobs/_table.erb
384
385
  - engine/app/views/good_job/jobs/index.html.erb
385
386
  - engine/app/views/good_job/jobs/show.html.erb
387
+ - engine/app/views/good_job/processes/index.html.erb
386
388
  - engine/app/views/good_job/shared/_chart.erb
387
389
  - engine/app/views/good_job/shared/_filter.erb
388
390
  - engine/app/views/good_job/shared/icons/_arrow_clockwise.html.erb
@@ -402,12 +404,16 @@ files:
402
404
  - lib/generators/good_job/templates/update/migrations/01_create_good_jobs.rb.erb
403
405
  - lib/generators/good_job/templates/update/migrations/02_add_cron_at_to_good_jobs.rb.erb
404
406
  - lib/generators/good_job/templates/update/migrations/03_add_cron_key_cron_at_index_to_good_jobs.rb.erb
407
+ - lib/generators/good_job/templates/update/migrations/04_create_good_job_processes.rb.erb
405
408
  - lib/generators/good_job/update_generator.rb
406
409
  - lib/good_job.rb
407
410
  - lib/good_job/active_job_extensions.rb
408
411
  - lib/good_job/active_job_extensions/concurrency.rb
409
412
  - lib/good_job/active_job_job.rb
410
413
  - lib/good_job/adapter.rb
414
+ - lib/good_job/assignable_connection.rb
415
+ - lib/good_job/base_record.rb
416
+ - lib/good_job/cleanup_tracker.rb
411
417
  - lib/good_job/cli.rb
412
418
  - lib/good_job/configuration.rb
413
419
  - lib/good_job/cron_entry.rb
@@ -423,8 +429,10 @@ files:
423
429
  - lib/good_job/log_subscriber.rb
424
430
  - lib/good_job/multi_scheduler.rb
425
431
  - lib/good_job/notifier.rb
432
+ - lib/good_job/notifier/process_registration.rb
426
433
  - lib/good_job/poller.rb
427
434
  - lib/good_job/probe_server.rb
435
+ - lib/good_job/process.rb
428
436
  - lib/good_job/railtie.rb
429
437
  - lib/good_job/scheduler.rb
430
438
  - lib/good_job/version.rb