good_job 0.2.1 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 66c19e40862f09e5c7816911c0a21e8cb88f036980a9c33df62cba058b3993e5
4
- data.tar.gz: 218153476e92ca847e7279d9979c4396ed02f36a41918b32b58bf1fb17b8cd92
3
+ metadata.gz: c20439024a46084c22e46bb42e38589687c2ab2cea47d90705abc4fa042ad449
4
+ data.tar.gz: e5ea8e7fa607a10fd936c62623a39d32b891ef629636e0b04fb3d36aa7ab1f3f
5
5
  SHA512:
6
- metadata.gz: d5b0305c889392d2879ccd818eb98b09a867d85a5b853dc8ec35b5a17fee3490cc082f032827f631b15e0a8e1143ea676eea05738d901f97ec4737adf0a514d0
7
- data.tar.gz: 96a8537b85de48b07f676746642bb43ff18369db559b0c058c21d101a7740a0269a772a3b8629fefe3242d2e8a1df0d34a0f12b85b5928caf6272572fcff0d67
6
+ metadata.gz: 4af34ce400c409a61892c9d35693f69e023b5dbd3d54cacfd6d9b044aaf481c616e3fdb9da53470f79919830a916a08f67550ce807eda1c59a43fa1da4092f07
7
+ data.tar.gz: 81304be6bc9e6925b16fa030d2031377bd90bac894b4c8792cfe5e02ff81677bf04d476b1a3435e951a804e6d284e4d8aa7452c6090969f841dbd6e59a9c301c
@@ -1,8 +1,62 @@
1
1
  # Changelog
2
2
 
3
- ## [Unreleased](https://github.com/bensheldon/good_job/tree/HEAD)
3
+ ## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
4
4
 
5
- [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.0...HEAD)
5
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.5.0...v0.6.0)
6
+
7
+ **Closed issues:**
8
+
9
+ - Improve the command line options [\#32](https://github.com/bensheldon/good_job/issues/32)
10
+ - Allow config.active\_job.queue\_adapter = :good\_job to work [\#5](https://github.com/bensheldon/good_job/issues/5)
11
+
12
+ **Merged pull requests:**
13
+
14
+ - Improve generation of changelog [\#36](https://github.com/bensheldon/good_job/pull/36) ([bensheldon](https://github.com/bensheldon))
15
+ - Update Github Action Workflow for Backlog Project Board [\#35](https://github.com/bensheldon/good_job/pull/35) ([bensheldon](https://github.com/bensheldon))
16
+ - Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
17
+ - Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
18
+ - Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
19
+
20
+ ## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
21
+
22
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.4.0...v0.5.0)
23
+
24
+ **Merged pull requests:**
25
+
26
+ - Update development Ruby to 2.6.6 and gems [\#29](https://github.com/bensheldon/good_job/pull/29) ([bensheldon](https://github.com/bensheldon))
27
+
28
+ ## [v0.4.0](https://github.com/bensheldon/good_job/tree/v0.4.0) (2020-03-31)
29
+
30
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.3.0...v0.4.0)
31
+
32
+ **Merged pull requests:**
33
+
34
+ - Improve ActiveRecord usage for advisory locking [\#24](https://github.com/bensheldon/good_job/pull/24) ([bensheldon](https://github.com/bensheldon))
35
+ - Remove support for Rails 5.1 [\#23](https://github.com/bensheldon/good_job/pull/23) ([bensheldon](https://github.com/bensheldon))
36
+
37
+ ## [v0.3.0](https://github.com/bensheldon/good_job/tree/v0.3.0) (2020-03-22)
38
+
39
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.2...v0.3.0)
40
+
41
+ **Merged pull requests:**
42
+
43
+ - Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
44
+ - Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
45
+ - Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
46
+ - Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))
47
+
48
+ ## [v0.2.2](https://github.com/bensheldon/good_job/tree/v0.2.2) (2020-03-08)
49
+
50
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.1...v0.2.2)
51
+
52
+ **Merged pull requests:**
53
+
54
+ - Gracefully shutdown Scheduler when executable receives TERM or INT [\#17](https://github.com/bensheldon/good_job/pull/17) ([bensheldon](https://github.com/bensheldon))
55
+ - Update Appraisals [\#16](https://github.com/bensheldon/good_job/pull/16) ([bensheldon](https://github.com/bensheldon))
56
+
57
+ ## [v0.2.1](https://github.com/bensheldon/good_job/tree/v0.2.1) (2020-03-07)
58
+
59
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.0...v0.2.1)
6
60
 
7
61
  **Merged pull requests:**
8
62
 
data/README.md CHANGED
@@ -14,12 +14,12 @@ Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Q
14
14
  Add this line to your application's Gemfile:
15
15
 
16
16
  ```ruby
17
- gem 'good_job', github: 'bensheldon/good_job'
17
+ gem 'good_job'
18
18
  ```
19
19
 
20
20
  And then execute:
21
21
  ```bash
22
- $ bundle
22
+ $ bundle install
23
23
  ```
24
24
 
25
25
  ## Usage
@@ -56,17 +56,51 @@ $ bundle
56
56
 
57
57
  1. Configure the ActiveJob adapter:
58
58
  ```ruby
59
- # config/environments/production.rb
60
- config.active_job.queue_adapter = GoodJob::Adapter.new
61
-
59
+ # config/application.rb
60
+ config.active_job.queue_adapter = :good_job
61
+ ```
62
+
63
+ By default, using `:good_job` is equivalent to manually configuring the adapter:
64
+
65
+ ```ruby
62
66
  # config/environments/development.rb
63
67
  config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
68
+
69
+ # config/environments/test.rb
70
+ config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
71
+
72
+ # config/environments/production.rb
73
+ config.active_job.queue_adapter = GoodJob::Adapter.new
64
74
  ```
65
75
 
66
76
  1. In production, the scheduler is designed to run in its own process:
67
77
  ```bash
68
78
  $ bundle exec good_job
69
79
  ```
80
+
81
+ Configuration options available with `help`:
82
+ ```bash
83
+ $ bundle exec good_job help start
84
+
85
+ # Usage:
86
+ # good_job start
87
+ #
88
+ # Options:
89
+ # [--max-threads=N] # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
90
+ # [--queues=queue1,queue2] # Queues to work from. Separate multiple queues with commas (default: *)
91
+ # [--poll-interval=N] # Interval between polls for available jobs in seconds (default: 1)
92
+ ```
93
+
94
+ ### Configuring Job Execution Threads
95
+
96
+ GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
97
+
98
+ - Each GoodJob execution thread requires its own database connection, which are automatically checked out from Rails’s connection pool. _Allowing GoodJob to schedule more threads than are available in the database connection pool can lead to timeouts and is not recommended._
99
+ - The maximum number of GoodJob threads can be configured, in decreasing precedence:
100
+ 1. `$ bundle exec good_job --max_threads 4`
101
+ 2. `$ GOOD_JOB_MAX_THREADS=4 bundle exec good_job`
102
+ 3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
103
+ 4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
70
104
 
71
105
  ## Development
72
106
 
@@ -83,6 +117,17 @@ $ bin/setup_test
83
117
  $ bin/rspec
84
118
  ```
85
119
 
120
+ This gem uses Appraisal to run tests against multiple versions of Rails:
121
+
122
+ ```bash
123
+ # Install Appraisal(s) gemfiles
124
+ $ bundle exec appraisal
125
+
126
+ # Run tests
127
+ $ bundle exec appraisal bin/rspec
128
+
129
+ ```
130
+
86
131
  For developing locally within another Ruby on Rails project:
87
132
 
88
133
  ```bash
@@ -103,17 +148,10 @@ Package maintainers can release this gem with the following [gem-release](https:
103
148
  # Sign into rubygems
104
149
  $ gem signin
105
150
 
106
- # Increase the version number
107
- $ gem bump -v minor --no-commit
108
-
109
- # Update the changelog
110
- $ bundle exec rake changelog
111
-
112
- # Commit the version and changelog to git
113
- $ bundle exec rake commit_version
151
+ # Update version number, changelog, and create git commit:
152
+ $ bundle exec rake commit_version[minor] # major,minor,patch
114
153
 
115
- # Push to rubygems.org
116
- $ gem release
154
+ # ..and follow subsequent directions.
117
155
  ```
118
156
 
119
157
  ## Contributing
@@ -0,0 +1,13 @@
1
+ module ActiveJob
2
+ module QueueAdapters
3
+ class GoodJobAdapter < GoodJob::Adapter
4
+ def initialize
5
+ if Rails.env.development? || Rails.env.test?
6
+ super(inline: true)
7
+ else
8
+ super(inline: false)
9
+ end
10
+ end
11
+ end
12
+ end
13
+ end
@@ -4,10 +4,11 @@ require 'good_job/railtie'
4
4
  require 'good_job/logging'
5
5
  require 'good_job/lockable'
6
6
  require 'good_job/job'
7
- require 'good_job/inline_scheduler'
8
7
  require "good_job/scheduler"
9
- require "good_job/job_wrapper"
10
8
  require 'good_job/adapter'
9
+ require 'good_job/pg_locks'
10
+
11
+ require 'active_job/queue_adapters/good_job_adapter'
11
12
 
12
13
  module GoodJob
13
14
  include Logging
@@ -1,41 +1,37 @@
1
1
  module GoodJob
2
2
  class Adapter
3
- def initialize(options = {})
4
- @options = options
5
- @scheduler = InlineScheduler.new if inline?
3
+ def initialize(inline: false)
4
+ @inline = inline
6
5
  end
7
6
 
8
- def enqueue(job)
9
- enqueue_at(job, nil)
7
+ def enqueue(active_job)
8
+ enqueue_at(active_job, nil)
10
9
  end
11
10
 
12
- def enqueue_at(job, timestamp)
13
- params = {
14
- queue_name: job.queue_name,
15
- priority: job.priority,
16
- serialized_params: job.serialize,
17
- }
18
- params[:scheduled_at] = Time.at(timestamp) if timestamp
19
-
20
- good_job = GoodJob::Job.create(params)
21
- job.provider_job_id = good_job.id
22
-
23
- GoodJob.tag_logger do
24
- ActiveSupport::Notifications.instrument("create.good_job", { good_job: good_job, job: job })
25
- @scheduler.enqueue(good_job) if inline?
11
+ def enqueue_at(active_job, timestamp)
12
+ good_job = GoodJob::Job.enqueue(
13
+ active_job,
14
+ scheduled_at: timestamp ? Time.at(timestamp) : nil,
15
+ create_with_advisory_lock: inline?
16
+ )
17
+
18
+ if inline?
19
+ begin
20
+ good_job.perform
21
+ ensure
22
+ good_job.advisory_unlock
23
+ end
26
24
  end
27
25
 
28
26
  good_job
29
27
  end
30
28
 
31
- def shutdown(wait: true)
32
- @scheduler&.shutdown(wait: wait)
29
+ def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
30
+ nil
33
31
  end
34
32
 
35
- private
36
-
37
33
  def inline?
38
- @options.fetch(:inline, false)
34
+ @inline
39
35
  end
40
36
  end
41
37
  end
@@ -4,15 +4,68 @@ module GoodJob
4
4
  class CLI < Thor
5
5
  RAILS_ENVIRONMENT_RB = File.expand_path("config/environment.rb")
6
6
 
7
- desc :start, "Start jobs"
7
+ desc :start, "Start job worker"
8
+ method_option :max_threads,
9
+ type: :numeric,
10
+ desc: "Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)"
11
+ method_option :queues,
12
+ type: :string,
13
+ banner: "queue1,queue2",
14
+ desc: "Queues to work from. Separate multiple queues with commas (default: *)"
15
+ method_option :poll_interval,
16
+ type: :numeric,
17
+ desc: "Interval between polls for available jobs in seconds (default: 1)"
8
18
  def start
9
19
  require RAILS_ENVIRONMENT_RB
10
20
 
11
- GoodJob::Scheduler.new
21
+ max_threads = (
22
+ options[:max_threads] ||
23
+ ENV['GOOD_JOB_MAX_THREADS'] ||
24
+ ENV['RAILS_MAX_THREADS'] ||
25
+ ActiveRecord::Base.connection_pool.size
26
+ ).to_i
27
+
28
+ queue_names = (
29
+ options[:queues] ||
30
+ ENV['GOOD_JOB_QUEUES'] ||
31
+ '*'
32
+ ).split(',').map(&:strip)
33
+
34
+ poll_interval = (
35
+ options[:poll_interval] ||
36
+ ENV['GOOD_JOB_POLL_INTERVAL']
37
+ ).to_i
38
+
39
+ job_query = GoodJob::Job.all
40
+ queue_names_without_all = queue_names.reject { |q| q == '*' }
41
+ job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
42
+
43
+ job_performer = job_query.only_scheduled.priority_ordered.to_performer
44
+
45
+ $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
46
+
47
+ timer_options = {}
48
+ timer_options[:execution_interval] = poll_interval if poll_interval.positive?
49
+
50
+ pool_options = {
51
+ max_threads: max_threads,
52
+ }
53
+
54
+ scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
55
+
56
+ @stop_good_job_executable = false
57
+ %w[INT TERM].each do |signal|
58
+ trap(signal) { @stop_good_job_executable = true }
59
+ end
12
60
 
13
61
  Kernel.loop do
14
- sleep 1
62
+ sleep 0.1
63
+ break if @stop_good_job_executable || scheduler.shutdown?
15
64
  end
65
+
66
+ $stdout.puts "\nFinishing GoodJob's current jobs before exiting..."
67
+ scheduler.shutdown
68
+ $stdout.puts "GoodJob's jobs finished, exiting..."
16
69
  end
17
70
 
18
71
  default_task :start
@@ -1,6 +1,62 @@
1
1
  module GoodJob
2
2
  class Job < ActiveRecord::Base
3
3
  include Lockable
4
+
4
5
  self.table_name = 'good_jobs'
6
+
7
+ scope :only_scheduled, -> { where("scheduled_at < ?", Time.current).or(where(scheduled_at: nil)) }
8
+ scope :priority_ordered, -> { order(priority: :desc) }
9
+ scope :to_performer, -> { Performer.new(self) }
10
+
11
+ class Performer
12
+ def initialize(query)
13
+ @query = query
14
+ end
15
+
16
+ def next
17
+ good_job = nil
18
+
19
+ @query.limit(1).with_advisory_lock do |good_jobs|
20
+ good_job = good_jobs.first
21
+ break unless good_job
22
+
23
+ good_job.perform
24
+ end
25
+
26
+ good_job
27
+ end
28
+ end
29
+
30
+ def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
31
+ good_job = nil
32
+ ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
33
+ good_job = GoodJob::Job.new(
34
+ queue_name: active_job.queue_name,
35
+ priority: active_job.priority,
36
+ serialized_params: active_job.serialize,
37
+ scheduled_at: scheduled_at,
38
+ create_with_advisory_lock: create_with_advisory_lock
39
+ )
40
+
41
+ instrument_payload[:good_job] = good_job
42
+
43
+ good_job.save!
44
+ active_job.provider_job_id = good_job.id
45
+ end
46
+
47
+ good_job
48
+ end
49
+
50
+ def perform
51
+ ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
52
+ ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
53
+ params = serialized_params.merge(
54
+ "provider_job_id" => id
55
+ )
56
+ ActiveJob::Base.execute(params)
57
+
58
+ destroy!
59
+ end
60
+ end
5
61
  end
6
62
  end
@@ -5,98 +5,97 @@ module GoodJob
5
5
  RecordAlreadyAdvisoryLockedError = Class.new(StandardError)
6
6
 
7
7
  included do
8
+ scope :advisory_lock, (lambda do
9
+ original_query = self
10
+
11
+ cte_table = Arel::Table.new(:rows)
12
+ composed_cte = Arel::Nodes::As.new(cte_table, original_query.select(primary_key).except(:limit).arel)
13
+
14
+ query = cte_table.project(cte_table[:id])
15
+ .with(composed_cte)
16
+ .where(Arel.sql(sanitize_sql_for_conditions(["pg_try_advisory_lock(('x'||substr(md5(:table_name || \"#{cte_table.name}\".\"#{primary_key}\"::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
17
+
18
+ limit = original_query.arel.ast.limit
19
+ query.limit = limit.value if limit.present?
20
+
21
+ unscoped.where(arel_table[:id].in(query)).merge(original_query.only(:order))
22
+ end)
23
+
8
24
  scope :joins_advisory_locks, (lambda do
9
- joins(<<~SQL)
25
+ join_sql = <<~SQL
10
26
  LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
11
27
  AND pg_locks.objsubid = 1
12
- AND pg_locks.classid = ('x'||substr(md5(good_jobs.id::text), 1, 16))::bit(32)::int
13
- AND pg_locks.objid = (('x'||substr(md5(good_jobs.id::text), 1, 16))::bit(64) << 32)::bit(32)::int
28
+ AND pg_locks.classid = ('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(32)::int
29
+ AND pg_locks.objid = (('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(64) << 32)::bit(32)::int
14
30
  SQL
31
+
32
+ joins(sanitize_sql_for_conditions([join_sql, { table_name: table_name }]))
15
33
  end)
16
34
 
17
35
  scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
18
- scope :with_advisory_lock, (lambda do
19
- where(<<~SQL)
20
- pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
21
- SQL
22
- end)
36
+ scope :advisory_locked, -> { joins_advisory_locks.where.not(pg_locks: { locktype: nil }) }
37
+ scope :owns_advisory_locked, -> { joins_advisory_locks.where('"pg_locks"."pid" = pg_backend_pid()') }
23
38
 
24
- def self.first_advisory_locked_row(query)
25
- find_by_sql(<<~SQL)
26
- WITH rows AS (#{query.to_sql})
27
- SELECT rows.id
28
- FROM rows
29
- WHERE pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
30
- SQL
31
- end
32
- # private_class_method :first_advisory_locked_row
33
-
34
- # https://www.postgresql.org/docs/9.6/view-pg-locks.html
35
- # Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
36
- # A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
37
- # The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
38
- # Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
39
- # The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
40
- def self.advisory_lock_details
41
- connection.select("SELECT * FROM pg_locks WHERE locktype = 'advisory' AND objsubid = 1")
42
- end
39
+ attr_accessor :create_with_advisory_lock
40
+ after_create -> { advisory_lock }, if: :create_with_advisory_lock
41
+ end
43
42
 
44
- def advisory_lock
45
- self.class.connection.execute(sanitize_sql_for_conditions(["SELECT 1 as one WHERE pg_try_advisory_lock(('x'||substr(md5(?), 1, 16))::bit(64)::bigint)", id])).ntuples.positive?
43
+ class_methods do
44
+ def with_advisory_lock(&block)
45
+ records = advisory_lock.to_a
46
+ begin
47
+ block.call(records)
48
+ ensure
49
+ records.each(&:advisory_unlock)
50
+ end
46
51
  end
52
+ end
47
53
 
48
- def advisory_lock!
49
- result = advisory_lock
50
- result || raise(RecordAlreadyAdvisoryLockedError)
51
- end
54
+ def advisory_lock
55
+ query = <<~SQL
56
+ SELECT 1 AS one
57
+ WHERE pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
58
+ SQL
59
+ self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
60
+ end
52
61
 
53
- def with_advisory_lock
54
- advisory_lock!
55
- yield
56
- rescue StandardError => e
57
- advisory_unlock unless e.is_a? RecordAlreadyAdvisoryLockedError
58
- raise
59
- end
62
+ def advisory_unlock
63
+ query = <<~SQL
64
+ SELECT 1 AS one
65
+ WHERE pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
66
+ SQL
67
+ self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
68
+ end
60
69
 
61
- def advisory_locked?
62
- self.class.connection.execute(<<~SQL).ntuples.positive?
63
- SELECT 1 as one
64
- FROM pg_locks
65
- WHERE
66
- locktype = 'advisory'
67
- AND objsubid = 1
68
- AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
69
- AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
70
- SQL
71
- end
70
+ def advisory_lock!
71
+ result = advisory_lock
72
+ result || raise(RecordAlreadyAdvisoryLockedError)
73
+ end
72
74
 
73
- def owns_advisory_lock?
74
- self.class.connection.execute(<<~SQL).ntuples.positive?
75
- SELECT 1 as one
76
- FROM pg_locks
77
- WHERE
78
- locktype = 'advisory'
79
- AND objsubid = 1
80
- AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
81
- AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
82
- AND pid = pg_backend_pid()
83
- SQL
84
- end
75
+ def with_advisory_lock
76
+ advisory_lock!
77
+ yield
78
+ ensure
79
+ advisory_unlock unless $ERROR_INFO.is_a? RecordAlreadyAdvisoryLockedError
80
+ end
85
81
 
86
- def advisory_unlock
87
- self.class.connection.execute("SELECT pg_advisory_unlock(('x'||substr(md5('#{id}'), 1, 16))::bit(64)::bigint)").first["pg_advisory_unlock"]
88
- end
82
+ def advisory_locked?
83
+ self.class.advisory_locked.where(id: send(self.class.primary_key)).any?
84
+ end
89
85
 
90
- def advisory_unlock!
91
- advisory_unlock while advisory_locked?
92
- end
86
+ def owns_advisory_lock?
87
+ self.class.owns_advisory_locked.where(id: send(self.class.primary_key)).any?
88
+ end
93
89
 
94
- private
90
+ def advisory_unlock!
91
+ advisory_unlock while advisory_locked?
92
+ end
95
93
 
96
- def sanitize_sql_for_conditions(*args)
97
- # Made public in Rails 5.2
98
- self.class.send(:sanitize_sql_for_conditions, *args)
99
- end
94
+ private
95
+
96
+ def sanitize_sql_for_conditions(*args)
97
+ # Made public in Rails 5.2
98
+ self.class.send(:sanitize_sql_for_conditions, *args)
100
99
  end
101
100
  end
102
101
  end
@@ -42,6 +42,18 @@ module GoodJob
42
42
  end
43
43
  end
44
44
 
45
+ def scheduler_start_shutdown(_event)
46
+ info do
47
+ "Shutting down scheduler..."
48
+ end
49
+ end
50
+
51
+ def scheduler_shutdown(_event)
52
+ info do
53
+ "Scheduler is shut down."
54
+ end
55
+ end
56
+
45
57
  private
46
58
 
47
59
  def logger
@@ -0,0 +1,21 @@
1
+ module GoodJob
2
+ class PgLocks < ActiveRecord::Base
3
+ self.table_name = 'pg_locks'.freeze
4
+
5
+ # https://www.postgresql.org/docs/9.6/view-pg-locks.html
6
+ # Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
7
+ # A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
8
+ # The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
9
+ # Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
10
+ # The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
11
+ def self.advisory_lock_details
12
+ connection.select <<~SQL
13
+ SELECT *
14
+ FROM pg_locks
15
+ WHERE
16
+ locktype = 'advisory' AND
17
+ objsubid = 1
18
+ SQL
19
+ end
20
+ end
21
+ end
@@ -1,11 +1,9 @@
1
- require "concurrent/scheduled_task"
2
1
  require "concurrent/executor/thread_pool_executor"
2
+ require "concurrent/timer_task"
3
3
  require "concurrent/utility/processor_counter"
4
4
 
5
5
  module GoodJob
6
6
  class Scheduler
7
- MAX_THREADS = Concurrent.processor_count
8
-
9
7
  DEFAULT_TIMER_OPTIONS = {
10
8
  execution_interval: 1,
11
9
  timeout_interval: 1,
@@ -15,77 +13,67 @@ module GoodJob
15
13
  DEFAULT_POOL_OPTIONS = {
16
14
  name: 'good_job',
17
15
  min_threads: 0,
18
- max_threads: MAX_THREADS,
16
+ max_threads: Concurrent.processor_count,
19
17
  auto_terminate: true,
20
18
  idletime: 0,
21
19
  max_queue: 0,
22
20
  fallback_policy: :abort, # shouldn't matter -- 0 max queue
23
21
  }.freeze
24
22
 
25
- def initialize(query = GoodJob::Job.all, **_options)
26
- @query = query
23
+ def initialize(performer, timer_options: {}, pool_options: {})
24
+ raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
27
25
 
28
- @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS)
29
- @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS) do
26
+ @performer = performer
27
+ @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
28
+ @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
30
29
  idle_threads = @pool.max_length - @pool.length
31
30
  create_thread if idle_threads.positive?
32
31
  end
33
- @timer.add_observer(TimerObserver.new)
32
+ @timer.add_observer(self, :timer_observer)
34
33
  @timer.execute
35
34
  end
36
35
 
37
- def ordered_query
38
- @query.where("scheduled_at < ?", Time.current).or(@query.where(scheduled_at: nil)).order(priority: :desc)
39
- end
40
-
41
36
  def execute
42
37
  end
43
38
 
44
39
  def shutdown(wait: true)
45
- if @timer.running?
46
- @timer.shutdown
47
- @timer.wait_for_termination if wait
48
- end
40
+ @_shutdown = true
49
41
 
50
- if @pool.running?
51
- @pool.shutdown
52
- @pool.wait_for_termination if wait
42
+ ActiveSupport::Notifications.instrument("scheduler_start_shutdown.good_job", { wait: wait })
43
+ ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait }) do
44
+ if @timer.running?
45
+ @timer.shutdown
46
+ @timer.wait_for_termination if wait
47
+ end
48
+
49
+ if @pool.running?
50
+ @pool.shutdown
51
+ @pool.wait_for_termination if wait
52
+ end
53
53
  end
54
+ end
54
55
 
55
- true
56
+ def shutdown?
57
+ @_shutdown
56
58
  end
57
59
 
58
60
  def create_thread
59
- future = Concurrent::Future.new(args: [ordered_query], executor: @pool) do |query|
60
- Rails.application.executor.wrap do
61
- loop do
62
- good_job = query.with_advisory_lock.first
63
- break unless good_job
64
-
65
- ActiveSupport::Notifications.instrument("job_started.good_job", { good_job: good_job })
66
-
67
- JobWrapper.new(good_job).perform
68
-
69
- good_job.advisory_unlock
70
- end
71
- end
72
-
73
- true
61
+ future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
62
+ result = nil
63
+ Rails.application.executor.wrap { result = performer.next }
64
+ result
74
65
  end
75
- future.add_observer(TaskObserver.new)
66
+ future.add_observer(self, :task_observer)
76
67
  future.execute
77
68
  end
78
69
 
79
- class TimerObserver
80
- def update(time, result, error)
81
- ActiveSupport::Notifications.instrument("timer_task_finished.good_job", { result: result, error: error, time: time })
82
- end
70
+ def timer_observer(time, executed_task, error)
71
+ ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
83
72
  end
84
73
 
85
- class TaskObserver
86
- def update(time, result, error)
87
- ActiveSupport::Notifications.instrument("job_finished.good_job", { result: result, error: error, time: time })
88
- end
74
+ def task_observer(time, result, error)
75
+ ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: result, error: error, time: time })
76
+ create_thread if result
89
77
  end
90
78
  end
91
79
  end
@@ -1,3 +1,3 @@
1
1
  module GoodJob
2
- VERSION = '0.2.1'.freeze
2
+ VERSION = '0.6.0'.freeze
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: good_job
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.1
4
+ version: 0.6.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ben Sheldon
8
8
  autorequire:
9
9
  bindir: exe
10
10
  cert_chain: []
11
- date: 2020-03-07 00:00:00.000000000 Z
11
+ date: 2020-07-15 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: concurrent-ruby
@@ -94,6 +94,20 @@ dependencies:
94
94
  - - ">="
95
95
  - !ruby/object:Gem::Version
96
96
  version: '0'
97
+ - !ruby/object:Gem::Dependency
98
+ name: foreman
99
+ requirement: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ type: :development
105
+ prerelease: false
106
+ version_requirements: !ruby/object:Gem::Requirement
107
+ requirements:
108
+ - - ">="
109
+ - !ruby/object:Gem::Version
110
+ version: '0'
97
111
  - !ruby/object:Gem::Dependency
98
112
  name: gem-release
99
113
  requirement: !ruby/object:Gem::Requirement
@@ -122,6 +136,20 @@ dependencies:
122
136
  - - ">="
123
137
  - !ruby/object:Gem::Version
124
138
  version: '0'
139
+ - !ruby/object:Gem::Dependency
140
+ name: pry
141
+ requirement: !ruby/object:Gem::Requirement
142
+ requirements:
143
+ - - ">="
144
+ - !ruby/object:Gem::Version
145
+ version: '0'
146
+ type: :development
147
+ prerelease: false
148
+ version_requirements: !ruby/object:Gem::Requirement
149
+ requirements:
150
+ - - ">="
151
+ - !ruby/object:Gem::Version
152
+ version: '0'
125
153
  - !ruby/object:Gem::Dependency
126
154
  name: rspec-rails
127
155
  requirement: !ruby/object:Gem::Requirement
@@ -179,14 +207,14 @@ files:
179
207
  - LICENSE.txt
180
208
  - README.md
181
209
  - exe/good_job
210
+ - lib/active_job/queue_adapters/good_job_adapter.rb
182
211
  - lib/good_job.rb
183
212
  - lib/good_job/adapter.rb
184
213
  - lib/good_job/cli.rb
185
- - lib/good_job/inline_scheduler.rb
186
214
  - lib/good_job/job.rb
187
- - lib/good_job/job_wrapper.rb
188
215
  - lib/good_job/lockable.rb
189
216
  - lib/good_job/logging.rb
217
+ - lib/good_job/pg_locks.rb
190
218
  - lib/good_job/railtie.rb
191
219
  - lib/good_job/scheduler.rb
192
220
  - lib/good_job/version.rb
@@ -221,7 +249,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
221
249
  - !ruby/object:Gem::Version
222
250
  version: '0'
223
251
  requirements: []
224
- rubygems_version: 3.1.2
252
+ rubygems_version: 3.0.3
225
253
  signing_key:
226
254
  specification_version: 4
227
255
  summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
@@ -1,10 +0,0 @@
1
- module GoodJob
2
- class InlineScheduler
3
- def enqueue(good_job)
4
- JobWrapper.new(good_job).perform
5
- end
6
-
7
- def shutdown(wait: true)
8
- end
9
- end
10
- end
@@ -1,16 +0,0 @@
1
- module GoodJob
2
- class JobWrapper
3
- def initialize(good_job)
4
- @good_job = good_job
5
- end
6
-
7
- def perform
8
- serialized_params = @good_job.serialized_params.merge(
9
- "provider_job_id" => @good_job.id
10
- )
11
- ActiveJob::Base.execute(serialized_params)
12
-
13
- @good_job.destroy!
14
- end
15
- end
16
- end