good_job 0.2.0 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2c0fc31238c6d66b5b9ed3ddc0d4b9393a0a5a2cca85c7fcd0f2f24650892675
4
- data.tar.gz: d6484d63b86eb7bc748b308cbfebe1ca7798fa1fa77d9542754dc99504f12244
3
+ metadata.gz: c11065689d3442c14ab8afb08ad17a96c45414287e9c24440d26be766eff3566
4
+ data.tar.gz: 316168ca2ecf9aa9a239d007b85b589c34ec68064ad3ec376665bb893aceae4c
5
5
  SHA512:
6
- metadata.gz: 7e4d9fc5ae59750231906be49188aba712f1bdfbe2d66ed441f82c69ad267b0a056fed12998fe594c33c862f81d36d912a56d598c8e436e1e0ea225634abe66a
7
- data.tar.gz: b02e2c08035b4a90713a0378e2b83173232baec7bf6b22ecbd56033d8fd38e78be0e3d992a7ca4e3339485014bc97fac62efd3d3fd94df195fceb9248703f004
6
+ metadata.gz: 9cb0b54f3e2a7189684fb5f3ea98ea4febd6e1651ff7f9f47b78be19e9a4948c731c9e990e89921b3763236a33cbf45905d8c05c417d5044caaf844b67600510
7
+ data.tar.gz: 8f45cc0b5ccd1c61c3e55ac748359156d3f4081646537b327476bd964e15e45e6a953c01ae0f237b0c08fd7bcbd3bd94bf9d7bb484c45d26600790f7870b9ccf
@@ -0,0 +1,77 @@
1
+ # Changelog
2
+
3
+ ## [0.5.0](https://github.com/bensheldon/good_job/tree/0.5.0) (2020-07-13)
4
+
5
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.4.0...0.5.0)
6
+
7
+ **Merged pull requests:**
8
+
9
+ - Update development Ruby to 2.6.6 and gems [\#29](https://github.com/bensheldon/good_job/pull/29) ([bensheldon](https://github.com/bensheldon))
10
+ - Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
11
+
12
+ ## [v0.4.0](https://github.com/bensheldon/good_job/tree/v0.4.0) (2020-03-31)
13
+
14
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.3.0...v0.4.0)
15
+
16
+ **Merged pull requests:**
17
+
18
+ - Improve ActiveRecord usage for advisory locking [\#24](https://github.com/bensheldon/good_job/pull/24) ([bensheldon](https://github.com/bensheldon))
19
+ - Remove support for Rails 5.1 [\#23](https://github.com/bensheldon/good_job/pull/23) ([bensheldon](https://github.com/bensheldon))
20
+
21
+ ## [v0.3.0](https://github.com/bensheldon/good_job/tree/v0.3.0) (2020-03-22)
22
+
23
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.2...v0.3.0)
24
+
25
+ **Merged pull requests:**
26
+
27
+ - Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
28
+ - Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
29
+ - Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
30
+ - Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))
31
+
32
+ ## [v0.2.2](https://github.com/bensheldon/good_job/tree/v0.2.2) (2020-03-08)
33
+
34
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.1...v0.2.2)
35
+
36
+ **Merged pull requests:**
37
+
38
+ - Gracefully shutdown Scheduler when executable receives TERM or INT [\#17](https://github.com/bensheldon/good_job/pull/17) ([bensheldon](https://github.com/bensheldon))
39
+ - Update Appraisals [\#16](https://github.com/bensheldon/good_job/pull/16) ([bensheldon](https://github.com/bensheldon))
40
+
41
+ ## [v0.2.1](https://github.com/bensheldon/good_job/tree/v0.2.1) (2020-03-07)
42
+
43
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.0...v0.2.1)
44
+
45
+ **Merged pull requests:**
46
+
47
+ - Clean up Gemspec [\#15](https://github.com/bensheldon/good_job/pull/15) ([bensheldon](https://github.com/bensheldon))
48
+ - Set up Rubocop [\#14](https://github.com/bensheldon/good_job/pull/14) ([bensheldon](https://github.com/bensheldon))
49
+ - Bump nokogiri from 1.10.7 to 1.10.9 [\#12](https://github.com/bensheldon/good_job/pull/12) ([dependabot[bot]](https://github.com/apps/dependabot))
50
+ - Add Appraisal with tests for Rails 5.1, 5.2, 6.0 [\#11](https://github.com/bensheldon/good_job/pull/11) ([bensheldon](https://github.com/bensheldon))
51
+
52
+ ## [v0.2.0](https://github.com/bensheldon/good_job/tree/v0.2.0) (2020-03-06)
53
+
54
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.1.0...v0.2.0)
55
+
56
+ **Merged pull requests:**
57
+
58
+ - Add pg gem as explicit dependency [\#13](https://github.com/bensheldon/good_job/pull/13) ([bensheldon](https://github.com/bensheldon))
59
+ - Use Rails.logger and ActiveSupport::Notifications for logging instead of puts [\#10](https://github.com/bensheldon/good_job/pull/10) ([bensheldon](https://github.com/bensheldon))
60
+ - Remove minitest files [\#9](https://github.com/bensheldon/good_job/pull/9) ([bensheldon](https://github.com/bensheldon))
61
+ - Use scheduled\_at and priority for scheduling [\#8](https://github.com/bensheldon/good_job/pull/8) ([bensheldon](https://github.com/bensheldon))
62
+ - Create Github Action workflow for PRs and Issues [\#7](https://github.com/bensheldon/good_job/pull/7) ([bensheldon](https://github.com/bensheldon))
63
+
64
+ ## [v0.1.0](https://github.com/bensheldon/good_job/tree/v0.1.0) (2020-03-03)
65
+
66
+ [Full Changelog](https://github.com/bensheldon/good_job/compare/6866006239f1a6b7fcb7103f5df60d904952fb84...v0.1.0)
67
+
68
+ **Merged pull requests:**
69
+
70
+ - Add executable with Thor [\#4](https://github.com/bensheldon/good_job/pull/4) ([bensheldon](https://github.com/bensheldon))
71
+ - Refactor adapter enqueing methods; expand Readme, tests, editorconfig [\#3](https://github.com/bensheldon/good_job/pull/3) ([bensheldon](https://github.com/bensheldon))
72
+ - Fetch new jobs within the worker thread itself; incrementally grow worker threads [\#2](https://github.com/bensheldon/good_job/pull/2) ([bensheldon](https://github.com/bensheldon))
73
+ - Set up Github Workflows for tests [\#1](https://github.com/bensheldon/good_job/pull/1) ([bensheldon](https://github.com/bensheldon))
74
+
75
+
76
+
77
+ \* *This Changelog was automatically generated by [github_changelog_generator](https://github.com/github-changelog-generator/github-changelog-generator)*
data/README.md CHANGED
@@ -14,12 +14,12 @@ Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Q
14
14
  Add this line to your application's Gemfile:
15
15
 
16
16
  ```ruby
17
- gem 'good_job', github: 'bensheldon/good_job'
17
+ gem 'good_job'
18
18
  ```
19
19
 
20
20
  And then execute:
21
21
  ```bash
22
- $ bundle
22
+ $ bundle install
23
23
  ```
24
24
 
25
25
  ## Usage
@@ -56,17 +56,38 @@ $ bundle
56
56
 
57
57
  1. Configure the ActiveJob adapter:
58
58
  ```ruby
59
- # config/environments/production.rb
60
- config.active_job.queue_adapter = GoodJob::Adapter.new
61
-
59
+ # config/application.rb
60
+ config.active_job.queue_adapter = :good_job
61
+ ```
62
+
63
+ By default, using `:good_job` is equivalent to manually configuring the adapter:
64
+
65
+ ```ruby
62
66
  # config/environments/development.rb
63
67
  config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
68
+
69
+ # config/environments/test.rb
70
+ config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
71
+
72
+ # config/environments/production.rb
73
+ config.active_job.queue_adapter = GoodJob::Adapter.new
64
74
  ```
65
75
 
66
76
  1. In production, the scheduler is designed to run in its own process:
67
77
  ```bash
68
78
  $ bundle exec good_job
69
79
  ```
80
+
81
+ ### Configuring Job Execution Threads
82
+
83
+ GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
84
+
85
+ - Each GoodJob execution thread requires its own database connection, which are automatically checked out from Rails’s connection pool. _Allowing GoodJob to schedule more threads than are available in the database connection pool can lead to timeouts and is not recommended._
86
+ - The maximum number of GoodJob threads can be configured, in decreasing precedence:
87
+ 1. `$ bundle exec good_job --max_threads 4`
88
+ 2. `$ GOOD_JOB_MAX_THREADS=4 bundle exec good_job`
89
+ 3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
90
+ 4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
70
91
 
71
92
  ## Development
72
93
 
@@ -83,6 +104,17 @@ $ bin/setup_test
83
104
  $ bin/rspec
84
105
  ```
85
106
 
107
+ This gem uses Appraisal to run tests against multiple versions of Rails:
108
+
109
+ ```bash
110
+ # Install Appraisal(s) gemfiles
111
+ $ bundle exec appraisal
112
+
113
+ # Run tests
114
+ $ bundle exec appraisal bin/rspec
115
+
116
+ ```
117
+
86
118
  For developing locally within another Ruby on Rails project:
87
119
 
88
120
  ```bash
@@ -103,17 +135,10 @@ Package maintainers can release this gem with the following [gem-release](https:
103
135
  # Sign into rubygems
104
136
  $ gem signin
105
137
 
106
- # Increase the version number
107
- $ gem bump -v minor --no-commit
108
-
109
- # Update the changelog
110
- $ bundle exec changelog
111
-
112
- # Commit the version and changelog to git
113
- $ bundle commit_version
138
+ # Update version number, changelog, and create git commit:
139
+ $ bundle exec rake commit_version
114
140
 
115
- # Push to rubygems.org
116
- $ gem release
141
+ # ..and follow subsequent directions.
117
142
  ```
118
143
 
119
144
  ## Contributing
File without changes
@@ -0,0 +1,13 @@
1
+ module ActiveJob
2
+ module QueueAdapters
3
+ class GoodJobAdapter < GoodJob::Adapter
4
+ def initialize
5
+ if Rails.env.development? || Rails.env.test?
6
+ super(inline: true)
7
+ else
8
+ super(inline: false)
9
+ end
10
+ end
11
+ end
12
+ end
13
+ end
@@ -4,10 +4,11 @@ require 'good_job/railtie'
4
4
  require 'good_job/logging'
5
5
  require 'good_job/lockable'
6
6
  require 'good_job/job'
7
- require 'good_job/inline_scheduler'
8
7
  require "good_job/scheduler"
9
- require "good_job/job_wrapper"
10
8
  require 'good_job/adapter'
9
+ require 'good_job/pg_locks'
10
+
11
+ require 'active_job/queue_adapters/good_job_adapter'
11
12
 
12
13
  module GoodJob
13
14
  include Logging
@@ -1,39 +1,34 @@
1
1
  module GoodJob
2
2
  class Adapter
3
- def initialize(options = {})
4
- @options = options
5
- @scheduler = InlineScheduler.new if inline?
3
+ def initialize(inline: false)
4
+ @inline = inline
6
5
  end
7
6
 
8
- def inline?
9
- @options.fetch(:inline, false)
10
- end
11
-
12
- def enqueue(job)
13
- enqueue_at(job, nil)
7
+ def enqueue(active_job)
8
+ enqueue_at(active_job, nil)
14
9
  end
15
10
 
16
- def enqueue_at(job, timestamp)
17
- params = {
18
- queue_name: job.queue_name,
19
- priority: job.priority,
20
- serialized_params: job.serialize,
21
- }
22
- params[:scheduled_at] = Time.at(timestamp) if timestamp
23
-
24
- good_job = GoodJob::Job.create(params)
25
- job.provider_job_id = good_job.id
11
+ def enqueue_at(active_job, timestamp)
12
+ good_job = GoodJob::Job.enqueue(
13
+ active_job,
14
+ scheduled_at: timestamp ? Time.at(timestamp) : nil,
15
+ create_with_advisory_lock: inline?
16
+ )
26
17
 
27
- GoodJob.tag_logger do
28
- ActiveSupport::Notifications.instrument("create.good_job", { good_job: good_job, job: job })
29
- @scheduler.enqueue(good_job) if inline?
18
+ if inline?
19
+ good_job.perform
20
+ good_job.advisory_unlock
30
21
  end
31
22
 
32
23
  good_job
33
24
  end
34
25
 
35
- def shutdown(wait: true)
36
- @scheduler&.shutdown(wait: wait)
26
+ def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
27
+ nil
28
+ end
29
+
30
+ def inline?
31
+ @inline
37
32
  end
38
33
  end
39
34
  end
@@ -5,13 +5,31 @@ module GoodJob
5
5
  RAILS_ENVIRONMENT_RB = File.expand_path("config/environment.rb")
6
6
 
7
7
  desc :start, "Start jobs"
8
+ method_option :max_threads, type: :numeric
8
9
  def start
9
10
  require RAILS_ENVIRONMENT_RB
10
11
 
11
- scheduler = GoodJob::Scheduler.new
12
+ max_threads = options[:max_threads] ||
13
+ ENV['GOOD_JOB_MAX_THREADS'] ||
14
+ ENV['RAILS_MAX_THREADS'] ||
15
+ ActiveRecord::Base.connection_pool.size
16
+
17
+ $stdout.puts "GoodJob starting with max_threads=#{max_threads}"
18
+ scheduler = GoodJob::Scheduler.new(pool_options: { max_threads: max_threads })
19
+
20
+ %w[INT TERM].each do |signal|
21
+ trap(signal) { @stop_good_job_executable = true }
22
+ end
23
+ @stop_good_job_executable = false
24
+
12
25
  Kernel.loop do
13
- sleep 1
26
+ sleep 0.1
27
+ break if @stop_good_job_executable || scheduler.shutdown?
14
28
  end
29
+
30
+ $stdout.puts "\nFinishing GoodJob's current jobs before exiting..."
31
+ scheduler.shutdown
32
+ $stdout.puts "GoodJob's jobs finished, exiting..."
15
33
  end
16
34
 
17
35
  default_task :start
@@ -1,6 +1,39 @@
1
1
  module GoodJob
2
2
  class Job < ActiveRecord::Base
3
3
  include Lockable
4
+
4
5
  self.table_name = 'good_jobs'
6
+
7
+ def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
8
+ good_job = nil
9
+ ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
10
+ good_job = GoodJob::Job.new(
11
+ queue_name: active_job.queue_name,
12
+ priority: active_job.priority,
13
+ serialized_params: active_job.serialize,
14
+ scheduled_at: scheduled_at,
15
+ create_with_advisory_lock: create_with_advisory_lock
16
+ )
17
+
18
+ instrument_payload[:good_job] = good_job
19
+
20
+ good_job.save!
21
+ active_job.provider_job_id = good_job.id
22
+ end
23
+
24
+ good_job
25
+ end
26
+
27
+ def perform
28
+ ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
29
+ ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
30
+ params = serialized_params.merge(
31
+ "provider_job_id" => id
32
+ )
33
+ ActiveJob::Base.execute(params)
34
+
35
+ destroy!
36
+ end
37
+ end
5
38
  end
6
39
  end
@@ -5,93 +5,97 @@ module GoodJob
5
5
  RecordAlreadyAdvisoryLockedError = Class.new(StandardError)
6
6
 
7
7
  included do
8
+ scope :advisory_lock, (lambda do
9
+ original_query = self
10
+
11
+ cte_table = Arel::Table.new(:rows)
12
+ composed_cte = Arel::Nodes::As.new(cte_table, original_query.select(primary_key).except(:limit).arel)
13
+
14
+ query = cte_table.project(cte_table[:id])
15
+ .with(composed_cte)
16
+ .where(Arel.sql(sanitize_sql_for_conditions(["pg_try_advisory_lock(('x'||substr(md5(:table_name || \"#{cte_table.name}\".\"#{primary_key}\"::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
17
+
18
+ limit = original_query.arel.ast.limit
19
+ query.limit = limit.value if limit.present?
20
+
21
+ unscoped.where(arel_table[:id].in(query)).merge(original_query.only(:order))
22
+ end)
23
+
8
24
  scope :joins_advisory_locks, (lambda do
9
- joins(<<~SQL)
25
+ join_sql = <<~SQL
10
26
  LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
11
27
  AND pg_locks.objsubid = 1
12
- AND pg_locks.classid = ('x'||substr(md5(good_jobs.id::text), 1, 16))::bit(32)::int
13
- AND pg_locks.objid = (('x'||substr(md5(good_jobs.id::text), 1, 16))::bit(64) << 32)::bit(32)::int
28
+ AND pg_locks.classid = ('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(32)::int
29
+ AND pg_locks.objid = (('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(64) << 32)::bit(32)::int
14
30
  SQL
15
- end)
16
31
 
17
- scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
18
- scope :with_advisory_lock, (lambda do
19
- where(<<~SQL)
20
- pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
21
- SQL
32
+ joins(sanitize_sql_for_conditions([join_sql, { table_name: table_name }]))
22
33
  end)
23
34
 
24
- def self.first_advisory_locked_row(query)
25
- find_by_sql(<<~SQL)
26
- WITH rows AS (#{query.to_sql})
27
- SELECT rows.id
28
- FROM rows
29
- WHERE pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
30
- SQL
31
- end
32
- # private_class_method :first_advisory_locked_row
33
-
34
- # https://www.postgresql.org/docs/9.6/view-pg-locks.html
35
- # Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
36
- # A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
37
- # The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
38
- # Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
39
- # The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
40
- def self.advisory_lock_details
41
- connection.select("SELECT * FROM pg_locks WHERE locktype = 'advisory' AND objsubid = 1")
42
- end
43
-
44
- def advisory_lock
45
- self.class.connection.execute(self.class.sanitize_sql_for_conditions(["SELECT 1 as one WHERE pg_try_advisory_lock(('x'||substr(md5(?), 1, 16))::bit(64)::bigint)", id])).ntuples > 0
46
- end
35
+ scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
36
+ scope :advisory_locked, -> { joins_advisory_locks.where.not(pg_locks: { locktype: nil }) }
37
+ scope :owns_advisory_locked, -> { joins_advisory_locks.where('"pg_locks"."pid" = pg_backend_pid()') }
47
38
 
48
- def advisory_lock!
49
- result = advisory_lock
50
- result || raise(RecordAlreadyAdvisoryLockedError)
51
- end
39
+ attr_accessor :create_with_advisory_lock
40
+ after_create -> { advisory_lock }, if: :create_with_advisory_lock
41
+ end
52
42
 
53
- def with_advisory_lock
43
+ class_methods do
44
+ def with_advisory_lock(&block)
45
+ records = advisory_lock.to_a
54
46
  begin
55
- advisory_lock!
56
- yield
57
- rescue StandardError => e
58
- advisory_unlock unless e.is_a? RecordAlreadyAdvisoryLockedError
59
- raise
47
+ block.call(records)
48
+ ensure
49
+ records.each(&:advisory_unlock)
60
50
  end
61
51
  end
52
+ end
62
53
 
63
- def advisory_locked?
64
- self.class.connection.execute(<<~SQL).ntuples > 0
65
- SELECT 1 as one
66
- FROM pg_locks
67
- WHERE
68
- locktype = 'advisory'
69
- AND objsubid = 1
70
- AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
71
- AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
72
- SQL
73
- end
54
+ def advisory_lock
55
+ query = <<~SQL
56
+ SELECT 1 AS one
57
+ WHERE pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
58
+ SQL
59
+ self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
60
+ end
74
61
 
75
- def owns_advisory_lock?
76
- self.class.connection.execute(<<~SQL).ntuples > 0
77
- SELECT 1 as one
78
- FROM pg_locks
79
- WHERE
80
- locktype = 'advisory'
81
- AND objsubid = 1
82
- AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
83
- AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
84
- AND pid = pg_backend_pid()
85
- SQL
86
- end
62
+ def advisory_unlock
63
+ query = <<~SQL
64
+ SELECT 1 AS one
65
+ WHERE pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
66
+ SQL
67
+ self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
68
+ end
87
69
 
88
- def advisory_unlock
89
- self.class.connection.execute("SELECT pg_advisory_unlock(('x'||substr(md5('#{id}'), 1, 16))::bit(64)::bigint)").first["pg_advisory_unlock"]
90
- end
70
+ def advisory_lock!
71
+ result = advisory_lock
72
+ result || raise(RecordAlreadyAdvisoryLockedError)
73
+ end
91
74
 
92
- def advisory_unlock!
93
- advisory_unlock while advisory_locked?
94
- end
75
+ def with_advisory_lock
76
+ advisory_lock!
77
+ yield
78
+ ensure
79
+ advisory_unlock unless $ERROR_INFO.is_a? RecordAlreadyAdvisoryLockedError
80
+ end
81
+
82
+ def advisory_locked?
83
+ self.class.advisory_locked.where(id: send(self.class.primary_key)).any?
84
+ end
85
+
86
+ def owns_advisory_lock?
87
+ self.class.owns_advisory_locked.where(id: send(self.class.primary_key)).any?
88
+ end
89
+
90
+ def advisory_unlock!
91
+ advisory_unlock while advisory_locked?
92
+ end
93
+
94
+ private
95
+
96
+ def sanitize_sql_for_conditions(*args)
97
+ # Made public in Rails 5.2
98
+ self.class.send(:sanitize_sql_for_conditions, *args)
95
99
  end
96
100
  end
97
101
  end
@@ -1,58 +1,68 @@
1
- module GoodJob::Logging
2
- extend ActiveSupport::Concern
1
+ module GoodJob
2
+ module Logging
3
+ extend ActiveSupport::Concern
3
4
 
4
- included do
5
- cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
5
+ included do
6
+ cattr_accessor :logger, default: ActiveSupport::TaggedLogging.new(ActiveSupport::Logger.new(STDOUT))
6
7
 
7
- def self.tag_logger(*tags)
8
- if logger.respond_to?(:tagged)
9
- tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
10
- logger.tagged(*tags) { yield }
11
- else
12
- yield
8
+ def self.tag_logger(*tags)
9
+ if logger.respond_to?(:tagged)
10
+ tags.unshift "GoodJob" unless logger.formatter.current_tags.include?("GoodJob")
11
+ logger.tagged(*tags) { yield }
12
+ else
13
+ yield
14
+ end
13
15
  end
14
16
  end
15
- end
16
17
 
17
- class LogSubscriber < ActiveSupport::LogSubscriber
18
- def create(event)
19
- good_job = event.payload[:good_job]
18
+ class LogSubscriber < ActiveSupport::LogSubscriber
19
+ def create(event)
20
+ good_job = event.payload[:good_job]
20
21
 
21
- info do
22
- "Created GoodJob resource with id #{good_job.id}"
22
+ info do
23
+ "Created GoodJob resource with id #{good_job.id}"
24
+ end
23
25
  end
24
- end
25
26
 
26
- def timer_task_finished(event)
27
- result = event.payload[:result]
28
- exception = event.payload[:error]
27
+ def timer_task_finished(event)
28
+ exception = event.payload[:error]
29
+ return unless exception
29
30
 
30
- if exception
31
31
  error do
32
32
  "ERROR: #{exception}\n #{exception.backtrace}"
33
33
  end
34
34
  end
35
- end
36
35
 
37
- def job_finished(event)
38
- result = event.payload[:result]
39
- exception = event.payload[:error]
36
+ def job_finished(event)
37
+ exception = event.payload[:error]
38
+ return unless exception
40
39
 
41
- if exception
42
40
  error do
43
41
  "ERROR: #{exception}\n #{exception.backtrace}"
44
42
  end
45
43
  end
46
- end
47
44
 
48
- private
45
+ def scheduler_start_shutdown(_event)
46
+ info do
47
+ "Shutting down scheduler..."
48
+ end
49
+ end
49
50
 
50
- def logger
51
- GoodJob.logger
52
- end
51
+ def scheduler_shutdown(_event)
52
+ info do
53
+ "Scheduler is shut down."
54
+ end
55
+ end
56
+
57
+ private
53
58
 
54
- def thread_name
55
- Thread.current.name || Thread.current.object_id
59
+ def logger
60
+ GoodJob.logger
61
+ end
62
+
63
+ def thread_name
64
+ Thread.current.name || Thread.current.object_id
65
+ end
56
66
  end
57
67
  end
58
68
  end
@@ -0,0 +1,21 @@
1
+ module GoodJob
2
+ class PgLocks < ActiveRecord::Base
3
+ self.table_name = 'pg_locks'.freeze
4
+
5
+ # https://www.postgresql.org/docs/9.6/view-pg-locks.html
6
+ # Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
7
+ # A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
8
+ # The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
9
+ # Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
10
+ # The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
11
+ def self.advisory_lock_details
12
+ connection.select <<~SQL
13
+ SELECT *
14
+ FROM pg_locks
15
+ WHERE
16
+ locktype = 'advisory' AND
17
+ objsubid = 1
18
+ SQL
19
+ end
20
+ end
21
+ end
@@ -1,36 +1,34 @@
1
- require "concurrent/scheduled_task"
2
1
  require "concurrent/executor/thread_pool_executor"
2
+ require "concurrent/timer_task"
3
3
  require "concurrent/utility/processor_counter"
4
4
 
5
5
  module GoodJob
6
6
  class Scheduler
7
- MAX_THREADS = Concurrent.processor_count
8
-
9
7
  DEFAULT_TIMER_OPTIONS = {
10
8
  execution_interval: 1,
11
9
  timeout_interval: 1,
12
- run_now: true
10
+ run_now: true,
13
11
  }.freeze
14
12
 
15
13
  DEFAULT_POOL_OPTIONS = {
16
14
  name: 'good_job',
17
15
  min_threads: 0,
18
- max_threads: MAX_THREADS,
16
+ max_threads: Concurrent.processor_count,
19
17
  auto_terminate: true,
20
18
  idletime: 0,
21
19
  max_queue: 0,
22
- fallback_policy: :abort # shouldn't matter -- 0 max queue
20
+ fallback_policy: :abort, # shouldn't matter -- 0 max queue
23
21
  }.freeze
24
22
 
25
- def initialize(query = GoodJob::Job.all, **options)
23
+ def initialize(query = GoodJob::Job.all, timer_options: {}, pool_options: {})
26
24
  @query = query
27
25
 
28
- @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS)
29
- @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS) do
26
+ @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
27
+ @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
30
28
  idle_threads = @pool.max_length - @pool.length
31
29
  create_thread if idle_threads.positive?
32
30
  end
33
- @timer.add_observer(TimerObserver.new)
31
+ @timer.add_observer(self, :timer_observer)
34
32
  @timer.execute
35
33
  end
36
34
 
@@ -42,44 +40,52 @@ module GoodJob
42
40
  end
43
41
 
44
42
  def shutdown(wait: true)
45
- if @timer.running?
46
- @timer.shutdown
47
- @timer.wait_for_termination if wait
48
- end
43
+ @_shutdown = true
44
+
45
+ ActiveSupport::Notifications.instrument("scheduler_start_shutdown.good_job", { wait: wait })
46
+ ActiveSupport::Notifications.instrument("scheduler_shutdown.good_job", { wait: wait }) do
47
+ if @timer.running?
48
+ @timer.shutdown
49
+ @timer.wait_for_termination if wait
50
+ end
49
51
 
50
- if @pool.running?
51
- @pool.shutdown
52
- @pool.wait_for_termination if wait
52
+ if @pool.running?
53
+ @pool.shutdown
54
+ @pool.wait_for_termination if wait
55
+ end
53
56
  end
54
57
  end
55
58
 
59
+ def shutdown?
60
+ @_shutdown
61
+ end
62
+
56
63
  def create_thread
57
64
  future = Concurrent::Future.new(args: [ordered_query], executor: @pool) do |query|
58
- Rails.application.executor.wrap do
59
- while good_job = query.with_advisory_lock.first
60
- ActiveSupport::Notifications.instrument("job_started.good_job", { good_job: good_job })
65
+ good_job = nil
61
66
 
62
- JobWrapper.new(good_job).perform
67
+ Rails.application.executor.wrap do
68
+ query.limit(1).with_advisory_lock do |good_jobs|
69
+ good_job = good_jobs.first
70
+ break unless good_job
63
71
 
64
- good_job.advisory_unlock
72
+ good_job.perform
65
73
  end
66
74
  end
67
- true
75
+
76
+ good_job
68
77
  end
69
- future.add_observer(TaskObserver.new)
78
+ future.add_observer(self, :task_observer)
70
79
  future.execute
71
80
  end
72
81
 
73
- class TimerObserver
74
- def update(time, result, error)
75
- ActiveSupport::Notifications.instrument("timer_task_finished.good_job", { result: result, error: error, time: time })
76
- end
82
+ def timer_observer(time, executed_task, error)
83
+ ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
77
84
  end
78
85
 
79
- class TaskObserver
80
- def update(time, result, error)
81
- ActiveSupport::Notifications.instrument("job_finished.good_job", { result: result, error: error, time: time })
82
- end
86
+ def task_observer(time, performed_job, error)
87
+ ActiveSupport::Notifications.instrument("finished_job_task.good_job", { good_job: performed_job, error: error, time: time })
88
+ create_thread if performed_job
83
89
  end
84
90
  end
85
91
  end
@@ -1,3 +1,3 @@
1
1
  module GoodJob
2
- VERSION = '0.2.0'
2
+ VERSION = '0.5.0'.freeze
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: good_job
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Ben Sheldon
8
8
  autorequire:
9
- bindir: bin
9
+ bindir: exe
10
10
  cert_chain: []
11
- date: 2020-03-06 00:00:00.000000000 Z
11
+ date: 2020-07-13 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: concurrent-ruby
@@ -16,37 +16,65 @@ dependencies:
16
16
  requirements:
17
17
  - - ">="
18
18
  - !ruby/object:Gem::Version
19
- version: '0'
19
+ version: 1.0.2
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
- version: '0'
26
+ version: 1.0.2
27
+ - !ruby/object:Gem::Dependency
28
+ name: pg
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - ">="
32
+ - !ruby/object:Gem::Version
33
+ version: 1.0.0
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - ">="
39
+ - !ruby/object:Gem::Version
40
+ version: 1.0.0
27
41
  - !ruby/object:Gem::Dependency
28
42
  name: rails
29
43
  requirement: !ruby/object:Gem::Requirement
30
44
  requirements:
31
45
  - - ">="
32
46
  - !ruby/object:Gem::Version
33
- version: '0'
47
+ version: 5.1.0
34
48
  type: :runtime
35
49
  prerelease: false
36
50
  version_requirements: !ruby/object:Gem::Requirement
37
51
  requirements:
38
52
  - - ">="
39
53
  - !ruby/object:Gem::Version
40
- version: '0'
54
+ version: 5.1.0
41
55
  - !ruby/object:Gem::Dependency
42
56
  name: thor
43
57
  requirement: !ruby/object:Gem::Requirement
44
58
  requirements:
45
59
  - - ">="
46
60
  - !ruby/object:Gem::Version
47
- version: '0'
61
+ version: 0.14.1
48
62
  type: :runtime
49
63
  prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - ">="
67
+ - !ruby/object:Gem::Version
68
+ version: 0.14.1
69
+ - !ruby/object:Gem::Dependency
70
+ name: appraisal
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - ">="
74
+ - !ruby/object:Gem::Version
75
+ version: '0'
76
+ type: :development
77
+ prerelease: false
50
78
  version_requirements: !ruby/object:Gem::Requirement
51
79
  requirements:
52
80
  - - ">="
@@ -66,6 +94,20 @@ dependencies:
66
94
  - - ">="
67
95
  - !ruby/object:Gem::Version
68
96
  version: '0'
97
+ - !ruby/object:Gem::Dependency
98
+ name: foreman
99
+ requirement: !ruby/object:Gem::Requirement
100
+ requirements:
101
+ - - ">="
102
+ - !ruby/object:Gem::Version
103
+ version: '0'
104
+ type: :development
105
+ prerelease: false
106
+ version_requirements: !ruby/object:Gem::Requirement
107
+ requirements:
108
+ - - ">="
109
+ - !ruby/object:Gem::Version
110
+ version: '0'
69
111
  - !ruby/object:Gem::Dependency
70
112
  name: gem-release
71
113
  requirement: !ruby/object:Gem::Requirement
@@ -95,7 +137,7 @@ dependencies:
95
137
  - !ruby/object:Gem::Version
96
138
  version: '0'
97
139
  - !ruby/object:Gem::Dependency
98
- name: pg
140
+ name: pry
99
141
  requirement: !ruby/object:Gem::Requirement
100
142
  requirements:
101
143
  - - ">="
@@ -122,49 +164,93 @@ dependencies:
122
164
  - - ">="
123
165
  - !ruby/object:Gem::Version
124
166
  version: '0'
125
- description: GoodJob is a minimal postgres based job queue system for Rails
167
+ - !ruby/object:Gem::Dependency
168
+ name: rubocop
169
+ requirement: !ruby/object:Gem::Requirement
170
+ requirements:
171
+ - - ">="
172
+ - !ruby/object:Gem::Version
173
+ version: '0'
174
+ type: :development
175
+ prerelease: false
176
+ version_requirements: !ruby/object:Gem::Requirement
177
+ requirements:
178
+ - - ">="
179
+ - !ruby/object:Gem::Version
180
+ version: '0'
181
+ - !ruby/object:Gem::Dependency
182
+ name: rubocop-rspec
183
+ requirement: !ruby/object:Gem::Requirement
184
+ requirements:
185
+ - - ">="
186
+ - !ruby/object:Gem::Version
187
+ version: '0'
188
+ type: :development
189
+ prerelease: false
190
+ version_requirements: !ruby/object:Gem::Requirement
191
+ requirements:
192
+ - - ">="
193
+ - !ruby/object:Gem::Version
194
+ version: '0'
195
+ description: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
126
196
  email:
127
197
  - bensheldon@gmail.com
128
198
  executables:
129
199
  - good_job
130
200
  extensions: []
131
- extra_rdoc_files: []
201
+ extra_rdoc_files:
202
+ - README.md
203
+ - CHANGELOG.md
204
+ - LICENSE.txt
132
205
  files:
206
+ - CHANGELOG.md
133
207
  - LICENSE.txt
134
208
  - README.md
135
- - bin/good_job
209
+ - exe/good_job
210
+ - lib/active_job/queue_adapters/good_job_adapter.rb
136
211
  - lib/good_job.rb
137
212
  - lib/good_job/adapter.rb
138
213
  - lib/good_job/cli.rb
139
- - lib/good_job/inline_scheduler.rb
140
214
  - lib/good_job/job.rb
141
- - lib/good_job/job_wrapper.rb
142
215
  - lib/good_job/lockable.rb
143
216
  - lib/good_job/logging.rb
217
+ - lib/good_job/pg_locks.rb
144
218
  - lib/good_job/railtie.rb
145
219
  - lib/good_job/scheduler.rb
146
220
  - lib/good_job/version.rb
147
221
  homepage: https://github.com/benheldon/good_job
148
222
  licenses:
149
223
  - MIT
150
- metadata: {}
224
+ metadata:
225
+ bug_tracker_uri: https://github.com/bensheldon/good_job/issues
226
+ changelog_uri: https://github.com/bensheldon/good_job/blob/master/CHANGELOG.md
227
+ documentation_uri: https://rdoc.info/github/bensheldon/good_job
228
+ homepage_uri: https://github.com/benheldon/good_job
229
+ source_code_uri: https://github.com/bensheldon/good_job
151
230
  post_install_message:
152
- rdoc_options: []
231
+ rdoc_options:
232
+ - "--title"
233
+ - GoodJob - a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
234
+ - "--main"
235
+ - README.md
236
+ - "--line-numbers"
237
+ - "--inline-source"
238
+ - "--quiet"
153
239
  require_paths:
154
240
  - lib
155
241
  required_ruby_version: !ruby/object:Gem::Requirement
156
242
  requirements:
157
243
  - - ">="
158
244
  - !ruby/object:Gem::Version
159
- version: '0'
245
+ version: 2.4.0
160
246
  required_rubygems_version: !ruby/object:Gem::Requirement
161
247
  requirements:
162
248
  - - ">="
163
249
  - !ruby/object:Gem::Version
164
250
  version: '0'
165
251
  requirements: []
166
- rubygems_version: 3.1.2
252
+ rubygems_version: 3.0.3
167
253
  signing_key:
168
254
  specification_version: 4
169
- summary: GoodJob is a minimal postgres based job queue system for Rails
255
+ summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails
170
256
  test_files: []
@@ -1,10 +0,0 @@
1
- module GoodJob
2
- class InlineScheduler
3
- def enqueue(good_job)
4
- JobWrapper.new(good_job).perform
5
- end
6
-
7
- def shutdown(wait: true)
8
- end
9
- end
10
- end
@@ -1,16 +0,0 @@
1
- module GoodJob
2
- class JobWrapper
3
- def initialize(good_job)
4
- @good_job = good_job
5
- end
6
-
7
- def perform
8
- serialized_params = @good_job.serialized_params.merge(
9
- "provider_job_id" => @good_job.id
10
- )
11
- ActiveJob::Base.execute(serialized_params)
12
-
13
- @good_job.destroy!
14
- end
15
- end
16
- end