good_job 0.2.2 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a808beca38372ee8dcd23871600c9ed9b0181aa7756656c46e0382e9eef927b0
-  data.tar.gz: 7fa59a1e31b4e720149ed19009f91691e840096570d38e519d37eb9b930cb655
+  metadata.gz: 4f7dda29494df3fc05a199d6bf0efbc825a47653c3af507a0d1ceb92456d4307
+  data.tar.gz: 881ad046bab6b17c5d532d035fddef54420639288a0a4bd8f4f07b980c1fa239
 SHA512:
-  metadata.gz: 9ce51a00e0aeabbd2be39e796168343d0daf96b368cb75ae8011f05bec891fcab820323189c43c79b97ae5a4c2586a7ec4996cd68717a83bb66613b7db523a95
-  data.tar.gz: d0cb6bc52ba43a1e86a4ba12a8a09204d90f3b393de0d0dc54de64de37c85290f8228362e76f8c023a0d78f28d63706b4c8f2b04626134bc4240d9ad055544db
+  metadata.gz: c0cfc3c4d61666844a6bd7c532670e0b1be40fd6f805eff63919d3600d06933b8029f0febbc81efbf52b1f3d459317caeb522affe20dde9d3f4d7672622fd708
+  data.tar.gz: c4c9a9fddb2108769e3672baeaeaca2f7c966ee708f95867ed57970e688debcf1ce41a3eacd65d88c0192700f1b293cd4a9ad63fee9e4af93d44934b0876120f

data/CHANGELOG.md

@@ -1,8 +1,67 @@
 # Changelog
 
-## [0.2.2](https://github.com/bensheldon/good_job/tree/0.2.2) (2020-03-07)
+## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-15)
 
-[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.1...0.2.2)
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.7.0)
+
+**Closed issues:**
+
+- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
+
+**Merged pull requests:**
+
+- Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
+- Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
+- Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.5.0...v0.6.0)
+
+**Closed issues:**
+
+- Improve the command line options [\#32](https://github.com/bensheldon/good_job/issues/32)
+- Allow config.active\_job.queue\_adapter = :good\_job to work [\#5](https://github.com/bensheldon/good_job/issues/5)
+
+**Merged pull requests:**
+
+- Improve generation of changelog [\#36](https://github.com/bensheldon/good_job/pull/36) ([bensheldon](https://github.com/bensheldon))
+- Update Github Action Workflow for Backlog Project Board [\#35](https://github.com/bensheldon/good_job/pull/35) ([bensheldon](https://github.com/bensheldon))
+- Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
+- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
+- Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.4.0...v0.5.0)
+
+**Merged pull requests:**
+
+- Update development Ruby to 2.6.6 and gems [\#29](https://github.com/bensheldon/good_job/pull/29) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.4.0](https://github.com/bensheldon/good_job/tree/v0.4.0) (2020-03-31)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.3.0...v0.4.0)
+
+**Merged pull requests:**
+
+- Improve ActiveRecord usage for advisory locking [\#24](https://github.com/bensheldon/good_job/pull/24) ([bensheldon](https://github.com/bensheldon))
+- Remove support for Rails 5.1 [\#23](https://github.com/bensheldon/good_job/pull/23) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.3.0](https://github.com/bensheldon/good_job/tree/v0.3.0) (2020-03-22)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.2...v0.3.0)
+
+**Merged pull requests:**
+
+- Update development Ruby to 2.6.5 [\#22](https://github.com/bensheldon/good_job/pull/22) ([bensheldon](https://github.com/bensheldon))
+- Simplify the internal API, removing JobWrapper and InlineScheduler [\#21](https://github.com/bensheldon/good_job/pull/21) ([bensheldon](https://github.com/bensheldon))
+- Generate a new future for every executed job [\#20](https://github.com/bensheldon/good_job/pull/20) ([bensheldon](https://github.com/bensheldon))
+- Configuration for maximum number of job execution threads [\#18](https://github.com/bensheldon/good_job/pull/18) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.2.2](https://github.com/bensheldon/good_job/tree/v0.2.2) (2020-03-08)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.2.1...v0.2.2)
 
 **Merged pull requests:**
 

data/README.md

@@ -7,19 +7,27 @@ Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Q
 - Stand on the shoulders of ActiveJob. For example, [exception](https://edgeguides.rubyonrails.org/active_job_basics.html#exceptions) and [retry](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs) behavior. 
 - Stand on the shoulders of Ruby on Rails. For example, ActiveRecord ORM, connection pools, and [multithreaded support](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent-Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
 - Stand on the shoulders of Postgres. For example, Advisory Locks.
-- Convention over simplicity over performance. 
+- Convention over simplicity over performance.
+
+GoodJob supports all ActiveJob functionality:
+- Async. GoodJob has the ability to run the job in a non-blocking manner.
+- Queues. Jobs may set which queue they are run in with queue_as or by using the set method.
+- Delayed. GoodJob will run the job in the future through perform_later.
+- Priorities. The order in which jobs are processed can be configured differently.
+- Timeouts. GoodJob defers to ActiveJob where it can be implemented as an `around` hook. See [Taking advantage of ActiveJob](#taking-advantage-of-activejob).
+- Retries. GoodJob will automatically retry uncompleted jobs immediately. See [Taking advantage of ActiveJob](#taking-advantage-of-activejob).
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
 ```ruby
-gem 'good_job', github: 'bensheldon/good_job'
+gem 'good_job'
 ```
 
 And then execute:
 ```bash
-$ bundle
+$ bundle install
 ```
 
 ## Usage
@@ -43,6 +51,9 @@ $ bundle
           t.integer :priority
           t.jsonb :serialized_params
           t.timestamp :scheduled_at
+   
+          t.index :scheduled_at
+          t.index [:queue_name, :scheduled_at]
         end
       end
     end
@@ -56,17 +67,75 @@ $ bundle
     
 1. Configure the ActiveJob adapter:
     ```ruby
+    # config/application.rb
+    config.active_job.queue_adapter = :good_job
+    ```
+    
+    By default, using `:good_job` is equivalent to manually configuring the adapter:
+    
+    ```ruby
+    # config/environments/development.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+   
+    # config/environments/test.rb
+    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+   
     # config/environments/production.rb
     config.active_job.queue_adapter = GoodJob::Adapter.new
+    ```
 
-    # config/environments/development.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+1. Queue your job 🎉: 
+    ```ruby
+    YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
     ```bash
     $ bundle exec good_job
     ```
+   
+   Configuration options available with `help`:
+   ```bash
+   $ bundle exec good_job help start
+   
+   # Usage:
+   #  good_job start
+   #
+   # Options:
+   #   [--max-threads=N]         # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
+   #   [--queues=queue1,queue2]  # Queues to work from. Separate multiple queues with commas (default: *)
+   #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
+   ```
+
+### Taking advantage of ActiveJob
+
+ActiveJob has a rich set of built-in functionality for timeouts, error handling, and retrying. For example:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  # Retry errors an infinite number of times with exponential back-off
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+  # Timeout jobs after 10 minutes
+  JobTimeoutError = Class.new(StandardError)
+  around_perform do |_job, block|
+    Timeout.timeout(10.minutes, JobTimeoutError) do
+      block.call
+    end
+  end
+end
+```
+   
+### Configuring Job Execution Threads
+    
+GoodJob executes enqueued jobs using threads. There is a lot than can be said about [multithreaded behavior in Ruby on Rails](https://guides.rubyonrails.org/threading_and_code_execution.html), but briefly:
+
+- Each GoodJob execution thread requires its own database connection, which are automatically checked out from Rails’s connection pool. _Allowing GoodJob to schedule more threads than are available in the database connection pool can lead to timeouts and is not recommended._ 
+- The maximum number of GoodJob threads can be configured, in decreasing precedence:
+    1. `$ bundle exec good_job --max_threads 4`
+    2. `$ GOOD_JOB_MAX_THREADS=4 bundle exec good_job`
+    3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
+    4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
 ## Development
 
@@ -83,6 +152,17 @@ $ bin/setup_test
 $ bin/rspec
 ```
 
+This gem uses Appraisal to run tests against multiple versions of Rails:
+
+```bash
+# Install Appraisal(s) gemfiles
+$ bundle exec appraisal
+
+# Run tests
+$ bundle exec appraisal bin/rspec
+
+```
+
 For developing locally within another Ruby on Rails project:
 
 ```bash
@@ -103,17 +183,10 @@ Package maintainers can release this gem with the following [gem-release](https:
 # Sign into rubygems
 $ gem signin
 
-# Increase the version number
-$ gem bump -v minor --no-commit
-
-# Update the changelog
-$ bundle exec rake changelog
-
-# Commit the version and changelog to git
-$ bundle exec rake commit_version
+# Update version number, changelog, and create git commit:
+$ bundle exec rake commit_version[minor] # major,minor,patch
 
-# Push to rubygems.org
-$ gem release
+# ..and follow subsequent directions. 
 ```
 
 ## Contributing

data/lib/active_job/queue_adapters/good_job_adapter.rb

@@ -0,0 +1,13 @@
+module ActiveJob
+  module QueueAdapters
+    class GoodJobAdapter < GoodJob::Adapter
+      def initialize
+        if Rails.env.development? || Rails.env.test?
+          super(inline: true)
+        else
+          super(inline: false)
+        end
+      end
+    end
+  end
+end

data/lib/good_job.rb

@@ -4,10 +4,11 @@ require 'good_job/railtie'
 require 'good_job/logging'
 require 'good_job/lockable'
 require 'good_job/job'
-require 'good_job/inline_scheduler'
 require "good_job/scheduler"
-require "good_job/job_wrapper"
 require 'good_job/adapter'
+require 'good_job/pg_locks'
+
+require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
   include Logging

data/lib/good_job/adapter.rb

@@ -1,41 +1,37 @@
 module GoodJob
   class Adapter
-    def initialize(options = {})
-      @options = options
-      @scheduler = InlineScheduler.new if inline?
+    def initialize(inline: false)
+      @inline = inline
     end
 
-    def enqueue(job)
-      enqueue_at(job, nil)
+    def enqueue(active_job)
+      enqueue_at(active_job, nil)
     end
 
-    def enqueue_at(job, timestamp)
-      params = {
-        queue_name: job.queue_name,
-        priority: job.priority,
-        serialized_params: job.serialize,
-      }
-      params[:scheduled_at] = Time.at(timestamp) if timestamp
-
-      good_job = GoodJob::Job.create(params)
-      job.provider_job_id = good_job.id
-
-      GoodJob.tag_logger do
-        ActiveSupport::Notifications.instrument("create.good_job", { good_job: good_job, job: job })
-        @scheduler.enqueue(good_job) if inline?
+    def enqueue_at(active_job, timestamp)
+      good_job = GoodJob::Job.enqueue(
+        active_job,
+        scheduled_at: timestamp ? Time.zone.at(timestamp) : nil,
+        create_with_advisory_lock: inline?
+      )
+
+      if inline?
+        begin
+          good_job.perform
+        ensure
+          good_job.advisory_unlock
+        end
       end
 
       good_job
     end
 
-    def shutdown(wait: true)
-      @scheduler&.shutdown(wait: wait)
+    def shutdown(wait: true) # rubocop:disable Lint/UnusedMethodArgument
+      nil
     end
 
-    private
-
     def inline?
-      @options.fetch(:inline, false)
+      @inline
     end
   end
 end

data/lib/good_job/cli.rb

@@ -4,18 +4,59 @@ module GoodJob
   class CLI < Thor
     RAILS_ENVIRONMENT_RB = File.expand_path("config/environment.rb")
 
-    desc :start, "Start jobs"
+    desc :start, "Start job worker"
+    method_option :max_threads,
+                  type: :numeric,
+                  desc: "Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)"
+    method_option :queues,
+                  type: :string,
+                  banner: "queue1,queue2",
+                  desc: "Queues to work from. Separate multiple queues with commas (default: *)"
+    method_option :poll_interval,
+                  type: :numeric,
+                  desc: "Interval between polls for available jobs in seconds (default: 1)"
     def start
       require RAILS_ENVIRONMENT_RB
 
-      scheduler = GoodJob::Scheduler.new
+      max_threads = (
+        options[:max_threads] ||
+        ENV['GOOD_JOB_MAX_THREADS'] ||
+        ENV['RAILS_MAX_THREADS'] ||
+        ActiveRecord::Base.connection_pool.size
+      ).to_i
 
+      queue_names = (
+        options[:queues] ||
+        ENV['GOOD_JOB_QUEUES'] ||
+        '*'
+      ).split(',').map(&:strip)
+
+      poll_interval = (
+        options[:poll_interval] ||
+        ENV['GOOD_JOB_POLL_INTERVAL']
+      ).to_i
+
+      job_query = GoodJob::Job.all
+      queue_names_without_all = queue_names.reject { |q| q == '*' }
+      job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
+
+      job_performer = job_query.only_scheduled.priority_ordered.to_performer
+
+      $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
+
+      timer_options = {}
+      timer_options[:execution_interval] = poll_interval if poll_interval.positive?
+
+      pool_options = {
+        max_threads: max_threads,
+      }
+
+      scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
+
+      @stop_good_job_executable = false
       %w[INT TERM].each do |signal|
         trap(signal) { @stop_good_job_executable = true }
       end
-      @stop_good_job_executable = false
-
-      $stdout.puts "GoodJob waiting for jobs..."
 
       Kernel.loop do
         sleep 0.1

data/lib/good_job/job.rb

@@ -1,6 +1,65 @@
 module GoodJob
   class Job < ActiveRecord::Base
     include Lockable
-    self.table_name = 'good_jobs'
+
+    DEFAULT_QUEUE_NAME = 'default'.freeze
+    DEFAULT_PRIORITY = 0
+
+    self.table_name = 'good_jobs'.freeze
+
+    scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
+    scope :priority_ordered, -> { order(priority: :desc) }
+    scope :to_performer, -> { Performer.new(self) }
+
+    class Performer
+      def initialize(query)
+        @query = query
+      end
+
+      def next
+        good_job = nil
+
+        @query.limit(1).with_advisory_lock do |good_jobs|
+          good_job = good_jobs.first
+          break unless good_job
+
+          good_job.perform
+        end
+
+        good_job
+      end
+    end
+
+    def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
+      good_job = nil
+      ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
+        good_job = GoodJob::Job.new(
+          queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
+          priority: active_job.priority || DEFAULT_PRIORITY,
+          serialized_params: active_job.serialize,
+          scheduled_at: scheduled_at || Time.current,
+          create_with_advisory_lock: create_with_advisory_lock
+        )
+
+        instrument_payload[:good_job] = good_job
+
+        good_job.save!
+        active_job.provider_job_id = good_job.id
+      end
+
+      good_job
+    end
+
+    def perform
+      ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
+      ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
+        params = serialized_params.merge(
+          "provider_job_id" => id
+        )
+        ActiveJob::Base.execute(params)
+
+        destroy!
+      end
+    end
   end
 end

data/lib/good_job/lockable.rb

@@ -5,98 +5,102 @@ module GoodJob
     RecordAlreadyAdvisoryLockedError = Class.new(StandardError)
 
     included do
+      scope :advisory_lock, (lambda do
+        original_query = self
+
+        cte_table = Arel::Table.new(:rows)
+        composed_cte = Arel::Nodes::As.new(cte_table, original_query.select(primary_key).except(:limit).arel)
+
+        query = cte_table.project(cte_table[:id])
+                  .with(composed_cte)
+                  .where(Arel.sql(sanitize_sql_for_conditions(["pg_try_advisory_lock(('x'||substr(md5(:table_name || \"#{cte_table.name}\".\"#{primary_key}\"::text), 1, 16))::bit(64)::bigint)", { table_name: table_name }])))
+
+        limit = original_query.arel.ast.limit
+        query.limit = limit.value if limit.present?
+
+        unscoped.where(arel_table[:id].in(query)).merge(original_query.only(:order))
+      end)
+
       scope :joins_advisory_locks, (lambda do
-        joins(<<~SQL)
+        join_sql = <<~SQL
           LEFT JOIN pg_locks ON pg_locks.locktype = 'advisory'
             AND pg_locks.objsubid = 1
-            AND pg_locks.classid = ('x'||substr(md5(good_jobs.id::text), 1, 16))::bit(32)::int
-            AND pg_locks.objid = (('x'||substr(md5(good_jobs.id::text), 1, 16))::bit(64) << 32)::bit(32)::int
+            AND pg_locks.classid = ('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(32)::int
+            AND pg_locks.objid = (('x'||substr(md5(:table_name || "#{table_name}"."#{primary_key}"::text), 1, 16))::bit(64) << 32)::bit(32)::int
         SQL
-      end)
 
-      scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
-      scope :with_advisory_lock, (lambda do
-        where(<<~SQL)
-          pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
-        SQL
+        joins(sanitize_sql_for_conditions([join_sql, { table_name: table_name }]))
       end)
 
-      def self.first_advisory_locked_row(query)
-        find_by_sql(<<~SQL)
-          WITH rows AS (#{query.to_sql})
-          SELECT rows.id
-          FROM rows
-          WHERE pg_try_advisory_lock(('x'||substr(md5(id::text), 1, 16))::bit(64)::bigint)
-        SQL
-      end
-      # private_class_method :first_advisory_locked_row
-
-      # https://www.postgresql.org/docs/9.6/view-pg-locks.html
-      # Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
-      # A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
-      # The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
-      # Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
-      # The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
-      def self.advisory_lock_details
-        connection.select("SELECT * FROM pg_locks WHERE locktype = 'advisory' AND objsubid = 1")
-      end
+      scope :advisory_unlocked, -> { joins_advisory_locks.where(pg_locks: { locktype: nil }) }
+      scope :advisory_locked, -> { joins_advisory_locks.where.not(pg_locks: { locktype: nil }) }
+      scope :owns_advisory_locked, -> { joins_advisory_locks.where('"pg_locks"."pid" = pg_backend_pid()') }
 
-      def advisory_lock
-        self.class.connection.execute(sanitize_sql_for_conditions(["SELECT 1 as one WHERE pg_try_advisory_lock(('x'||substr(md5(?), 1, 16))::bit(64)::bigint)", id])).ntuples.positive?
-      end
+      attr_accessor :create_with_advisory_lock
 
-      def advisory_lock!
-        result = advisory_lock
-        result || raise(RecordAlreadyAdvisoryLockedError)
-      end
+      after_create -> { advisory_lock }, if: :create_with_advisory_lock
+    end
 
+    class_methods do
       def with_advisory_lock
-        advisory_lock!
-        yield
-      rescue StandardError => e
-        advisory_unlock unless e.is_a? RecordAlreadyAdvisoryLockedError
-        raise
-      end
+        raise ArgumentError, "Must provide a block" unless block_given?
 
-      def advisory_locked?
-        self.class.connection.execute(<<~SQL).ntuples.positive?
-          SELECT 1 as one
-          FROM pg_locks
-          WHERE
-            locktype = 'advisory'
-            AND objsubid = 1
-            AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
-            AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
-        SQL
+        records = advisory_lock.to_a
+        begin
+          yield(records)
+        ensure
+          records.each(&:advisory_unlock)
+        end
       end
+    end
 
-      def owns_advisory_lock?
-        self.class.connection.execute(<<~SQL).ntuples.positive?
-          SELECT 1 as one
-          FROM pg_locks
-          WHERE
-            locktype = 'advisory'
-            AND objsubid = 1
-            AND classid = ('x'||substr(md5('#{id}'), 1, 16))::bit(32)::int
-            AND objid = (('x'||substr(md5('#{id}'), 1, 16))::bit(64) << 32)::bit(32)::int
-            AND pid = pg_backend_pid()
-        SQL
-      end
+    def advisory_lock
+      query = <<~SQL
+        SELECT 1 AS one
+        WHERE pg_try_advisory_lock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
+      SQL
+      self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
+    end
 
-      def advisory_unlock
-        self.class.connection.execute("SELECT pg_advisory_unlock(('x'||substr(md5('#{id}'), 1, 16))::bit(64)::bigint)").first["pg_advisory_unlock"]
-      end
+    def advisory_unlock
+      query = <<~SQL
+        SELECT 1 AS one
+        WHERE pg_advisory_unlock(('x'||substr(md5(:table_name || :id::text), 1, 16))::bit(64)::bigint)
+      SQL
+      self.class.connection.execute(sanitize_sql_for_conditions([query, { table_name: self.class.table_name, id: send(self.class.primary_key) }])).ntuples.positive?
+    end
 
-      def advisory_unlock!
-        advisory_unlock while advisory_locked?
-      end
+    def advisory_lock!
+      result = advisory_lock
+      result || raise(RecordAlreadyAdvisoryLockedError)
+    end
 
-      private
+    def with_advisory_lock
+      raise ArgumentError, "Must provide a block" unless block_given?
 
-      def sanitize_sql_for_conditions(*args)
-        # Made public in Rails 5.2
-        self.class.send(:sanitize_sql_for_conditions, *args)
-      end
+      advisory_lock!
+      yield
+    ensure
+      advisory_unlock unless $ERROR_INFO.is_a? RecordAlreadyAdvisoryLockedError
+    end
+
+    def advisory_locked?
+      self.class.advisory_locked.where(id: send(self.class.primary_key)).any?
+    end
+
+    def owns_advisory_lock?
+      self.class.owns_advisory_locked.where(id: send(self.class.primary_key)).any?
+    end
+
+    def advisory_unlock!
+      advisory_unlock while advisory_locked?
+    end
+
+    private
+
+    def sanitize_sql_for_conditions(*args)
+      # Made public in Rails 5.2
+      self.class.send(:sanitize_sql_for_conditions, *args)
     end
   end
 end

data/lib/good_job/pg_locks.rb

@@ -0,0 +1,21 @@
+module GoodJob
+  class PgLocks < ActiveRecord::Base
+    self.table_name = 'pg_locks'.freeze
+
+    # https://www.postgresql.org/docs/9.6/view-pg-locks.html
+    # Advisory locks can be acquired on keys consisting of either a single bigint value or two integer values.
+    # A bigint key is displayed with its high-order half in the classid column, its low-order half in the objid column, and objsubid equal to 1.
+    # The original bigint value can be reassembled with the expression (classid::bigint << 32) | objid::bigint.
+    # Integer keys are displayed with the first key in the classid column, the second key in the objid column, and objsubid equal to 2.
+    # The actual meaning of the keys is up to the user. Advisory locks are local to each database, so the database column is meaningful for an advisory lock.
+    def self.advisory_lock_details
+      connection.select <<~SQL
+        SELECT *
+        FROM pg_locks
+        WHERE
+          locktype = 'advisory' AND
+          objsubid = 1
+      SQL
+    end
+  end
+end

data/lib/good_job/scheduler.rb

@@ -1,11 +1,9 @@
-require "concurrent/scheduled_task"
 require "concurrent/executor/thread_pool_executor"
+require "concurrent/timer_task"
 require "concurrent/utility/processor_counter"
 
 module GoodJob
   class Scheduler
-    MAX_THREADS = Concurrent.processor_count
-
     DEFAULT_TIMER_OPTIONS = {
       execution_interval: 1,
       timeout_interval: 1,
@@ -15,29 +13,26 @@ module GoodJob
     DEFAULT_POOL_OPTIONS = {
       name: 'good_job',
       min_threads: 0,
-      max_threads: MAX_THREADS,
+      max_threads: Concurrent.processor_count,
       auto_terminate: true,
       idletime: 0,
       max_queue: 0,
       fallback_policy: :abort, # shouldn't matter -- 0 max queue
     }.freeze
 
-    def initialize(query = GoodJob::Job.all, **_options)
-      @query = query
+    def initialize(performer, timer_options: {}, pool_options: {})
+      raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 
-      @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS)
-      @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS) do
+      @performer = performer
+      @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
+      @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
         idle_threads = @pool.max_length - @pool.length
         create_thread if idle_threads.positive?
       end
-      @timer.add_observer(TimerObserver.new)
+      @timer.add_observer(self, :timer_observer)
       @timer.execute
     end
 
-    def ordered_query
-      @query.where("scheduled_at < ?", Time.current).or(@query.where(scheduled_at: nil)).order(priority: :desc)
-    end
-
     def execute
     end
 
@@ -63,36 +58,22 @@ module GoodJob
     end
 
     def create_thread
-      future = Concurrent::Future.new(args: [ordered_query], executor: @pool) do |query|
-        Rails.application.executor.wrap do
-          loop do
-            good_job = query.with_advisory_lock.first
-            break unless good_job
-
-            ActiveSupport::Notifications.instrument("job_started.good_job", { good_job: good_job })
-
-            JobWrapper.new(good_job).perform
-
-            good_job.advisory_unlock
-          end
-        end
-
-        true
+      future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
+        result = nil
+        Rails.application.executor.wrap { result = performer.next }
+        result
       end
-      future.add_observer(TaskObserver.new)
+      future.add_observer(self, :task_observer)
       future.execute
     end
 
-    class TimerObserver
-      def update(time, result, error)
-        ActiveSupport::Notifications.instrument("timer_task_finished.good_job", { result: result, error: error, time: time })
-      end
+    def timer_observer(time, executed_task, error)
+      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
     end
 
-    class TaskObserver
-      def update(time, result, error)
-        ActiveSupport::Notifications.instrument("job_finished.good_job", { result: result, error: error, time: time })
-      end
+    def task_observer(time, result, error)
+      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: result, error: error, time: time })
+      create_thread if result
     end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '0.2.2'.freeze
+  VERSION = '0.7.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.7.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-03-08 00:00:00.000000000 Z
+date: 2020-07-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -94,6 +94,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: foreman
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: gem-release
   requirement: !ruby/object:Gem::Requirement
@@ -122,6 +136,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec-rails
   requirement: !ruby/object:Gem::Requirement
@@ -150,6 +178,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop-performance
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rubocop-rspec
   requirement: !ruby/object:Gem::Requirement
@@ -179,14 +235,14 @@ files:
 - LICENSE.txt
 - README.md
 - exe/good_job
+- lib/active_job/queue_adapters/good_job_adapter.rb
 - lib/good_job.rb
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
-- lib/good_job/inline_scheduler.rb
 - lib/good_job/job.rb
-- lib/good_job/job_wrapper.rb
 - lib/good_job/lockable.rb
 - lib/good_job/logging.rb
+- lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb
 - lib/good_job/scheduler.rb
 - lib/good_job/version.rb
@@ -221,7 +277,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.2
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: A multithreaded, Postgres-based ActiveJob backend for Ruby on Rails

data/lib/good_job/inline_scheduler.rb

@@ -1,10 +0,0 @@
-module GoodJob
-  class InlineScheduler
-    def enqueue(good_job)
-      JobWrapper.new(good_job).perform
-    end
-
-    def shutdown(wait: true)
-    end
-  end
-end

data/lib/good_job/job_wrapper.rb

@@ -1,16 +0,0 @@
-module GoodJob
-  class JobWrapper
-    def initialize(good_job)
-      @good_job = good_job
-    end
-
-    def perform
-      serialized_params = @good_job.serialized_params.merge(
-        "provider_job_id" => @good_job.id
-      )
-      ActiveJob::Base.execute(serialized_params)
-
-      @good_job.destroy!
-    end
-  end
-end