good_job 0.6.0 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c20439024a46084c22e46bb42e38589687c2ab2cea47d90705abc4fa042ad449
-  data.tar.gz: e5ea8e7fa607a10fd936c62623a39d32b891ef629636e0b04fb3d36aa7ab1f3f
+  metadata.gz: bddc60fa297293af01b1334d4adf51981bfb8b314ba4577a7b3bff1d7385bd5a
+  data.tar.gz: 3533f21ffee4dba606558fa0fadd1f37a943aa285b4f6bebed4bb3b89045b61e
 SHA512:
-  metadata.gz: 4af34ce400c409a61892c9d35693f69e023b5dbd3d54cacfd6d9b044aaf481c616e3fdb9da53470f79919830a916a08f67550ce807eda1c59a43fa1da4092f07
-  data.tar.gz: 81304be6bc9e6925b16fa030d2031377bd90bac894b4c8792cfe5e02ff81677bf04d476b1a3435e951a804e6d284e4d8aa7452c6090969f841dbd6e59a9c301c
+  metadata.gz: 74c0d3df2df08e78d06baa8982401d633af536277231a17d87687ff3c7ba1553d4123f7c1ce49ae23094bd781db5ee23eb9584eeba94aae49fb8cc83da0c68e0
+  data.tar.gz: 7df20f03621cb66b864a2c2950e4a11f60199cd4954ad5a622a1631254fa04bab1243ee376567600a2aed294a8c14fa9802fd1024855dc7d473a7ff8f7166515

data/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## [v0.9.0](https://github.com/bensheldon/good_job/tree/v0.9.0) (2020-07-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.9.0)
+
+**Closed issues:**
+
+- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
+- Add a job timeout configuration to time out jobs that have run too long [\#19](https://github.com/bensheldon/good_job/issues/19)
+
+**Merged pull requests:**
+
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
+- Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
+- Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
+- Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
+- Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
+- Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
+- Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
+- Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))
+
 ## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v0.5.0...v0.6.0)

data/README.md

@@ -1,13 +1,13 @@
 # GoodJob
 
-GoodJob is a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails.
+GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 
-Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob’s design principles are:
+**Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.**
 
-- Stand on the shoulders of ActiveJob. For example, [exception](https://edgeguides.rubyonrails.org/active_job_basics.html#exceptions) and [retry](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs) behavior. 
-- Stand on the shoulders of Ruby on Rails. For example, ActiveRecord ORM, connection pools, and [multithreaded support](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent-Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
-- Stand on the shoulders of Postgres. For example, Advisory Locks.
-- Convention over simplicity over performance. 
+- **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration. 
+- **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby). 
+- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
+- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
 ## Installation
 
@@ -43,6 +43,12 @@ $ bundle install
           t.integer :priority
           t.jsonb :serialized_params
           t.timestamp :scheduled_at
+          t.timestamp :performed_at
+          t.timestamp :finished_at
+          t.text :error
+
+          add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
+          add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
         end
       end
     end
@@ -64,13 +70,18 @@ $ bundle install
     
     ```ruby
     # config/environments/development.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
    
     # config/environments/test.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
    
     # config/environments/production.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
+    ```
+
+1. Queue your job 🎉: 
+    ```ruby
+    YourJob.set(queue: :some_queue, wait: 5.minutes, priority: 10).perform_later
     ```
 
 1. In production, the scheduler is designed to run in its own process:
@@ -90,6 +101,25 @@ $ bundle install
    #   [--queues=queue1,queue2]  # Queues to work from. Separate multiple queues with commas (default: *)
    #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
    ```
+
+### Taking advantage of ActiveJob
+
+ActiveJob has a rich set of built-in functionality for timeouts, error handling, and retrying. For example:
+
+```ruby
+class ApplicationJob < ActiveJob::Base  
+  # Retry errors an infinite number of times with exponential back-off
+  retry_on StandardError, wait: :exponentially_longer, attempts: Float::INFINITY
+
+  # Timeout jobs after 10 minutes
+  JobTimeoutError = Class.new(StandardError)
+  around_perform do |_job, block|
+    Timeout.timeout(10.minutes, JobTimeoutError) do
+      block.call
+    end
+  end
+end
+```
    
 ### Configuring Job Execution Threads
     
@@ -102,6 +132,55 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Migrating to GoodJob from a different ActiveJob backend
+
+If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
+
+1. Enqueue newly created jobs on GoodJob either entirely by setting `ActiveJob::Base.queue_adapter = :good_job` or progressively via individual job classes:
+
+    ```ruby
+    # jobs/specific_job.rb
+    class SpecificJob < ApplicationJob
+      self.queue_adapter = :good_job
+      # ...
+    end
+    ```
+
+1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
+    ```procfile
+    # Procfile
+    # ...
+    worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
+    ```
+
+1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.
+
+### Monitoring and preserving worked jobs
+
+GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
+
+By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`). 
+
+To preserve job records for later inspection, set an initializer:
+
+```ruby
+# config/initializers/good_job.rb
+GoodJob.preserve_job_records = true
+```
+
+It is also necessary to delete these preserved jobs from the database after a certain time period:
+
+- For example, in a Rake task:
+  
+    ```ruby
+    # GoodJob::Job.finished(1.day.ago).delete_all
+    ```
+- For example, using the `good_job` command-line utility:
+
+    ```bash
+    $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
+    ```
+
 ## Development
 
 To run tests:

data/lib/active_job/queue_adapters/good_job_adapter.rb

@@ -1,12 +1,18 @@
 module ActiveJob
   module QueueAdapters
     class GoodJobAdapter < GoodJob::Adapter
-      def initialize
-        if Rails.env.development? || Rails.env.test?
-          super(inline: true)
-        else
-          super(inline: false)
-        end
+      def initialize(execution_mode: nil)
+        execution_mode = if execution_mode
+                           execution_mode
+                         elsif ENV['GOOD_JOB_EXECUTION_MODE'].present?
+                           ENV['GOOD_JOB_EXECUTION_MODE'].to_sym
+                         elsif Rails.env.development? || Rails.env.test?
+                           :inline
+                         else
+                           :external
+                         end
+
+        super(execution_mode: execution_mode)
       end
     end
   end

data/lib/good_job.rb

@@ -4,13 +4,15 @@ require 'good_job/railtie'
 require 'good_job/logging'
 require 'good_job/lockable'
 require 'good_job/job'
-require "good_job/scheduler"
+require 'good_job/scheduler'
 require 'good_job/adapter'
 require 'good_job/pg_locks'
+require 'good_job/performer'
 
 require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
+  mattr_accessor :preserve_job_records, default: false
   include Logging
 
   ActiveSupport.run_load_hooks(:good_job, self)

data/lib/good_job/adapter.rb

@@ -1,7 +1,18 @@
 module GoodJob
   class Adapter
-    def initialize(inline: false)
-      @inline = inline
+    EXECUTION_MODES = [:inline, :external].freeze # TODO: async
+
+    def initialize(execution_mode: nil, inline: false)
+      if inline
+        ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
+        @execution_mode = :inline
+      elsif execution_mode
+        raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(execution_mode)
+
+        @execution_mode = execution_mode
+      else
+        @execution_mode = :external
+      end
     end
 
     def enqueue(active_job)
@@ -11,11 +22,11 @@ module GoodJob
     def enqueue_at(active_job, timestamp)
       good_job = GoodJob::Job.enqueue(
         active_job,
-        scheduled_at: timestamp ? Time.at(timestamp) : nil,
-        create_with_advisory_lock: inline?
+        scheduled_at: timestamp ? Time.zone.at(timestamp) : nil,
+        create_with_advisory_lock: execute_inline?
       )
 
-      if inline?
+      if execute_inline?
         begin
           good_job.perform
         ensure
@@ -30,8 +41,17 @@ module GoodJob
       nil
     end
 
+    def execute_inline?
+      @execution_mode == :inline
+    end
+
     def inline?
-      @inline
+      ActiveSupport::Deprecation.warn('GoodJob::Adapter::inline? is deprecated; use GoodJob::Adapter::execute_inline? instead')
+      execute_inline?
+    end
+
+    def execute_externally?
+      @execution_mode == :external
     end
   end
 end

data/lib/good_job/cli.rb

@@ -36,11 +36,16 @@ module GoodJob
         ENV['GOOD_JOB_POLL_INTERVAL']
       ).to_i
 
-      job_query = GoodJob::Job.all
+      job_query = GoodJob::Job.all.priority_ordered
       queue_names_without_all = queue_names.reject { |q| q == '*' }
       job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
 
-      job_performer = job_query.only_scheduled.priority_ordered.to_performer
+      performer_method = if GoodJob.preserve_job_records
+                           :perform_with_advisory_lock_and_preserve_job_records
+                         else
+                           :perform_with_advisory_lock_and_destroy_job_records
+                         end
+      job_performer = GoodJob::Performer.new(job_query, performer_method)
 
       $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
 
@@ -68,6 +73,19 @@ module GoodJob
       $stdout.puts "GoodJob's jobs finished, exiting..."
     end
 
+    desc :cleanup_preserved_jobs, "Delete preserved job records"
+    method_option :before_seconds_ago,
+                  type: :numeric,
+                  default: 24 * 60 * 60,
+                  desc: "Delete records finished more than this many seconds ago"
+    def cleanup_preserved_jobs
+      require RAILS_ENVIRONMENT_RB
+
+      timestamp = Time.current - options[:before_seconds_ago]
+      result = GoodJob::Job.finished(timestamp).delete_all
+      $stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
+    end
+
     default_task :start
   end
 end

data/lib/good_job/job.rb

@@ -2,39 +2,56 @@ module GoodJob
   class Job < ActiveRecord::Base
     include Lockable
 
-    self.table_name = 'good_jobs'
+    PreviouslyPerformedError = Class.new(StandardError)
 
-    scope :only_scheduled, -> { where("scheduled_at < ?", Time.current).or(where(scheduled_at: nil)) }
-    scope :priority_ordered, -> { order(priority: :desc) }
-    scope :to_performer, -> { Performer.new(self) }
+    DEFAULT_QUEUE_NAME = 'default'.freeze
+    DEFAULT_PRIORITY = 0
 
-    class Performer
-      def initialize(query)
-        @query = query
-      end
+    self.table_name = 'good_jobs'.freeze
 
-      def next
-        good_job = nil
+    scope :unfinished, (lambda do
+      if column_names.include?('finished_at')
+        where(finished_at: nil)
+      else
+        ActiveSupport::Deprecation.warn('GoodJob expects a good_jobs.finished_at column to exist. Please see the GoodJob README.md for migration instructions.')
+        nil
+      end
+    end)
+    scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
+    scope :priority_ordered, -> { order(priority: :desc) }
+    scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
 
-        @query.limit(1).with_advisory_lock do |good_jobs|
-          good_job = good_jobs.first
-          break unless good_job
+    def self.perform_with_advisory_lock(destroy_after: !GoodJob.preserve_job_records)
+      good_job = nil
+      result = nil
+      error = nil
 
-          good_job.perform
-        end
+      unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
+        good_job = good_jobs.first
+        break unless good_job
 
-        good_job
+        result, error = good_job.perform(destroy_after: destroy_after)
       end
+
+      [good_job, result, error] if good_job
+    end
+
+    def self.perform_with_advisory_lock_and_preserve_job_records
+      perform_with_advisory_lock(destroy_after: false)
+    end
+
+    def self.perform_with_advisory_lock_and_destroy_job_records
+      perform_with_advisory_lock(destroy_after: true)
     end
 
     def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
       good_job = nil
       ActiveSupport::Notifications.instrument("enqueue_job.good_job", { active_job: active_job, scheduled_at: scheduled_at, create_with_advisory_lock: create_with_advisory_lock }) do |instrument_payload|
         good_job = GoodJob::Job.new(
-          queue_name: active_job.queue_name,
-          priority: active_job.priority,
+          queue_name: active_job.queue_name.presence || DEFAULT_QUEUE_NAME,
+          priority: active_job.priority || DEFAULT_PRIORITY,
           serialized_params: active_job.serialize,
-          scheduled_at: scheduled_at,
+          scheduled_at: scheduled_at || Time.current,
           create_with_advisory_lock: create_with_advisory_lock
         )
 
@@ -47,16 +64,43 @@ module GoodJob
       good_job
     end
 
-    def perform
+    def perform(destroy_after: true)
+      raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
+
+      result = nil
+      error = nil
+
       ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
+      self.performed_at = Time.current
+      save! unless destroy_after
+
       ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
         params = serialized_params.merge(
           "provider_job_id" => id
         )
-        ActiveJob::Base.execute(params)
+        begin
+          result = ActiveJob::Base.execute(params)
+        rescue StandardError => e
+          error = e
+        end
+      end
+
+      if error.nil? && result.is_a?(Exception)
+        error = result
+        result = nil
+      end
 
+      error_message = "#{error.class}: #{error.message}" if error
+      self.error = error_message
+      self.finished_at = Time.current
+
+      if destroy_after
         destroy!
+      else
+        save!
       end
+
+      [result, error]
     end
   end
 end

data/lib/good_job/lockable.rb

@@ -37,14 +37,17 @@ module GoodJob
       scope :owns_advisory_locked, -> { joins_advisory_locks.where('"pg_locks"."pid" = pg_backend_pid()') }
 
       attr_accessor :create_with_advisory_lock
+
       after_create -> { advisory_lock }, if: :create_with_advisory_lock
     end
 
     class_methods do
-      def with_advisory_lock(&block)
+      def with_advisory_lock
+        raise ArgumentError, "Must provide a block" unless block_given?
+
         records = advisory_lock.to_a
         begin
-          block.call(records)
+          yield(records)
         ensure
           records.each(&:advisory_unlock)
         end
@@ -73,6 +76,8 @@ module GoodJob
     end
 
     def with_advisory_lock
+      raise ArgumentError, "Must provide a block" unless block_given?
+
       advisory_lock!
       yield
     ensure

data/lib/good_job/performer.rb

@@ -0,0 +1,12 @@
+module GoodJob
+  class Performer
+    def initialize(target, method_name)
+      @target = target
+      @method_name = method_name
+    end
+
+    def next
+      @target.public_send(@method_name)
+    end
+  end
+end

data/lib/good_job/scheduler.rb

@@ -59,21 +59,21 @@ module GoodJob
 
     def create_thread
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
-        result = nil
-        Rails.application.executor.wrap { result = performer.next }
-        result
+        output = nil
+        Rails.application.executor.wrap { output = performer.next }
+        output
       end
       future.add_observer(self, :task_observer)
       future.execute
     end
 
-    def timer_observer(time, executed_task, error)
-      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
+    def timer_observer(time, executed_task, thread_error)
+      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
     end
 
-    def task_observer(time, result, error)
-      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: result, error: error, time: time })
-      create_thread if result
+    def task_observer(time, output, thread_error)
+      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
+      create_thread if output
     end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '0.6.0'.freeze
+  VERSION = '0.9.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.9.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-07-15 00:00:00.000000000 Z
+date: 2020-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -178,6 +178,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop-performance
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop-rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rubocop-rspec
   requirement: !ruby/object:Gem::Requirement
@@ -214,18 +242,19 @@ files:
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
 - lib/good_job/logging.rb
+- lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb
 - lib/good_job/scheduler.rb
 - lib/good_job/version.rb
-homepage: https://github.com/benheldon/good_job
+homepage: https://github.com/bensheldon/good_job
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/bensheldon/good_job/issues
   changelog_uri: https://github.com/bensheldon/good_job/blob/master/CHANGELOG.md
   documentation_uri: https://rdoc.info/github/bensheldon/good_job
-  homepage_uri: https://github.com/benheldon/good_job
+  homepage_uri: https://github.com/bensheldon/good_job
   source_code_uri: https://github.com/bensheldon/good_job
 post_install_message: 
 rdoc_options: