good_job 0.7.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4f7dda29494df3fc05a199d6bf0efbc825a47653c3af507a0d1ceb92456d4307
-  data.tar.gz: 881ad046bab6b17c5d532d035fddef54420639288a0a4bd8f4f07b980c1fa239
+  metadata.gz: ff9a5cf449c835415f08a2b5704ad5345519b4cf73ab987de72a011101700a96
+  data.tar.gz: 0d7a104ef5440c406d5bec5f65b80099d9a2678cc72bd6b399dde1912e49f0db
 SHA512:
-  metadata.gz: c0cfc3c4d61666844a6bd7c532670e0b1be40fd6f805eff63919d3600d06933b8029f0febbc81efbf52b1f3d459317caeb522affe20dde9d3f4d7672622fd708
-  data.tar.gz: c4c9a9fddb2108769e3672baeaeaca2f7c966ee708f95867ed57970e688debcf1ce41a3eacd65d88c0192700f1b293cd4a9ad63fee9e4af93d44934b0876120f
+  metadata.gz: 6c027b651e5d9bc9980f4b8e912aa64f5a393b78b0b291195446901b1f2dabddd2022b1ed50e4fab8783983496c4b00025339630d49dc7a894cfb63b8f8292b6
+  data.tar.gz: eea0bc9425b6aad9a6947336ffdad87da6180febb1128eec3d5f99a2d6b3613be5d1293ec6666b900f9ca38ad6a657fd3f6a2ee75fa1271cb6767513aa688374

data/CHANGELOG.md

@@ -1,15 +1,21 @@
 # Changelog
 
-## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-15)
+## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
 
-[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.7.0)
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v1.0.0)
 
 **Closed issues:**
 
 - Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
+- Add a job timeout configuration to time out jobs that have run too long [\#19](https://github.com/bensheldon/good_job/issues/19)
 
 **Merged pull requests:**
 
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
+- Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
+- Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
+- Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
+- Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
 - Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
 - Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
 - Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))

data/README.md

@@ -1,21 +1,13 @@
 # GoodJob
 
-GoodJob is a multithreaded, Postgres-based ActiveJob backend for Ruby on Rails.
+GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 
-Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob’s design principles are:
+**Inspired by [Delayed::Job](https://github.com/collectiveidea/delayed_job) and [Que](https://github.com/que-rb/que), GoodJob is designed for maximum compatibility with Ruby on Rails, ActiveJob, and Postgres to be simple and performant for most workloads.**
 
-- Stand on the shoulders of ActiveJob. For example, [exception](https://edgeguides.rubyonrails.org/active_job_basics.html#exceptions) and [retry](https://edgeguides.rubyonrails.org/active_job_basics.html#retrying-or-discarding-failed-jobs) behavior. 
-- Stand on the shoulders of Ruby on Rails. For example, ActiveRecord ORM, connection pools, and [multithreaded support](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent-Ruby](https://github.com/ruby-concurrency/concurrent-ruby).
-- Stand on the shoulders of Postgres. For example, Advisory Locks.
-- Convention over simplicity over performance.
-
-GoodJob supports all ActiveJob functionality:
-- Async. GoodJob has the ability to run the job in a non-blocking manner.
-- Queues. Jobs may set which queue they are run in with queue_as or by using the set method.
-- Delayed. GoodJob will run the job in the future through perform_later.
-- Priorities. The order in which jobs are processed can be configured differently.
-- Timeouts. GoodJob defers to ActiveJob where it can be implemented as an `around` hook. See [Taking advantage of ActiveJob](#taking-advantage-of-activejob).
-- Retries. GoodJob will automatically retry uncompleted jobs immediately. See [Taking advantage of ActiveJob](#taking-advantage-of-activejob).
+- **Designed for ActiveJob.** Complete support for [async, queues, delays, priorities, timeouts, and retries](https://edgeguides.rubyonrails.org/active_job_basics.html) with near-zero configuration. 
+- **Built for Rails.** Fully adopts Ruby on Rails [threading and code execution guidelines](https://guides.rubyonrails.org/threading_and_code_execution.html) with [Concurrent::Ruby](https://github.com/ruby-concurrency/concurrent-ruby). 
+- **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
+- **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
 ## Installation
 
@@ -51,9 +43,12 @@ $ bundle install
           t.integer :priority
           t.jsonb :serialized_params
           t.timestamp :scheduled_at
-   
-          t.index :scheduled_at
-          t.index [:queue_name, :scheduled_at]
+          t.timestamp :performed_at
+          t.timestamp :finished_at
+          t.text :error
+
+          add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
+          add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
         end
       end
     end
@@ -75,13 +70,13 @@ $ bundle install
     
     ```ruby
     # config/environments/development.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
    
     # config/environments/test.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new(inline: true)
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :inline)
    
     # config/environments/production.rb
-    config.active_job.queue_adapter = GoodJob::Adapter.new
+    config.active_job.queue_adapter = GoodJob::Adapter.new(execution_mode: :external)
     ```
 
 1. Queue your job 🎉: 
@@ -137,6 +132,55 @@ GoodJob executes enqueued jobs using threads. There is a lot than can be said ab
     3. `$ RAILS_MAX_THREADS=4 bundle exec good_job`
     4. Implicitly via Rails's database connection pool size (`ActiveRecord::Base.connection_pool.size`)
 
+### Migrating to GoodJob from a different ActiveJob backend
+
+If your application is already using an ActiveJob backend, you will need to install GoodJob to enqueue and perform newly created jobs _and_ finish performing pre-existing jobs on the previous backend.
+
+1. Enqueue newly created jobs on GoodJob either entirely by setting `ActiveJob::Base.queue_adapter = :good_job` or progressively via individual job classes:
+
+    ```ruby
+    # jobs/specific_job.rb
+    class SpecificJob < ApplicationJob
+      self.queue_adapter = :good_job
+      # ...
+    end
+    ```
+
+1. Continue running executors for both backends. For example, on Heroku it's possible to run [two processes](https://help.heroku.com/CTFS2TJK/how-do-i-run-multiple-processes-on-a-dyno) within the same dyno:
+    ```procfile
+    # Procfile
+    # ...
+    worker: bundle exec que ./config/environment.rb & bundle exec good_job & wait -n
+    ```
+
+1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.
+
+### Monitoring and preserving worked jobs
+
+GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
+
+By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`). 
+
+To preserve job records for later inspection, set an initializer:
+
+```ruby
+# config/initializers/good_job.rb
+GoodJob.preserve_job_records = true
+```
+
+It is also necessary to delete these preserved jobs from the database after a certain time period:
+
+- For example, in a Rake task:
+  
+    ```ruby
+    # GoodJob::Job.finished(1.day.ago).delete_all
+    ```
+- For example, using the `good_job` command-line utility:
+
+    ```bash
+    $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
+    ```
+
 ## Development
 
 To run tests:

data/lib/active_job/queue_adapters/good_job_adapter.rb

@@ -1,12 +1,18 @@
 module ActiveJob
   module QueueAdapters
     class GoodJobAdapter < GoodJob::Adapter
-      def initialize
-        if Rails.env.development? || Rails.env.test?
-          super(inline: true)
-        else
-          super(inline: false)
-        end
+      def initialize(execution_mode: nil)
+        execution_mode = if execution_mode
+                           execution_mode
+                         elsif ENV['GOOD_JOB_EXECUTION_MODE'].present?
+                           ENV['GOOD_JOB_EXECUTION_MODE'].to_sym
+                         elsif Rails.env.development? || Rails.env.test?
+                           :inline
+                         else
+                           :external
+                         end
+
+        super(execution_mode: execution_mode)
       end
     end
   end

data/lib/good_job.rb

@@ -4,13 +4,15 @@ require 'good_job/railtie'
 require 'good_job/logging'
 require 'good_job/lockable'
 require 'good_job/job'
-require "good_job/scheduler"
+require 'good_job/scheduler'
 require 'good_job/adapter'
 require 'good_job/pg_locks'
+require 'good_job/performer'
 
 require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
+  mattr_accessor :preserve_job_records, default: false
   include Logging
 
   ActiveSupport.run_load_hooks(:good_job, self)

data/lib/good_job/adapter.rb

@@ -1,7 +1,18 @@
 module GoodJob
   class Adapter
-    def initialize(inline: false)
-      @inline = inline
+    EXECUTION_MODES = [:inline, :external].freeze # TODO: async
+
+    def initialize(execution_mode: nil, inline: false)
+      if inline
+        ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
+        @execution_mode = :inline
+      elsif execution_mode
+        raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(execution_mode)
+
+        @execution_mode = execution_mode
+      else
+        @execution_mode = :external
+      end
     end
 
     def enqueue(active_job)
@@ -12,10 +23,10 @@ module GoodJob
       good_job = GoodJob::Job.enqueue(
         active_job,
         scheduled_at: timestamp ? Time.zone.at(timestamp) : nil,
-        create_with_advisory_lock: inline?
+        create_with_advisory_lock: execute_inline?
       )
 
-      if inline?
+      if execute_inline?
         begin
           good_job.perform
         ensure
@@ -30,8 +41,17 @@ module GoodJob
       nil
     end
 
+    def execute_inline?
+      @execution_mode == :inline
+    end
+
     def inline?
-      @inline
+      ActiveSupport::Deprecation.warn('GoodJob::Adapter::inline? is deprecated; use GoodJob::Adapter::execute_inline? instead')
+      execute_inline?
+    end
+
+    def execute_externally?
+      @execution_mode == :external
     end
   end
 end

data/lib/good_job/cli.rb

@@ -36,11 +36,16 @@ module GoodJob
         ENV['GOOD_JOB_POLL_INTERVAL']
       ).to_i
 
-      job_query = GoodJob::Job.all
+      job_query = GoodJob::Job.all.priority_ordered
       queue_names_without_all = queue_names.reject { |q| q == '*' }
       job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
 
-      job_performer = job_query.only_scheduled.priority_ordered.to_performer
+      performer_method = if GoodJob.preserve_job_records
+                           :perform_with_advisory_lock_and_preserve_job_records
+                         else
+                           :perform_with_advisory_lock_and_destroy_job_records
+                         end
+      job_performer = GoodJob::Performer.new(job_query, performer_method)
 
       $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
 
@@ -68,6 +73,19 @@ module GoodJob
       $stdout.puts "GoodJob's jobs finished, exiting..."
     end
 
+    desc :cleanup_preserved_jobs, "Delete preserved job records"
+    method_option :before_seconds_ago,
+                  type: :numeric,
+                  default: 24 * 60 * 60,
+                  desc: "Delete records finished more than this many seconds ago"
+    def cleanup_preserved_jobs
+      require RAILS_ENVIRONMENT_RB
+
+      timestamp = Time.current - options[:before_seconds_ago]
+      result = GoodJob::Job.finished(timestamp).delete_all
+      $stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
+    end
+
     default_task :start
   end
 end

data/lib/good_job/job.rb

@@ -2,32 +2,46 @@ module GoodJob
   class Job < ActiveRecord::Base
     include Lockable
 
+    PreviouslyPerformedError = Class.new(StandardError)
+
     DEFAULT_QUEUE_NAME = 'default'.freeze
     DEFAULT_PRIORITY = 0
 
     self.table_name = 'good_jobs'.freeze
 
+    scope :unfinished, (lambda do
+      if column_names.include?('finished_at')
+        where(finished_at: nil)
+      else
+        ActiveSupport::Deprecation.warn('GoodJob expects a good_jobs.finished_at column to exist. Please see the GoodJob README.md for migration instructions.')
+        nil
+      end
+    end)
     scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
     scope :priority_ordered, -> { order(priority: :desc) }
-    scope :to_performer, -> { Performer.new(self) }
+    scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
 
-    class Performer
-      def initialize(query)
-        @query = query
-      end
+    def self.perform_with_advisory_lock(destroy_after: !GoodJob.preserve_job_records)
+      good_job = nil
+      result = nil
+      error = nil
+
+      unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
+        good_job = good_jobs.first
+        break unless good_job
 
-      def next
-        good_job = nil
+        result, error = good_job.perform(destroy_after: destroy_after)
+      end
 
-        @query.limit(1).with_advisory_lock do |good_jobs|
-          good_job = good_jobs.first
-          break unless good_job
+      [good_job, result, error] if good_job
+    end
 
-          good_job.perform
-        end
+    def self.perform_with_advisory_lock_and_preserve_job_records
+      perform_with_advisory_lock(destroy_after: false)
+    end
 
-        good_job
-      end
+    def self.perform_with_advisory_lock_and_destroy_job_records
+      perform_with_advisory_lock(destroy_after: true)
     end
 
     def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
@@ -50,16 +64,43 @@ module GoodJob
       good_job
     end
 
-    def perform
+    def perform(destroy_after: true)
+      raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
+
+      result = nil
+      error = nil
+
       ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
+      self.performed_at = Time.current
+      save! unless destroy_after
+
       ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
         params = serialized_params.merge(
           "provider_job_id" => id
         )
-        ActiveJob::Base.execute(params)
+        begin
+          result = ActiveJob::Base.execute(params)
+        rescue StandardError => e
+          error = e
+        end
+      end
 
+      if error.nil? && result.is_a?(Exception)
+        error = result
+        result = nil
+      end
+
+      error_message = "#{error.class}: #{error.message}" if error
+      self.error = error_message
+      self.finished_at = Time.current
+
+      if destroy_after
         destroy!
+      else
+        save!
       end
+
+      [result, error]
     end
   end
 end

data/lib/good_job/performer.rb

@@ -0,0 +1,12 @@
+module GoodJob
+  class Performer
+    def initialize(target, method_name)
+      @target = target
+      @method_name = method_name
+    end
+
+    def next
+      @target.public_send(@method_name)
+    end
+  end
+end

data/lib/good_job/scheduler.rb

@@ -59,21 +59,21 @@ module GoodJob
 
     def create_thread
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
-        result = nil
-        Rails.application.executor.wrap { result = performer.next }
-        result
+        output = nil
+        Rails.application.executor.wrap { output = performer.next }
+        output
       end
       future.add_observer(self, :task_observer)
       future.execute
     end
 
-    def timer_observer(time, executed_task, error)
-      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
+    def timer_observer(time, executed_task, thread_error)
+      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
     end
 
-    def task_observer(time, result, error)
-      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: result, error: error, time: time })
-      create_thread if result
+    def task_observer(time, output, thread_error)
+      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
+      create_thread if output
     end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '0.7.0'.freeze
+  VERSION = '1.0.0'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 1.0.0
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-07-16 00:00:00.000000000 Z
+date: 2020-07-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -242,18 +242,19 @@ files:
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
 - lib/good_job/logging.rb
+- lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb
 - lib/good_job/scheduler.rb
 - lib/good_job/version.rb
-homepage: https://github.com/benheldon/good_job
+homepage: https://github.com/bensheldon/good_job
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/bensheldon/good_job/issues
   changelog_uri: https://github.com/bensheldon/good_job/blob/master/CHANGELOG.md
   documentation_uri: https://rdoc.info/github/bensheldon/good_job
-  homepage_uri: https://github.com/benheldon/good_job
+  homepage_uri: https://github.com/bensheldon/good_job
   source_code_uri: https://github.com/bensheldon/good_job
 post_install_message: 
 rdoc_options: