good_job 0.8.1 → 1.0.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9d3f717a6ef87a75232173cd29e90842edc441808036116ced1355d0c51957a2
-  data.tar.gz: df7aaaab677397d52694697df2016bc9fa7c65db6bebe4f3a112432b5b9f3a96
+  metadata.gz: f7c64e8766a52e6bbbb5e55b409bb0100e54790d6d88340893e61f930cbf8b9e
+  data.tar.gz: 7e89b7bf2d3aaa45cc5b07deab36f5334a61cac44c0aec06206ddbd6eb0ebc22
 SHA512:
-  metadata.gz: ec049cfa4441c5bfa61b70661ed6543cf22e9e3625036dd5ce824f528a2516347961dd95b1ecccc444587f5a31a353499f50a362005c43ea30263b9bb51d1524
-  data.tar.gz: 4e0c0adbee22e2e8ae825015917cb52fe15475d409d48eec0a43df659de5fca95447f611bcacddcb98095888f8a809136890bed0d1fa9465b7bed38b5b38295b
+  metadata.gz: 944b64f713ad584b56fe386dc63aa6afc0538e498cff4fb46aec9098306b7e436a3d29cebd968157f34e62c9b1b0ce932e15967b4a631f9f04c60ca43d77f4c8
+  data.tar.gz: 5000ffc97bc381cb94999478c1fe90202b681d5cf87d697f4700cc60b94a7417fd67ca3f5428c4b50acb195bde7850a3a5d809e32c5baaf08ae3a58a2101624a

data/CHANGELOG.md

@@ -1,17 +1,81 @@
 # Changelog
 
-## [v0.8.1](https://github.com/bensheldon/good_job/tree/v0.8.1) (2020-07-17)
+## [v1.0.2](https://github.com/bensheldon/good_job/tree/v1.0.2) (2020-07-24)
 
-[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.8.1)
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.1...v1.0.2)
+
+**Fixed bugs:**
+
+- Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
 
 **Closed issues:**
 
-- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
+- repeating/recurring jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
+
+**Merged pull requests:**
+
+- Add migration generator [\#56](https://github.com/bensheldon/good_job/pull/56) ([thedanbob](https://github.com/thedanbob))
+- Fix migration script in readme [\#55](https://github.com/bensheldon/good_job/pull/55) ([thedanbob](https://github.com/thedanbob))
+
+## [v1.0.1](https://github.com/bensheldon/good_job/tree/v1.0.1) (2020-07-22)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.0...v1.0.1)
+
+**Merged pull requests:**
+
+- Change threadpool idletime default to 60 seconds from 0 [\#49](https://github.com/bensheldon/good_job/pull/49) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.0.0](https://github.com/bensheldon/good_job/tree/v1.0.0) (2020-07-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.9.0...v1.0.0)
+
+## [v0.9.0](https://github.com/bensheldon/good_job/tree/v0.9.0) (2020-07-20)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.2...v0.9.0)
+
+**Merged pull requests:**
+
+- Allow preservation of finished job records [\#46](https://github.com/bensheldon/good_job/pull/46) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.8.2](https://github.com/bensheldon/good_job/tree/v0.8.2) (2020-07-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.1...v0.8.2)
+
+**Closed issues:**
+
+- Add a job timeout configuration to time out jobs that have run too long [\#19](https://github.com/bensheldon/good_job/issues/19)
+
+**Merged pull requests:**
+
+- Run Github Action tests on PRs from forks [\#44](https://github.com/bensheldon/good_job/pull/44) ([bensheldon](https://github.com/bensheldon))
+- Fix Rubygems homepage URL [\#43](https://github.com/bensheldon/good_job/pull/43) ([joshmn](https://github.com/joshmn))
+
+## [v0.8.1](https://github.com/bensheldon/good_job/tree/v0.8.1) (2020-07-18)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.8.0...v0.8.1)
 
 **Merged pull requests:**
 
 - Move where\(scheduled\_at: Time.current\) into dynamic part of GoodJob::Job::Performer [\#42](https://github.com/bensheldon/good_job/pull/42) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.8.0](https://github.com/bensheldon/good_job/tree/v0.8.0) (2020-07-17)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.7.0...v0.8.0)
+
+**Merged pull requests:**
+
 - Replace Adapter inline boolean kwarg with execution\_mode instead [\#41](https://github.com/bensheldon/good_job/pull/41) ([bensheldon](https://github.com/bensheldon))
+
+## [v0.7.0](https://github.com/bensheldon/good_job/tree/v0.7.0) (2020-07-16)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v0.6.0...v0.7.0)
+
+**Closed issues:**
+
+- Always store a default priority \(0\) and scheduled\_at\(Time.current\) [\#30](https://github.com/bensheldon/good_job/issues/30)
+
+**Merged pull requests:**
+
 - Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
 - Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
 - Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))

data/README.md

@@ -9,6 +9,8 @@ GoodJob is a multithreaded, Postgres-based, ActiveJob backend for Ruby on Rails.
 - **Backed by Postgres.** Relies upon Postgres integrity and session-level Advisory Locks to provide run-once safety and stay within the limits of `schema.rb`.
 - **For most workloads.** Targets full-stack teams, economy-minded solo developers, and applications that enqueue less than 1-million jobs/day.
 
+For more of the story of GoodJob, read the [introductory blog post](https://island94.org/2020/07/introducing-goodjob-1-0).
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -26,31 +28,9 @@ $ bundle install
 
 1. Create a database migration:
     ```bash
-    $ bin/rails g migration CreateGoodJobs
+    $ bin/rails g good_job:install
     ```
 
-    Add to the newly created migration file:
-
-    ```ruby
-    class CreateGoodJobs < ActiveRecord::Migration[6.0]
-      def change
-        enable_extension 'pgcrypto'
-
-        create_table :good_jobs, id: :uuid do |t|
-          t.timestamps
-
-          t.text :queue_name
-          t.integer :priority
-          t.jsonb :serialized_params
-          t.timestamp :scheduled_at
-   
-          t.index :scheduled_at
-          t.index [:queue_name, :scheduled_at]
-        end
-      end
-    end
-    ```
-    
     Run the migration:
     
     ```bash
@@ -152,6 +132,32 @@ If your application is already using an ActiveJob backend, you will need to inst
 
 1. Once you are confident that no unperformed jobs remain in the previous ActiveJob backend, code and configuration for that backend can be completely removed.
 
+### Monitoring and preserving worked jobs
+
+GoodJob is fully instrumented with [`ActiveSupport::Notifications`](https://edgeguides.rubyonrails.org/active_support_instrumentation.html#introduction-to-instrumentation).
+
+By default, GoodJob will delete job records after they are run, regardless of whether they succeed or not (raising a kind of `StandardError`), unless they are interrupted (raising a kind of `Exception`). 
+
+To preserve job records for later inspection, set an initializer:
+
+```ruby
+# config/initializers/good_job.rb
+GoodJob.preserve_job_records = true
+```
+
+It is also necessary to delete these preserved jobs from the database after a certain time period:
+
+- For example, in a Rake task:
+  
+    ```ruby
+    # GoodJob::Job.finished(1.day.ago).delete_all
+    ```
+- For example, using the `good_job` command-line utility:
+
+    ```bash
+    $ bundle exec good_job cleanup_preserved_jobs --before-seconds-ago=86400
+    ```
+
 ## Development
 
 To run tests:

data/lib/generators/good_job/install_generator.rb

@@ -0,0 +1,24 @@
+require 'rails/generators'
+require 'rails/generators/active_record'
+
+module GoodJob
+  class InstallGenerator < Rails::Generators::Base
+    include Rails::Generators::Migration
+
+    class << self
+      delegate :next_migration_number, to: ActiveRecord::Generators::Base
+    end
+
+    source_paths << File.join(File.dirname(__FILE__), "templates")
+
+    def create_migration_file
+      migration_template 'migration.rb.erb', 'db/migrate/create_good_jobs.rb', migration_version: migration_version
+    end
+
+    private
+
+    def migration_version
+      "[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]"
+    end
+  end
+end

data/lib/generators/good_job/templates/migration.rb.erb

@@ -0,0 +1,20 @@
+class CreateGoodJobs < ActiveRecord::Migration<%= migration_version %>
+  def change
+    enable_extension 'pgcrypto'
+
+    create_table :good_jobs, id: :uuid do |t|
+      t.text :queue_name
+      t.integer :priority
+      t.jsonb :serialized_params
+      t.timestamp :scheduled_at
+      t.timestamp :performed_at
+      t.timestamp :finished_at
+      t.text :error
+
+      t.timestamps
+    end
+
+    add_index :good_jobs, :scheduled_at, where: "(finished_at IS NULL)"
+    add_index :good_jobs, [:queue_name, :scheduled_at], where: "(finished_at IS NULL)"
+  end
+end

data/lib/good_job.rb

@@ -4,13 +4,15 @@ require 'good_job/railtie'
 require 'good_job/logging'
 require 'good_job/lockable'
 require 'good_job/job'
-require "good_job/scheduler"
+require 'good_job/scheduler'
 require 'good_job/adapter'
 require 'good_job/pg_locks'
+require 'good_job/performer'
 
 require 'active_job/queue_adapters/good_job_adapter'
 
 module GoodJob
+  mattr_accessor :preserve_job_records, default: false
   include Logging
 
   ActiveSupport.run_load_hooks(:good_job, self)

data/lib/good_job/cli.rb

@@ -40,7 +40,12 @@ module GoodJob
       queue_names_without_all = queue_names.reject { |q| q == '*' }
       job_query = job_query.where(queue_name: queue_names_without_all) unless queue_names_without_all.size.zero?
 
-      job_performer = job_query.to_performer
+      performer_method = if GoodJob.preserve_job_records
+                           :perform_with_advisory_lock_and_preserve_job_records
+                         else
+                           :perform_with_advisory_lock_and_destroy_job_records
+                         end
+      job_performer = GoodJob::Performer.new(job_query, performer_method)
 
       $stdout.puts "GoodJob worker starting with max_threads=#{max_threads} on queues=#{queue_names.join(',')}"
 
@@ -68,6 +73,19 @@ module GoodJob
       $stdout.puts "GoodJob's jobs finished, exiting..."
     end
 
+    desc :cleanup_preserved_jobs, "Delete preserved job records"
+    method_option :before_seconds_ago,
+                  type: :numeric,
+                  default: 24 * 60 * 60,
+                  desc: "Delete records finished more than this many seconds ago"
+    def cleanup_preserved_jobs
+      require RAILS_ENVIRONMENT_RB
+
+      timestamp = Time.current - options[:before_seconds_ago]
+      result = GoodJob::Job.finished(timestamp).delete_all
+      $stdout.puts "Deleted #{result} preserved #{'job'.pluralize(result)} finished before #{timestamp}."
+    end
+
     default_task :start
   end
 end

data/lib/good_job/job.rb

@@ -2,32 +2,46 @@ module GoodJob
   class Job < ActiveRecord::Base
     include Lockable
 
+    PreviouslyPerformedError = Class.new(StandardError)
+
     DEFAULT_QUEUE_NAME = 'default'.freeze
     DEFAULT_PRIORITY = 0
 
     self.table_name = 'good_jobs'.freeze
 
+    scope :unfinished, (lambda do
+      if column_names.include?('finished_at')
+        where(finished_at: nil)
+      else
+        ActiveSupport::Deprecation.warn('GoodJob expects a good_jobs.finished_at column to exist. Please see the GoodJob README.md for migration instructions.')
+        nil
+      end
+    end)
     scope :only_scheduled, -> { where(arel_table['scheduled_at'].lteq(Time.current)).or(where(scheduled_at: nil)) }
     scope :priority_ordered, -> { order(priority: :desc) }
-    scope :to_performer, -> { Performer.new(self) }
+    scope :finished, ->(timestamp = nil) { timestamp ? where(arel_table['finished_at'].lteq(timestamp)) : where.not(finished_at: nil) }
 
-    class Performer
-      def initialize(query)
-        @query = query
-      end
+    def self.perform_with_advisory_lock(destroy_after: !GoodJob.preserve_job_records)
+      good_job = nil
+      result = nil
+      error = nil
+
+      unfinished.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
+        good_job = good_jobs.first
+        break unless good_job
 
-      def next
-        good_job = nil
+        result, error = good_job.perform(destroy_after: destroy_after)
+      end
 
-        @query.only_scheduled.limit(1).with_advisory_lock do |good_jobs|
-          good_job = good_jobs.first
-          break unless good_job
+      [good_job, result, error] if good_job
+    end
 
-          good_job.perform
-        end
+    def self.perform_with_advisory_lock_and_preserve_job_records
+      perform_with_advisory_lock(destroy_after: false)
+    end
 
-        good_job
-      end
+    def self.perform_with_advisory_lock_and_destroy_job_records
+      perform_with_advisory_lock(destroy_after: true)
     end
 
     def self.enqueue(active_job, scheduled_at: nil, create_with_advisory_lock: false)
@@ -50,16 +64,43 @@ module GoodJob
       good_job
     end
 
-    def perform
+    def perform(destroy_after: true)
+      raise PreviouslyPerformedError, 'Cannot perform a job that has already been performed' if finished_at
+
+      result = nil
+      error = nil
+
       ActiveSupport::Notifications.instrument("before_perform_job.good_job", { good_job: self })
+      self.performed_at = Time.current
+      save! unless destroy_after
+
       ActiveSupport::Notifications.instrument("perform_job.good_job", { good_job: self }) do
         params = serialized_params.merge(
           "provider_job_id" => id
         )
-        ActiveJob::Base.execute(params)
+        begin
+          result = ActiveJob::Base.execute(params)
+        rescue StandardError => e
+          error = e
+        end
+      end
 
+      if error.nil? && result.is_a?(Exception)
+        error = result
+        result = nil
+      end
+
+      error_message = "#{error.class}: #{error.message}" if error
+      self.error = error_message
+      self.finished_at = Time.current
+
+      if destroy_after
         destroy!
+      else
+        save!
       end
+
+      [result, error]
     end
   end
 end

data/lib/good_job/performer.rb

@@ -0,0 +1,12 @@
+module GoodJob
+  class Performer
+    def initialize(target, method_name)
+      @target = target
+      @method_name = method_name
+    end
+
+    def next
+      @target.public_send(@method_name)
+    end
+  end
+end

data/lib/good_job/scheduler.rb

@@ -15,19 +15,18 @@ module GoodJob
       min_threads: 0,
       max_threads: Concurrent.processor_count,
       auto_terminate: true,
-      idletime: 0,
-      max_queue: 0,
-      fallback_policy: :abort, # shouldn't matter -- 0 max queue
+      idletime: 60,
+      max_queue: -1,
+      fallback_policy: :discard,
     }.freeze
 
     def initialize(performer, timer_options: {}, pool_options: {})
       raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 
       @performer = performer
-      @pool = Concurrent::ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
+      @pool = ThreadPoolExecutor.new(DEFAULT_POOL_OPTIONS.merge(pool_options))
       @timer = Concurrent::TimerTask.new(DEFAULT_TIMER_OPTIONS.merge(timer_options)) do
-        idle_threads = @pool.max_length - @pool.length
-        create_thread if idle_threads.positive?
+        create_thread
       end
       @timer.add_observer(self, :timer_observer)
       @timer.execute
@@ -58,22 +57,36 @@ module GoodJob
     end
 
     def create_thread
+      return false unless @pool.ready_worker_count.positive?
+
       future = Concurrent::Future.new(args: [@performer], executor: @pool) do |performer|
-        result = nil
-        Rails.application.executor.wrap { result = performer.next }
-        result
+        output = nil
+        Rails.application.executor.wrap { output = performer.next }
+        output
       end
       future.add_observer(self, :task_observer)
       future.execute
     end
 
-    def timer_observer(time, executed_task, error)
-      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: error, time: time })
+    def timer_observer(time, executed_task, thread_error)
+      ActiveSupport::Notifications.instrument("finished_timer_task.good_job", { result: executed_task, error: thread_error, time: time })
+    end
+
+    def task_observer(time, output, thread_error)
+      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: output, error: thread_error, time: time })
+      create_thread if output
     end
 
-    def task_observer(time, result, error)
-      ActiveSupport::Notifications.instrument("finished_job_task.good_job", { result: result, error: error, time: time })
-      create_thread if result
+    class ThreadPoolExecutor < Concurrent::ThreadPoolExecutor
+      # https://github.com/ruby-concurrency/concurrent-ruby/issues/684#issuecomment-427594437
+      def ready_worker_count
+        synchronize do
+          workers_still_to_be_created = @max_length - @pool.length
+          workers_created_but_waiting = @ready.length
+
+          workers_still_to_be_created + workers_created_but_waiting
+        end
+      end
     end
   end
 end

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '0.8.1'.freeze
+  VERSION = '1.0.2'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 0.8.1
+  version: 1.0.2
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-07-18 00:00:00.000000000 Z
+date: 2020-07-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -236,24 +236,27 @@ files:
 - README.md
 - exe/good_job
 - lib/active_job/queue_adapters/good_job_adapter.rb
+- lib/generators/good_job/install_generator.rb
+- lib/generators/good_job/templates/migration.rb.erb
 - lib/good_job.rb
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
 - lib/good_job/logging.rb
+- lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb
 - lib/good_job/scheduler.rb
 - lib/good_job/version.rb
-homepage: https://github.com/benheldon/good_job
+homepage: https://github.com/bensheldon/good_job
 licenses:
 - MIT
 metadata:
   bug_tracker_uri: https://github.com/bensheldon/good_job/issues
   changelog_uri: https://github.com/bensheldon/good_job/blob/master/CHANGELOG.md
   documentation_uri: https://rdoc.info/github/bensheldon/good_job
-  homepage_uri: https://github.com/benheldon/good_job
+  homepage_uri: https://github.com/bensheldon/good_job
   source_code_uri: https://github.com/bensheldon/good_job
 post_install_message: 
 rdoc_options: