good_job 1.1.0 → 1.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7c6d495a4455453f6f4af736d3fd31685fc2026c98e0cd564ea0f47b188cc473
-  data.tar.gz: 794137e732ed3fcebec859daa8e58a2c2c8fd283cc43c5cfc7824263a1ca93fd
+  metadata.gz: b7b7100f183bea75e74273f8c60b87eb00f234b56196f1239ad5c1677e216250
+  data.tar.gz: b22120c6cdb91f38e201ef2fd8dd150f7775462ba27e7b9646bad504384c94d0
 SHA512:
-  metadata.gz: 258757117262f25f5507ceb47c20f0f43a1f40427c7a0f9c3d0dc7803f25019062a160de08e96fd5ba74ceeeec5666d1084f6dff7ca9d381b2bbb198fabb10fd
-  data.tar.gz: e2a5d09cfb57a7f2a08d59f84c3b39e3582cb5b0b76a7e6ea2da421542d5e0ebee2e25a79298c1f5d3351d58e156e43ac0770e1d9e26e0f26169620f0bfcc493
+  metadata.gz: '0844d72f1fb34e1608f40d6c134a3a24e8e5e0876ca4c96eaa246568db87f64b47be8170e5061517f455148da5e40fdd0a278a8a790a93a7ddb64eb024fdb35f'
+  data.tar.gz: 20934ee8ab6fc277519c2a551ee44d6803d09405215faff35fda4227e88729d720968235003e694f83887cea82709caa68eabc7afbb10335b2a671b3bddfe604

data/CHANGELOG.md

@@ -1,6 +1,19 @@
 # Changelog
 
-## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-09)
+## [v1.1.1](https://github.com/bensheldon/good_job/tree/v1.1.1) (2020-08-12)
+
+[Full Changelog](https://github.com/bensheldon/good_job/compare/v1.1.0...v1.1.1)
+
+**Implemented enhancements:**
+
+- Allow multiple schedulers within the same process. e.g. `queues=mice:2,elephants:4` [\#45](https://github.com/bensheldon/good_job/issues/45)
+
+**Merged pull requests:**
+
+- Allow instantiation of multiple schedulers via --queues [\#76](https://github.com/bensheldon/good_job/pull/76) ([bensheldon](https://github.com/bensheldon))
+- Extract options parsing to Configuration object [\#74](https://github.com/bensheldon/good_job/pull/74) ([bensheldon](https://github.com/bensheldon))
+
+## [v1.1.0](https://github.com/bensheldon/good_job/tree/v1.1.0) (2020-08-10)
 
 [Full Changelog](https://github.com/bensheldon/good_job/compare/v1.0.3...v1.1.0)
 
@@ -44,10 +57,6 @@
 
 - Fix counting of available execution threads [\#58](https://github.com/bensheldon/good_job/pull/58) ([bensheldon](https://github.com/bensheldon))
 
-**Closed issues:**
-
-- repeating/recurring jobs [\#53](https://github.com/bensheldon/good_job/issues/53)
-
 **Merged pull requests:**
 
 - Add migration generator [\#56](https://github.com/bensheldon/good_job/pull/56) ([thedanbob](https://github.com/thedanbob))
@@ -115,6 +124,7 @@
 - Add more examples to Readme [\#39](https://github.com/bensheldon/good_job/pull/39) ([bensheldon](https://github.com/bensheldon))
 - Add additional Rubocops and lint [\#38](https://github.com/bensheldon/good_job/pull/38) ([bensheldon](https://github.com/bensheldon))
 - Always store a default queue\_name, priority and scheduled\_at; index by queue\_name and scheduled\_at [\#37](https://github.com/bensheldon/good_job/pull/37) ([bensheldon](https://github.com/bensheldon))
+- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.6.0](https://github.com/bensheldon/good_job/tree/v0.6.0) (2020-07-15)
 
@@ -130,7 +140,6 @@
 - Improve generation of changelog [\#36](https://github.com/bensheldon/good_job/pull/36) ([bensheldon](https://github.com/bensheldon))
 - Update Github Action Workflow for Backlog Project Board [\#35](https://github.com/bensheldon/good_job/pull/35) ([bensheldon](https://github.com/bensheldon))
 - Add configuration options to good\_job executable [\#33](https://github.com/bensheldon/good_job/pull/33) ([bensheldon](https://github.com/bensheldon))
-- Extract Job querying behavior out of Scheduler [\#31](https://github.com/bensheldon/good_job/pull/31) ([bensheldon](https://github.com/bensheldon))
 - Allow configuration of Rails queue adapter with `:good\_job` [\#28](https://github.com/bensheldon/good_job/pull/28) ([bensheldon](https://github.com/bensheldon))
 
 ## [v0.5.0](https://github.com/bensheldon/good_job/tree/v0.5.0) (2020-07-13)

data/README.md

@@ -72,17 +72,55 @@ $ bundle install
    
    Configuration options available with `help`:
 
-   ```bash
-   $ bundle exec good_job help start
+    ```bash
+    $ bundle exec good_job help start
    
-   # Usage:
-   #  good_job start
-   #
-   # Options:
-   #   [--max-threads=N]         # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
-   #   [--queues=queue1,queue2]  # Queues to work from. Separate multiple queues with commas (default: *)
-   #   [--poll-interval=N]       # Interval between polls for available jobs in seconds (default: 1)
-   ```
+    Usage:
+      good_job start
+    
+    Options:
+      [--max-threads=N]                           # Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)
+      [--queues=queue1,queue2(;queue3,queue4:5)]  # Queues to work from. Separate multiple queues with commas; separate isolated execution pools with semicolons and threads with colons (default: *)
+      [--poll-interval=N]                         # Interval between polls for available jobs in seconds (default: 1)
+    
+    Start job worker
+    ```
+
+1. Optimize execution to reduce congestion and execution latency. By default, GoodJob creates a single thread execution pool that will execute jobs from any queue. Depending on your application's workload, job types, and service level objectives, you may wish to optimize execution resources; for example, providing dedicated execution resources for transactional emails so they are not delayed by long-running batch jobs. Some options:
+
+    - Multiple execution pools within a single process:
+        
+        ```bash
+        $ bundle exec good_job --queues=*;transactional_messages:2;batch_processing:1 --max-threads=5
+        ```
+      
+        This configuration will result in a single process with 3 isolated thread execution pools. A pool that will run jobs from any queue, `*`, with up to 5 threads; a pool that will only run jobs enqueued on `transactional_messages` with up to 2 threads; and a pool dedicated to the `batch_processing` queue with a single thread.
+        
+        For moderate workloads, multiple isolated thread execution pools offers a good balance between congestion management and economy. 
+        
+        Configuration can be injected by environment variables too:
+        
+        ```bash
+        $ GOOD_JOB_QUEUES="*;transactional_messages:2;batch_processing:1" GOOD_JOB_MAX_THREADS=5 bundle exec good_job
+        ```
+        
+    - Multiple processes; for example, on Heroku:
+    
+        ```procfile
+        # Procfile
+        
+        # Separate dyno types
+        worker: bundle exec good_job --max-threads=5
+        transactional_worker: bundle exec good_job --queues=transactional_messages --max-threads=2
+        batch_worker: bundle exec good_job --queues=batch_processing --max-threads=1
+      
+        # Combined multi-process dyno
+        combined_worker: bundle exec good_job --max-threads=5 & bundle exec good_job --queues=transactional_messages --max-threads=2 & bundle exec good_job --queues=batch_processing --max-threads=1 & wait -n
+        ```
+      
+      Running multiple processes can optimize for CPU performance at the expense of greater memory and system resource usage.
+
+    _Keep in mind, queue operations and management is an advanced discipline. This stuff is complex, especially for heavy workloads and unique processing requirements. Good job 👍_
 
 ### Error handling, retries, and reliability
 

data/lib/active_job/queue_adapters/good_job_adapter.rb

@@ -1,35 +1,9 @@
 module ActiveJob
   module QueueAdapters
     class GoodJobAdapter < GoodJob::Adapter
-      def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil)
-        execution_mode = if execution_mode
-                           execution_mode
-                         elsif ENV['GOOD_JOB_EXECUTION_MODE'].present?
-                           ENV['GOOD_JOB_EXECUTION_MODE'].to_sym
-                         elsif Rails.env.development?
-                           :inline
-                         elsif Rails.env.test?
-                           :inline
-                         else
-                           :external
-                         end
-
-        if execution_mode == :async && scheduler.blank?
-          max_threads = (
-            max_threads.presence ||
-            ENV['GOOD_JOB_MAX_THREADS'] ||
-            ENV['RAILS_MAX_THREADS'] ||
-            ActiveRecord::Base.connection_pool.size
-          ).to_i
-
-          poll_interval = (
-            poll_interval.presence ||
-            ENV['GOOD_JOB_POLL_INTERVAL'] ||
-            1
-          ).to_i
-        end
-
-        super(execution_mode: execution_mode, max_threads: max_threads, poll_interval: poll_interval, scheduler: scheduler)
+      def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
+        configuration = GoodJob::Configuration.new({ execution_mode: execution_mode }, env: ENV)
+        super(execution_mode: configuration.rails_execution_mode, max_threads: max_threads, poll_interval: poll_interval, scheduler: scheduler, inline: inline)
       end
     end
   end

data/lib/good_job.rb

@@ -1,10 +1,12 @@
 require "rails"
 require 'good_job/railtie'
 
+require 'good_job/configuration'
 require 'good_job/log_subscriber'
 require 'good_job/lockable'
 require 'good_job/job'
 require 'good_job/scheduler'
+require 'good_job/multi_scheduler'
 require 'good_job/adapter'
 require 'good_job/pg_locks'
 require 'good_job/performer'

data/lib/good_job/adapter.rb

@@ -3,28 +3,24 @@ module GoodJob
     EXECUTION_MODES = [:async, :external, :inline].freeze
 
     def initialize(execution_mode: nil, max_threads: nil, poll_interval: nil, scheduler: nil, inline: false)
-      if inline
+      if inline && execution_mode.nil?
         ActiveSupport::Deprecation.warn('GoodJob::Adapter#new(inline: true) is deprecated; use GoodJob::Adapter.new(execution_mode: :inline) instead')
-        @execution_mode = :inline
-      elsif execution_mode
-        raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(execution_mode)
-
-        @execution_mode = execution_mode
-      else
-        @execution_mode = :external
+        execution_mode = :inline
       end
 
-      @scheduler = scheduler
-      if @execution_mode == :async && @scheduler.blank? # rubocop:disable Style/GuardClause
-        timer_options = {}
-        timer_options[:execution_interval] = poll_interval if poll_interval.present?
+      configuration = GoodJob::Configuration.new({
+                                                   execution_mode: execution_mode,
+                                                   max_threads: max_threads,
+                                                   poll_interval: poll_interval,
+                                                 },
+                                                 env: ENV)
 
-        pool_options = {}
-        pool_options[:max_threads] = max_threads if max_threads.present?
+      raise ArgumentError, "execution_mode: must be one of #{EXECUTION_MODES.join(', ')}." unless EXECUTION_MODES.include?(configuration.execution_mode)
 
-        job_performer = GoodJob::Performer.new(GoodJob::Job, :perform_with_advisory_lock, name: '*')
-        @scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
-      end
+      @execution_mode = configuration.execution_mode
+
+      @scheduler = scheduler
+      @scheduler = GoodJob::Scheduler.from_configuration(configuration) if @execution_mode == :async && @scheduler.blank?
     end
 
     def enqueue(active_job)

data/lib/good_job/cli.rb

@@ -10,43 +10,16 @@ module GoodJob
                   desc: "Maximum number of threads to use for working jobs (default: ActiveRecord::Base.connection_pool.size)"
     method_option :queues,
                   type: :string,
-                  banner: "queue1,queue2",
-                  desc: "Queues to work from. Separate multiple queues with commas (default: *)"
+                  banner: "queue1,queue2(;queue3,queue4:5)",
+                  desc: "Queues to work from. Separate multiple queues with commas; separate isolated execution pools with semicolons and threads with colons (default: *)"
     method_option :poll_interval,
                   type: :numeric,
                   desc: "Interval between polls for available jobs in seconds (default: 1)"
     def start
       set_up_application!
 
-      max_threads = (
-        options[:max_threads] ||
-        ENV['GOOD_JOB_MAX_THREADS'] ||
-        ENV['RAILS_MAX_THREADS'] ||
-        ActiveRecord::Base.connection_pool.size
-      ).to_i
-
-      queue_string = (
-        options[:queues] ||
-        ENV['GOOD_JOB_QUEUES'] ||
-        '*'
-      )
-
-      poll_interval = (
-        options[:poll_interval] ||
-        ENV['GOOD_JOB_POLL_INTERVAL']
-      ).to_i
-
-      job_query = GoodJob::Job.queue_string(queue_string)
-      job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
-
-      timer_options = {}
-      timer_options[:execution_interval] = poll_interval if poll_interval.positive?
-
-      pool_options = {
-        max_threads: max_threads,
-      }
-
-      scheduler = GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
+      configuration = Configuration.new(options, env: ENV)
+      scheduler = Scheduler.from_configuration(configuration)
 
       @stop_good_job_executable = false
       %w[INT TERM].each do |signal|

data/lib/good_job/configuration.rb

@@ -0,0 +1,55 @@
+module GoodJob
+  class Configuration
+    attr_reader :options, :env
+
+    def initialize(options, env: ENV)
+      @options = options
+      @env = env
+    end
+
+    def execution_mode(default: :external)
+      if options[:execution_mode]
+        options[:execution_mode]
+      elsif env['GOOD_JOB_EXECUTION_MODE'].present?
+        env['GOOD_JOB_EXECUTION_MODE'].to_sym
+      else
+        default
+      end
+    end
+
+    def rails_execution_mode
+      if execution_mode(default: nil)
+        execution_mode
+      elsif Rails.env.development?
+        :inline
+      elsif Rails.env.test?
+        :inline
+      else
+        :external
+      end
+    end
+
+    def max_threads
+      (
+        options[:max_threads] ||
+        env['GOOD_JOB_MAX_THREADS'] ||
+        env['RAILS_MAX_THREADS'] ||
+        ActiveRecord::Base.connection_pool.size
+      ).to_i
+    end
+
+    def queue_string
+      options[:queues] ||
+        env['GOOD_JOB_QUEUES'] ||
+        '*'
+    end
+
+    def poll_interval
+      (
+        options[:poll_interval] ||
+        env['GOOD_JOB_POLL_INTERVAL'] ||
+        1
+      ).to_i
+    end
+  end
+end

data/lib/good_job/log_subscriber.rb

@@ -96,7 +96,7 @@ module GoodJob
     def info_and_stdout(progname = nil, tags: [], &block)
       unless ActiveSupport::Logger.logger_outputs_to?(logger, STDOUT)
         tags_string = (['GoodJob'] + tags).map { |t| "[#{t}]" }.join(' ')
-        stdout_message = "#{tags_string}#{yield}"
+        stdout_message = "#{tags_string} #{yield}"
         $stdout.puts stdout_message
       end
 

data/lib/good_job/multi_scheduler.rb

@@ -0,0 +1,25 @@
+module GoodJob
+  class MultiScheduler
+    attr_reader :schedulers
+
+    def initialize(schedulers)
+      @schedulers = schedulers
+    end
+
+    def shutdown(wait: true)
+      schedulers.each { |s| s.shutdown(wait: wait) }
+    end
+
+    def shutdown?
+      schedulers.all?(&:shutdown?)
+    end
+
+    def restart(wait: true)
+      schedulers.each { |s| s.restart(wait: wait) }
+    end
+
+    def create_thread
+      schedulers.all?(&:create_thread)
+    end
+  end
+end

data/lib/good_job/scheduler.rb

@@ -22,6 +22,31 @@ module GoodJob
 
     cattr_reader :instances, default: [], instance_reader: false
 
+    def self.from_configuration(configuration)
+      schedulers = configuration.queue_string.split(';').map do |queue_string_and_max_threads|
+        queue_string, max_threads = queue_string_and_max_threads.split(':')
+        max_threads = (max_threads || configuration.max_threads).to_i
+
+        job_query = GoodJob::Job.queue_string(queue_string)
+        job_performer = GoodJob::Performer.new(job_query, :perform_with_advisory_lock, name: queue_string)
+
+        timer_options = {}
+        timer_options[:execution_interval] = configuration.poll_interval if configuration.poll_interval.positive?
+
+        pool_options = {
+          max_threads: max_threads,
+        }
+
+        GoodJob::Scheduler.new(job_performer, timer_options: timer_options, pool_options: pool_options)
+      end
+
+      if schedulers.size > 1
+        GoodJob::MultiScheduler.new(schedulers)
+      else
+        schedulers.first
+      end
+    end
+
     def initialize(performer, timer_options: {}, pool_options: {})
       raise ArgumentError, "Performer argument must implement #next" unless performer.respond_to?(:next)
 

data/lib/good_job/version.rb

@@ -1,3 +1,3 @@
 module GoodJob
-  VERSION = '1.1.0'.freeze
+  VERSION = '1.1.1'.freeze
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: good_job
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.1.1
 platform: ruby
 authors:
 - Ben Sheldon
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2020-08-10 00:00:00.000000000 Z
+date: 2020-08-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: concurrent-ruby
@@ -269,9 +269,11 @@ files:
 - lib/good_job.rb
 - lib/good_job/adapter.rb
 - lib/good_job/cli.rb
+- lib/good_job/configuration.rb
 - lib/good_job/job.rb
 - lib/good_job/lockable.rb
 - lib/good_job/log_subscriber.rb
+- lib/good_job/multi_scheduler.rb
 - lib/good_job/performer.rb
 - lib/good_job/pg_locks.rb
 - lib/good_job/railtie.rb