skiplock 1.0.10 → 1.0.14

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 681b41b73f6ddf95620515f3d83454ba1f623d57740248bf84041829c225318d
-  data.tar.gz: 0433b2e635174ae052b67d7502e4b2477c54736f79ec0acf0ad369a1be976256
+  metadata.gz: 55c98833e2792a216b6389d399e992aae9660f23ea076fcf852fb164cbc32d55
+  data.tar.gz: 43c8c2f90ee55ec423242042d1ab2986c0cb9f70fcc04db56aaf2c566bbb0382
 SHA512:
-  metadata.gz: 6860b6cff70a8881277fd0ce7d2d76d211723e8c1974a1e2b70db7ea31b05268af6d38b68f2437cc9144d0d896751de205e21890219d3664b0f5ae128e1c6ab0
-  data.tar.gz: ddf11fbb1b030dc437254f87142127fd135bacbb84301d03fbcb14e3029bc05736e9c02b1d08b6be8f2d6b90e3fbfdf8eff568c63c8d8c9b451c730a1a125b39
+  metadata.gz: 873492b815e6b5fc653630fed3bc64c5a363339551bbad304ecbad9453ea3738c24f3f0b874f4d6fe47cd4cc0ea5d46fb61b4c0dc322814d08dbc11002c24443
+  data.tar.gz: e84f7081e07ba0aff19834251de09e08df573d45ff159299fe08baf621b76845cd2fdf0461229da77b07cf707f151e7d8fc00d97b19a9bf30f1129376c2dd053

data/README.md

@@ -50,12 +50,13 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     ```yaml
     # config/skiplock.yml (default settings)
     ---
-    extensions: false
-    logging: true
     min_threads: 1
     max_threads: 5
     max_retries: 20
-    notification: none
+    logfile: log/skiplock.log
+    loglevel: info
+    notification: custom
+    extensions: false
     purge_completion: true
     queues:
       default: 200
@@ -63,12 +64,13 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     workers: 0
     ```
     Available configuration options are:
-    - **extensions** (*boolean*): enable or disable the class method extension.  See `ClassMethod extension` for more details
-    - **logging** (*boolean*): enable or disable file logging capability; the log file is stored at log/skiplock.log
     - **min_threads** (*integer*): sets minimum number of threads staying idle
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
-    - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired.  See `Retry System` for more details
-    - **notification** (*enumeration*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, none`).  Using `auto` will attempt to detect available gems in the application
+    - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired.  See `Retry system` for more details
+    - **logfile** (*string*): path filename for skiplock logs; empty logfile will disable logging
+    - **loglevel** (*string*): sets logging level (`debug, info, warn, error, fatal, unknown`)
+    - **notification** (*string*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, custom`); using `auto` will detect library if available.  See `Notification system` for more details
+    - **extensions** (*boolean*): enable or disable the class method extension.  See `ClassMethod extension` for more details
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
     - **queues** (*hash*): defines the set of queues with priorities; lower priority takes precedence
     - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
@@ -82,7 +84,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     $ bundle exec skiplock -h
     Usage: skiplock [options]
       -e, --environment STRING         Rails environment
-      -l, --logging STRING             Possible values: true, false, timestamp
+      -l, --logfile STRING             Full path to logfile
       -s, --graceful-shutdown NUM      Number of seconds to wait for graceful shutdown
       -r, --max-retries NUM            Number of maxixum retries
       -t, --max-threads NUM            Number of maximum threads
@@ -158,17 +160,17 @@ If the retry attempt limit configured in ActiveJob has been reached, then the co
 If the `retry_on` block is not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
 
 ## Notification system
-`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification` as shown in the **Configuration** section above.  Custom function can also be called whenever an exception occurs; it can be configured in an initializer like below:
+`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  Custom notification can also be called whenever an exception occurs; it can be configured in an initializer like below:
 ```ruby
   # config/initializers/skiplock.rb
   Skiplock.on_error do |ex, previous|
     if ex.backtrace != previous.try(:backtrace)
       # sends custom email on new exceptions only
       # the same repeated exceptions will only be sent once to avoid SPAM
-      # NOTE: exceptions generated from Job perform method executions will not provide 'previous' exceptions
+      # NOTE: exceptions generated from Job executions will not provide 'previous' exceptions
     end
   end
-  # supports multiple on_error callbacks
+  # supports multiple 'on_error' event callbacks
 ```
 ## ClassMethod extension
 `Skiplock` can add extension to allow all class methods to be performed as a background job; it is disabled in the default configuration.  To enable, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`.

data/bin/skiplock

@@ -5,7 +5,7 @@ begin
   op = OptionParser.new do |opts|
     opts.banner = "Usage: #{File.basename($0)} [options]"
     opts.on('-e', '--environment STRING', String, 'Rails environment')
-    opts.on('-l', '--logging STRING', String, 'Possible values: true, false, timestamp')
+    opts.on('-l', '--logfile STRING', String, 'Full path to logfile')
     opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
     opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
     opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')
@@ -25,4 +25,4 @@ options.transform_keys! { |k| k.to_s.gsub('-', '_').to_sym }
 env = options.delete(:environment)
 ENV['RAILS_ENV'] = env if env
 require File.expand_path("config/environment.rb")
-Skiplock::Manager.new(options.merge(standalone: true))
+Rails.application.config.skiplock.standalone(**options.merge(standalone: true))

data/lib/active_job/queue_adapters/skiplock_adapter.rb

@@ -2,7 +2,7 @@ module ActiveJob
   module QueueAdapters
     class SkiplockAdapter
       def initialize
-        Rails.application.config.after_initialize { Skiplock::Manager.new }
+        Rails.application.config.after_initialize { Rails.application.config.skiplock = Skiplock::Manager.new }
       end
 
       def enqueue(job)

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -60,7 +60,7 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
           ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
           END IF;
-          PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
+          PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.job_class,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
           RETURN NULL;
         END;
       $$ LANGUAGE plpgsql

data/lib/skiplock/cron.rb

@@ -2,6 +2,7 @@ require 'cron_parser'
 module Skiplock
   class Cron
     def self.setup
+      Rails.application.eager_load! if Rails.env.development?
       cronjobs = []
       ActiveJob::Base.descendants.each do |j|
         next unless j.const_defined?('CRON')

data/lib/skiplock/job.rb

@@ -2,35 +2,14 @@ module Skiplock
   class Job < ActiveRecord::Base
     self.implicit_order_column = 'created_at'
 
-    def self.dispatch(queues_order_query: nil, worker_id: nil, purge_completion: true, max_retries: 20)
+    def self.dispatch(worker_id: nil, purge_completion: true, max_retries: 20)
       job = nil
-      self.transaction do
-        job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
-        return (job ? job.scheduled_at.to_f : Float::INFINITY) if job.nil? || job.scheduled_at.to_f > Time.now.to_f
-        job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
+      self.connection.transaction do
+        job = self.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+        return if job.nil? || job.scheduled_at.to_f > Time.now.to_f
+        job = self.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
       end
-      job.data ||= {}
-      job.exception_executions ||= {}
-      job_data = job.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => job.id, 'enqueued_at' => job.updated_at, 'arguments' => (job.data['arguments'] || []))
-      job.executions = (job.executions || 0) + 1
-      Skiplock.logger.info "[Skiplock] Performing #{job.job_class} (#{job.id}) from queue '#{job.queue_name || 'default'}'..."
-      Thread.current[:skiplock_dispatch_job] = job
-      start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      begin
-        ActiveJob::Base.execute(job_data)
-      rescue Exception => ex
-        Skiplock.logger.error(ex)
-      end
-      end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      job_name = job.job_class
-      if job.job_class == 'Skiplock::Extension::ProxyJob'
-        target, method_name = ::YAML.load(job.data['arguments'].first)
-        job_name = "'#{target.name}.#{method_name}'"
-      end
-      Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{job.id}) from queue '#{job.queue_name || 'default'}' in #{end_time - start_time} seconds"
-      job.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
-    ensure
-      Thread.current[:skiplock_dispatch_job] = nil
+      self.dispatch(worker_id: worker_id, purge_completion: purge_completion, max_retries: max_retries) if job.execute(purge_completion: purge_completion, max_retries: max_retries)
     end
 
     def self.enqueue(activejob)
@@ -46,7 +25,7 @@ module Skiplock
         Thread.current[:skiplock_dispatch_job]
       else
         serialize = activejob.serialize
-        Job.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'] }, 'scheduled_at' => timestamp))
+        self.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'] }, 'scheduled_at' => timestamp))
       end
     end
 
@@ -55,8 +34,9 @@ module Skiplock
     end
 
     def dispose(ex, purge_completion: true, max_retries: 20)
-      dup = self.dup
+      yaml = [self, ex].to_yaml
       self.running = false
+      self.worker_id = nil
       self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
       if ex
         self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
@@ -80,6 +60,7 @@ module Skiplock
           self.scheduled_at = Time.at(next_cron_at)
           self.save!
         else
+          Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
           self.delete
         end
       elsif purge_completion
@@ -90,9 +71,46 @@ module Skiplock
         self.save!
       end
       self
-    rescue
-      File.write("tmp/skiplock/#{self.id}", [dup, ex].to_yaml)
+    rescue Exception => e
+      if Skiplock.logger
+        Skiplock.logger.error(e.to_s)
+        Skiplock.logger.error(e.backtrace.join("\n"))
+        File.write("tmp/skiplock/#{self.id}", yaml)
+      end
       nil
     end
+
+    def execute(purge_completion: true, max_retries: 20)
+      Skiplock.logger.info("[Skiplock] Performing #{self.job_class} (#{self.id}) from queue '#{self.queue_name || 'default'}'...") if Skiplock.logger
+      self.data ||= {}
+      self.exception_executions ||= {}
+      job_data = self.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => self.id, 'enqueued_at' => self.updated_at, 'arguments' => (self.data['arguments'] || []))
+      self.executions = (self.executions || 0) + 1
+      Thread.current[:skiplock_dispatch_job] = self
+      activejob = ActiveJob::Base.deserialize(job_data)
+      start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      begin
+        activejob.perform_now
+      rescue Exception => ex
+      end
+      if Skiplock.logger
+        if ex || self.exception_executions.key?('activejob_retry')
+          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.exception_executions.key?('activejob_retry') }")
+          if ex
+            Skiplock.logger.error(ex.to_s)
+            Skiplock.logger.error(ex.backtrace.join("\n"))
+          end
+        else
+          end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          job_name = self.job_class
+          if self.job_class == 'Skiplock::Extension::ProxyJob'
+            target, method_name = ::YAML.load(self.data['arguments'].first)
+            job_name = "'#{target.name}.#{method_name}'"
+          end
+          Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{self.id}) from queue '#{self.queue_name || 'default'}' in #{end_time - start_time} seconds"
+        end
+      end
+      self.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
+    end
   end
 end

data/lib/skiplock/manager.rb

@@ -6,46 +6,76 @@ module Skiplock
       @config.symbolize_keys!
       @config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
       @config.merge!(config)
-      Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
-      return unless @config[:standalone] || (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
       @config[:hostname] = `hostname -f`.strip
-      do_config
+      configure
+      Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
+      if (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
+        cleanup_workers
+        @worker = create_worker
+        @worker.start(**@config)
+        at_exit { @worker.shutdown }
+      end
+    rescue Exception => ex
+      @logger.error(ex.to_s)
+      @logger.error(ex.backtrace.join("\n"))
+    end
+
+    def standalone(**options)
+      @config.merge!(options)
+      Rails.logger.reopen('/dev/null')
+      Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
+      @config[:workers] = 1 if @config[:workers] <= 0
       banner
       cleanup_workers
-      create_worker
-      ActiveJob::Base.logger = nil
-      if @config[:standalone]
-        standalone
-      else
-        dispatcher = Dispatcher.new(worker: @worker, **@config)
-        thread = dispatcher.run
-        at_exit do
-          dispatcher.shutdown
-          thread.join(@config[:graceful_shutdown])
-          @worker.delete
+      @worker = create_worker
+      @parent_id = Process.pid
+      @shutdown = false
+      Signal.trap("INT") { @shutdown = true }
+      Signal.trap("TERM") { @shutdown = true }
+      (@config[:workers] - 1).times do |n|
+        fork do
+          sleep 1
+          worker = create_worker(master: false)
+          worker.start(worker_num: n + 1, **@config)
+          loop do
+            sleep 0.5
+            break if @shutdown || Process.ppid != @parent_id
+          end
+          worker.shutdown
         end
       end
+      @worker.start(**@config)
+      loop do
+        sleep 0.5
+        break if @shutdown
+      end
+      @logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
+      Process.waitall
+      @worker.shutdown
+      @logger.info "[Skiplock] Shutdown completed."
     end
 
-  private
+    private
 
     def banner
-      title = "[Skiplock] V#{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
+      title = "Skiplock #{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
       @logger.info "-"*(title.length)
       @logger.info title
       @logger.info "-"*(title.length)
-      @logger.info "ClassMethod Extensions: #{@config[:extensions]}"
+      @logger.info "ClassMethod extensions: #{@config[:extensions]}"
       @logger.info "      Purge completion: #{@config[:purge_completion]}"
-      @logger.info "          Notification: #{@config[:notification]}#{(' (' + @notification + ')') if @config[:notification] == 'auto'}"
+      @logger.info "          Notification: #{@config[:notification]}"
       @logger.info "           Max retries: #{@config[:max_retries]}"
       @logger.info "           Min threads: #{@config[:min_threads]}"
       @logger.info "           Max threads: #{@config[:max_threads]}"
       @logger.info "           Environment: #{Rails.env}"
-      @logger.info "               Logging: #{@config[:logging]}"
+      @logger.info "              Loglevel: #{@config[:loglevel]}"
+      @logger.info "               Logfile: #{@config[:logfile] || '(disabled)'}"
       @logger.info "               Workers: #{@config[:workers]}"
       @logger.info "                Queues: #{@config[:queues].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if @config[:queues].is_a?(Hash)
       @logger.info "                   PID: #{Process.pid}"
       @logger.info "-"*(title.length)
+      @logger.warn "[Skiplock] Custom notification has no registered 'on_error' callback" if Skiplock.on_errors.count == 0
     end
 
     def cleanup_workers
@@ -54,19 +84,18 @@ module Skiplock
         sid = Process.getsid(worker.pid) rescue nil
         delete_ids << worker.id if worker.sid != sid || worker.updated_at < 30.minutes.ago
       end
-      if delete_ids.count > 0
-        Job.where(running: true, worker_id: delete_ids).update_all(running: false, worker_id: nil)
-        Worker.where(id: delete_ids).delete_all
-      end
+      Worker.where(id: delete_ids).delete_all if delete_ids.count > 0
+      Job.where(running: true).where.not(worker_id: Worker.ids).update_all(running: false, worker_id: nil)
     end
 
-    def create_worker(pid: Process.pid, sid: Process.getsid(), master: true)
-      @worker = Worker.create!(pid: pid, sid: sid, master: master, hostname: @config[:hostname], capacity: @config[:max_threads])
+    def create_worker(master: true)
+      Worker.create!(pid: Process.pid, sid: Process.getsid(), master: master, hostname: @config[:hostname], capacity: @config[:max_threads])
     rescue
-      @worker = Worker.create!(pid: pid, sid: sid, master: false, hostname: @config[:hostname], capacity: @config[:max_threads])
+      Worker.create!(pid: Process.pid, sid: Process.getsid(), master: false, hostname: @config[:hostname], capacity: @config[:max_threads])
     end
 
-    def do_config
+    def configure
+      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
       @config[:graceful_shutdown] = 300 if @config[:graceful_shutdown] > 300
       @config[:graceful_shutdown] = nil if @config[:graceful_shutdown] < 0
       @config[:max_retries] = 20 if @config[:max_retries] > 20
@@ -75,22 +104,28 @@ module Skiplock
       @config[:max_threads] = 20 if @config[:max_threads] > 20
       @config[:min_threads] = 0 if @config[:min_threads] < 0
       @config[:workers] = 0 if @config[:workers] < 0
-      @config[:workers] = 1 if @config[:standalone] && @config[:workers] <= 0
+      @logger = ActiveSupport::Logger.new(STDOUT)
+      @logger.level = @config[:loglevel].to_sym
+      Skiplock.logger = @logger
+      raise "Cannot create logfile '#{@config[:logfile]}'" if @config[:logfile] && !File.writable?(File.dirname(@config[:logfile]))
+      @config[:logfile] = nil if @config[:logfile].to_s.length == 0
+      if @config[:logfile]
+        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(@config[:logfile])))
+        ActiveJob::Base.logger = nil
+      end
       @config[:queues].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if @config[:queues].is_a?(Hash)
-      @notification = @config[:notification]
-      if @notification == 'auto'
+      if @config[:notification] == 'auto'
         if defined?(Airbrake)
-          @notification = 'airbrake'
+          @config[:notification] = 'airbrake'
         elsif defined?(Bugsnag)
-          @notification = 'bugsnag'
+          @config[:notification] = 'bugsnag'
         elsif defined?(ExceptionNotifier)
-          @notification = 'exception_notification'
+          @config[:notification] = 'exception_notification'
         else
-          @logger.info "Unable to detect any known exception notification gem. Please define custom 'on_error' callback function and disable 'auto' notification in 'config/skiplock.yml'"
-          exit
+          raise "Unable to detect any known exception notification library. Please define custom 'on_error' event callbacks and change to 'custom' notification in 'config/skiplock.yml'"
         end
       end
-      case @notification
+      case @config[:notification]
       when 'airbrake'
         raise 'airbrake gem not found' unless defined?(Airbrake)
         Skiplock.on_error do |ex, previous|
@@ -106,53 +141,10 @@ module Skiplock
         Skiplock.on_error do |ex, previous|
           ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace)
         end
-      end
-      Skiplock.logger = ActiveSupport::Logger.new(STDOUT)
-      Skiplock.logger.level = Rails.logger.level
-      @logger = Skiplock.logger
-      if @config[:logging]
-        Skiplock.logger.extend(ActiveSupport::Logger.broadcast(::Logger.new('log/skiplock.log')))
-        if @config[:standalone]
-          Rails.logger.reopen('/dev/null')
-          Rails.logger.extend(ActiveSupport::Logger.broadcast(Skiplock.logger))
-        end
+      else
+        @config[:notification] = 'custom'
       end
       Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
     end
-
-    def standalone
-      parent_id = Process.pid
-      shutdown = false
-      Signal.trap("INT") { shutdown = true }
-      Signal.trap("TERM") { shutdown = true }
-      (@config[:workers] - 1).times do |n|
-        fork do
-          sleep 1
-          worker = create_worker(master: false)
-          dispatcher = Dispatcher.new(worker: worker, worker_num: n + 1, **@config)
-          thread = dispatcher.run
-          loop do
-            sleep 0.5
-            break if shutdown || Process.ppid != parent_id
-          end
-          dispatcher.shutdown
-          thread.join(@config[:graceful_shutdown])
-          worker.delete
-          exit
-        end
-      end
-      dispatcher = Dispatcher.new(worker: @worker, **@config)
-      thread = dispatcher.run
-      loop do
-        sleep 0.5
-        break if shutdown
-      end
-      @logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
-      Process.waitall
-      dispatcher.shutdown
-      thread.join(@config[:graceful_shutdown])
-      @worker.delete
-      @logger.info "[Skiplock] Shutdown completed."
-    end
   end
 end

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.10'
+  VERSION = Version = '1.0.14'
 end
 

data/lib/skiplock/worker.rb

@@ -1,5 +1,121 @@
 module Skiplock
   class Worker < ActiveRecord::Base
     self.implicit_order_column = 'created_at'
+
+    def start(worker_num: 0, **config)
+      @config = config
+      @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
+      @next_schedule_at = Time.now.to_f
+      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
+      @running = true
+      Process.setproctitle("skiplock-#{self.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
+      @executor.post do
+        Rails.application.reloader.wrap { run }
+      end
+    end
+
+    def shutdown
+      @running = false
+      @executor.shutdown
+      @executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
+      self.delete
+    end
+
+    private
+
+    def check_sync_errors
+      # get executed jobs that could not sync with database
+      Dir.glob('tmp/skiplock/*').each do |f|
+        job_from_db = Job.find_by(id: File.basename(f), running: true)
+        disposed = true
+        if job_from_db
+          job, ex = YAML.load_file(f) rescue nil
+          disposed = job.dispose(ex, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) if job
+        end
+        File.delete(f) if disposed
+      end
+    end
+
+    def get_next_available_job
+      @connection.transaction do
+        job = Job.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+        if job && job.scheduled_at.to_f <= Time.now.to_f
+          job = Job.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
+        end
+        job
+      end
+    end
+
+    def run
+      @connection = self.class.connection
+      @connection.exec_query('LISTEN "skiplock::jobs"')
+      if self.master
+        Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
+        check_sync_errors
+        Cron.setup
+      end
+      error = false
+      timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      while @running
+        Rails.application.reloader.wrap do
+          begin
+            if error
+              unless @connection.active?
+                @connection.reconnect!
+                sleep(0.5)
+                @connection.exec_query('LISTEN "skiplock::jobs"')
+                @next_schedule_at = Time.now.to_f
+              end
+              check_sync_errors if self.master
+              error = false
+            end
+            if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
+              job = get_next_available_job
+              if job.try(:running)
+                @executor.post do
+                  Rails.application.executor.wrap do
+                    ActiveSupport::Dependencies.interlock.permit_concurrent_loads { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) }
+                  end
+                end
+              else
+                @next_schedule_at = (job ? job.scheduled_at.to_f : Float::INFINITY)
+              end
+            end
+            job_notifications = []
+            @connection.raw_connection.wait_for_notify(0.4) do |channel, pid, payload|
+              job_notifications << payload if payload
+              loop do
+                payload = @connection.raw_connection.notifies
+                break unless @running && payload
+                job_notifications << payload[:extra]
+              end
+              job_notifications.each do |n|
+                op, id, worker_id, job_class, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
+                next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0
+                @next_schedule_at = scheduled_at.to_f if scheduled_at.to_f < @next_schedule_at
+              end
+            end
+            if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
+              self.touch
+              timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            end
+          rescue Exception => ex
+            # most likely error with database connection
+            Skiplock.logger.error(ex.to_s)
+            Skiplock.logger.error(ex.backtrace.join("\n"))
+            Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
+            error = true
+            t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            while @running
+              sleep(0.5)
+              break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > 5
+            end
+            @last_exception = ex
+          end
+          sleep(0.3)
+        end
+      end
+      @connection.exec_query('UNLISTEN *')
+    end
   end
 end

data/lib/skiplock.rb

@@ -3,7 +3,6 @@ require 'active_job/queue_adapters/skiplock_adapter'
 require 'active_record'
 require 'skiplock/counter'
 require 'skiplock/cron'
-require 'skiplock/dispatcher'
 require 'skiplock/extension'
 require 'skiplock/job'
 require 'skiplock/manager'
@@ -11,7 +10,7 @@ require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
-  DEFAULT_CONFIG = { 'extensions' => false, 'logging' => true, 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'none', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
+  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'log/skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
 
   def self.logger=(l)
     @logger = l
@@ -28,7 +27,7 @@ module Skiplock
   end
 
   def self.on_errors
-    @on_errors || [].freeze
+    @on_errors || []
   end
 
   def self.table_name_prefix

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.10
+  version: 1.0.14
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-08-29 00:00:00.000000000 Z
+date: 2021-09-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -84,7 +84,6 @@ files:
 - lib/skiplock.rb
 - lib/skiplock/counter.rb
 - lib/skiplock/cron.rb
-- lib/skiplock/dispatcher.rb
 - lib/skiplock/extension.rb
 - lib/skiplock/job.rb
 - lib/skiplock/manager.rb

data/lib/skiplock/dispatcher.rb

@@ -1,117 +0,0 @@
-module Skiplock
-  class Dispatcher
-    def initialize(worker:, worker_num: nil, **config)
-      @config = config
-      @worker = worker
-      @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
-      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads], max_threads: @config[:max_threads], max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
-      @last_dispatch_at = 0
-      @next_schedule_at = Time.now.to_f
-      Process.setproctitle("skiplock-#{@worker.master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if @config[:standalone]
-    end
-    
-    def run
-      @running = true
-      Thread.new do
-        ActiveRecord::Base.connection_pool.with_connection do |connection|
-          connection.exec_query('LISTEN "skiplock::jobs"')
-          if @worker.master
-            Rails.application.eager_load! if Rails.env.development?
-            Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
-            check_sync_errors
-            Cron.setup
-          end
-          error = false
-          timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-          while @running
-            begin
-              if error
-                unless connection.active?
-                  connection.reconnect!
-                  sleep(0.5)
-                  connection.exec_query('LISTEN "skiplock::jobs"')
-                  @next_schedule_at = Time.now.to_f
-                end
-                check_sync_errors
-                error = false
-              end
-              job_notifications = []
-              connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
-                job_notifications << payload if payload
-                loop do
-                  payload = connection.raw_connection.notifies
-                  break unless @running && payload
-                  job_notifications << payload[:extra]
-                end
-                job_notifications.each do |n|
-                  op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
-                  next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0  || scheduled_at.to_f < @last_dispatch_at
-                  if scheduled_at.to_f <= Time.now.to_f
-                    @next_schedule_at = Time.now.to_f
-                  elsif scheduled_at.to_f < @next_schedule_at
-                    @next_schedule_at = scheduled_at.to_f
-                  end
-                end
-              end
-              if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
-                @executor.post { do_work }
-              end
-              if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
-                @worker.touch
-                timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-              end
-            rescue Exception => ex
-              # most likely error with database connection
-              Skiplock.logger.error(ex)
-              Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
-              error = true
-              t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-              while @running
-                sleep(0.5)
-                break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > 5
-              end
-              @last_exception = ex
-            end
-            sleep(0.2)
-          end
-          connection.exec_query('UNLISTEN *')
-          @executor.shutdown
-          @executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
-        end
-      end
-    end
-
-    def shutdown
-      @running = false
-    end
-
-    private
-
-    def check_sync_errors
-      # get performed jobs that could not sync with database
-      Dir.glob('tmp/skiplock/*').each do |f|
-        job_from_db = Job.find_by(id: File.basename(f), running: true)
-        disposed = true
-        if job_from_db
-          job, ex = YAML.load_file(f) rescue nil
-          disposed = job.dispose(ex, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
-        end
-        File.delete(f) if disposed
-      end
-    end
-
-    def do_work
-      while @running
-        @last_dispatch_at = Time.now.to_f - 1  # 1 second allowance for time drift
-        result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
-        next if result.is_a?(Job) && Time.now.to_f >= @next_schedule_at
-        @next_schedule_at = result if result.is_a?(Float)
-        break
-      end
-    rescue Exception => ex
-      Skiplock.logger.error(ex)
-      Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
-      @last_exception = ex
-    end
-  end
-end