skiplock 1.0.11 → 1.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 432874dc801864f01a8c4de896f330ec62cecdb43d71ededc622ece7d9a17402
-  data.tar.gz: 6034dcb3cfa194b186465a5a84227899c8f00fdf4d62d5b20e0370c9c122cde5
+  metadata.gz: 5c965cac0fea118b7a8295bd82999bd33cd2807829dee639b6b782279a27697d
+  data.tar.gz: f5357c225546ef4fcec74a10b3180d07950c7f7f3308a39b2fe19c5668677a0f
 SHA512:
-  metadata.gz: 89f0ef53c0740cf5522e20aa1473ed9020bf5d116054acbb77f451472c1d9af2b123b47cdc593efb4545383829609537a4e2395b4c23c7482f04cff04b11ceab
-  data.tar.gz: 640fa077844855e312b6018d2858cb495915fba0f167c711540ff3b713c66dfc5aa76ca283d9dffaf3d778f37c7025a1f3f7ee3728d4120d87bc25079717bc60
+  metadata.gz: d42f8d156e11c25d117a246f30cee23e7c487ecfa80e74e0a10866013f2a6145afd42bb55e3f057ff1687f686ee4e030f851d88fd3c2d3a99b33df745c4617f0
+  data.tar.gz: 981e7adad4bae9281e40bda3e049f68b32458078b98f192f5a9721ba31b40d82347c19a4da1876b210826a066f49f3ff0f2b8c966e9708b2d5b7d7bb890ff8b4

data/README.md

@@ -4,7 +4,7 @@
 
 It only uses the `LISTEN/NOTIFY/SKIP LOCKED` features provided natively on PostgreSQL 9.5+ to efficiently and reliably dispatch jobs to worker processes and threads ensuring that each job can be completed successfully **only once**.  No other polling or timer is needed.
 
-The library is quite small compared to other PostgreSQL job queues (eg. *delay_job*, *queue_classic*, *que*, *good_job*) with less than 400 lines of codes; and it still provides similar set of features and more...
+The library is quite small compared to other PostgreSQL job queues (eg. *delay_job*, *queue_classic*, *que*, *good_job*) with less than 500 lines of codes; and it still provides similar set of features and more...
 
 #### Compatibility:
 
@@ -54,6 +54,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     max_threads: 5
     max_retries: 20
     logfile: log/skiplock.log
+    loglevel: info
     notification: custom
     extensions: false
     purge_completion: true
@@ -67,6 +68,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
     - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired.  See `Retry system` for more details
     - **logfile** (*string*): path filename for skiplock logs; empty logfile will disable logging
+    - **loglevel** (*string*): sets logging level (`debug, info, warn, error, fatal, unknown`)
     - **notification** (*string*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, custom`); using `auto` will detect library if available.  See `Notification system` for more details
     - **extensions** (*boolean*): enable or disable the class method extension.  See `ClassMethod extension` for more details
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
@@ -185,6 +187,37 @@ If the `retry_on` block is not defined, then the built-in retry system of `skipl
   Subscription.skiplock(wait_until: Date.tomorrow.noon).charge(amount: 100)
   ```
 
+## Fault tolerant
+`Skiplock` ensures that jobs will be executed sucessfully only once even if database connection is lost during or after the job was dispatched.  Successful jobs are marked as completed or removed (with `purge_completion` turned on), and failed or interrupted jobs are marked for retry; however, when the database connection is dropped for any reasons and the commit is lost, `Skiplock` will then save the commit data to local disk (as `tmp/skiplock/<job_id>`) and synchronize with the database when the connection resumes.  This also protects in-progress jobs that were terminated abruptly during a graceful shutdown with timeout; they will be queued for retry.
+
+## Scalability
+`Skiplock` can scale both vertically and horizontally.  To scale vertically, simply increase the number of `Skiplock` workers per host.  To scale horizontally, simply deploy `Skiplock` to multiple hosts sharing the same PostgreSQL database.
+
+## Statistics and counters
+The `skiplock.workers` database table contains all the `Skiplock` workers running on all the hosts.  Active worker will update its timestamp column (`updated_at`) every minute; and dispatched jobs would be associated with the running workers.  At any given time, a list of active workers running a list of jobs can be determined using the database table.
+
+The `skiplock.counters` database table contains all historical job dispatches, completions, expiries, failures and retries.  The counters are recorded by dates; so it's possible to get statistical data for any given day or range of dates.
+
+  - **completions**: numbers of jobs completed successfully
+  - **dispatches**: number of jobs dispatched for the first time (**retries** are not counted here)
+  - **expiries**: number of jobs exceeded `max_retry` and still failed to complete
+  - **failures**: number of jobs interrupted by graceful shutdown or errors (exceptions)
+  - **retries**: number of jobs dispatched for retrying
+
+Code examples of gathering counters information:
+  - get counter information for today
+    ```ruby
+    Skiplock::Counter.where(day: Date.today).first
+    ```
+  - get total number of successfully completed jobs within the past 30 days
+    ```ruby
+    Skiplock::Counter.where("day >= ?", 30.days.ago).sum(:completions)
+    ```
+  - get total number of expired jobs
+    ```ruby
+    Skiplock::Counter.sum(:expiries)
+    ```
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/vtt/skiplock.

data/bin/skiplock

@@ -5,7 +5,7 @@ begin
   op = OptionParser.new do |opts|
     opts.banner = "Usage: #{File.basename($0)} [options]"
     opts.on('-e', '--environment STRING', String, 'Rails environment')
-    opts.on('-l', '--logfile STRING', String, 'Full path to logfile')
+    opts.on('-l', '--logfile STRING', String, 'Log filename')
     opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
     opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
     opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')
@@ -25,4 +25,4 @@ options.transform_keys! { |k| k.to_s.gsub('-', '_').to_sym }
 env = options.delete(:environment)
 ENV['RAILS_ENV'] = env if env
 require File.expand_path("config/environment.rb")
-Skiplock::Manager.new(**options.merge(standalone: true))
+Rails.application.config.skiplock.standalone(**options)

data/lib/active_job/queue_adapters/skiplock_adapter.rb

@@ -2,7 +2,7 @@ module ActiveJob
   module QueueAdapters
     class SkiplockAdapter
       def initialize
-        Rails.application.config.after_initialize { Skiplock::Manager.new }
+        Rails.application.config.after_initialize { Rails.application.config.skiplock = Skiplock::Manager.new }
       end
 
       def enqueue(job)

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -43,24 +43,23 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
           record = NEW;
           IF (TG_OP = 'DELETE') THEN
             record = OLD;
-            IF (record.finished_at IS NOT NULL OR record.expired_at IS NOT NULL) THEN
-              RETURN NULL;
+            IF (record.running = TRUE) THEN
+              INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
             END IF;
-            INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
           ELSIF (record.running = TRUE) THEN
-            IF (record.executions IS NULL) THEN
-              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
-            ELSE
+            IF (record.executions > 0) THEN
               INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
+            ELSE
+              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
             END IF;
           ELSIF (record.finished_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
           ELSIF (record.expired_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
-          ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
+          ELSIF (record.executions > 0) THEN
             INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
           END IF;
-          PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
+          PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.job_class,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
           RETURN NULL;
         END;
       $$ LANGUAGE plpgsql

data/lib/skiplock/counter.rb

@@ -1,4 +1,5 @@
 module Skiplock
   class Counter < ActiveRecord::Base
+    self.implicit_order_column = 'day'
   end
 end

data/lib/skiplock/cron.rb

@@ -2,7 +2,6 @@ require 'cron_parser'
 module Skiplock
   class Cron
     def self.setup
-      Rails.application.eager_load! if Rails.env.development?
       cronjobs = []
       ActiveJob::Base.descendants.each do |j|
         next unless j.const_defined?('CRON')

data/lib/skiplock/job.rb

@@ -1,38 +1,32 @@
 module Skiplock
   class Job < ActiveRecord::Base
     self.implicit_order_column = 'created_at'
+    attr_accessor :activejob_retry
+    belongs_to :worker, inverse_of: :jobs, required: false
 
-    def self.dispatch(queues_order_query: nil, worker_id: nil, purge_completion: true, max_retries: 20)
-      job = nil
-      self.transaction do
-        job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
-        return (job ? job.scheduled_at.to_f : Float::INFINITY) if job.nil? || job.scheduled_at.to_f > Time.now.to_f
-        job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
-      end
-      job.data ||= {}
-      job.exception_executions ||= {}
-      job_data = job.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => job.id, 'enqueued_at' => job.updated_at, 'arguments' => (job.data['arguments'] || []))
-      job.executions = (job.executions || 0) + 1
-      Skiplock.logger.info "[Skiplock] Performing #{job.job_class} (#{job.id}) from queue '#{job.queue_name || 'default'}'..."
-      Thread.current[:skiplock_dispatch_job] = job
-      start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      begin
-        ActiveJob::Base.execute(job_data)
-      rescue Exception => ex
-        Skiplock.logger.error(ex)
-      end
-      unless ex
-        end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-        job_name = job.job_class
-        if job.job_class == 'Skiplock::Extension::ProxyJob'
-          target, method_name = ::YAML.load(job.data['arguments'].first)
-          job_name = "'#{target.name}.#{method_name}'"
+    # resynchronize jobs that could not commit to database and retry any abandoned jobs
+    def self.cleanup(purge_completion: true, max_retries: 20)
+      Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
+      Dir.glob('tmp/skiplock/*').each do |f|
+        job_from_db = self.find_by(id: File.basename(f), running: true)
+        disposed = true
+        if job_from_db
+          job, ex = YAML.load_file(f) rescue nil
+          disposed = job.dispose(ex, purge_completion: purge_completion, max_retries: max_retries) if job
         end
-        Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{job.id}) from queue '#{job.queue_name || 'default'}' in #{end_time - start_time} seconds"
+        (File.delete(f) rescue nil) if disposed
       end
-      job.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
-    ensure
-      Thread.current[:skiplock_dispatch_job] = nil
+      self.where(running: true).where.not(worker_id: Worker.ids).update_all(running: false, worker_id: nil)
+    end
+
+    def self.dispatch(purge_completion: true, max_retries: 20)
+      job = nil
+      self.connection.transaction do
+        job = self.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+        return if job.nil? || job.scheduled_at.to_f > Time.now.to_f
+        job = self.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
+      end
+      self.dispatch(purge_completion: purge_completion, max_retries: max_retries) if job.execute(purge_completion: purge_completion, max_retries: max_retries)
     end
 
     def self.enqueue(activejob)
@@ -42,13 +36,14 @@ module Skiplock
     def self.enqueue_at(activejob, timestamp)
       timestamp = Time.at(timestamp) if timestamp
       if Thread.current[:skiplock_dispatch_job].try(:id) == activejob.job_id
-        Thread.current[:skiplock_dispatch_job].exception_executions = activejob.exception_executions.merge('activejob_retry' => true)
+        Thread.current[:skiplock_dispatch_job].activejob_retry = true
         Thread.current[:skiplock_dispatch_job].executions = activejob.executions
+        Thread.current[:skiplock_dispatch_job].exception_executions = activejob.exception_executions
         Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
         Thread.current[:skiplock_dispatch_job]
       else
         serialize = activejob.serialize
-        Job.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'] }, 'scheduled_at' => timestamp))
+        self.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'] }, 'scheduled_at' => timestamp))
       end
     end
 
@@ -57,47 +52,82 @@ module Skiplock
     end
 
     def dispose(ex, purge_completion: true, max_retries: 20)
-      dup = self.dup
+      yaml = [self, ex].to_yaml
+      purging = false
       self.running = false
       self.worker_id = nil
-      self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
+      self.updated_at = Time.now > self.updated_at ? Time.now : self.updated_at + 1 # in case of clock drifting
       if ex
-        self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
-        if self.executions >= max_retries || self.exception_executions.key?('activejob_retry')
+        self.exception_executions ||= {}
+        self.exception_executions["[#{ex.class.name}]"] = self.exception_executions["[#{ex.class.name}]"].to_i + 1 unless self.activejob_retry
+        if self.executions.to_i >= max_retries || self.activejob_retry
           self.expired_at = Time.now
         else
-          self.scheduled_at = Time.now + (5 * 2**self.executions)
+          self.scheduled_at = Time.now + (5 * 2**self.executions.to_i)
         end
-        self.save!
         Skiplock.on_errors.each { |p| p.call(ex) }
-      elsif self.exception_executions.try(:key?, 'activejob_retry')
-        self.save!
-      elsif self.cron
-        self.data ||= {}
-        self.data['crons'] = (self.data['crons'] || 0) + 1
-        self.data['last_cron_at'] = Time.now.utc.to_s
-        next_cron_at = Cron.next_schedule_at(self.cron)
-        if next_cron_at
-          self.executions = nil
-          self.exception_executions = nil
-          self.scheduled_at = Time.at(next_cron_at)
-          self.save!
-        else
-          Skiplock.logger.error "[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}"
-          self.delete
+      elsif self.finished_at
+        if self.cron
+          self.data ||= {}
+          self.data['crons'] = (self.data['crons'] || 0) + 1
+          self.data['last_cron_at'] = Time.now.utc.to_s
+          next_cron_at = Cron.next_schedule_at(self.cron)
+          if next_cron_at
+            self.executions = nil
+            self.exception_executions = nil
+            self.scheduled_at = Time.at(next_cron_at)
+          else
+            Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
+            purging = true
+          end
+        elsif purge_completion
+          purging = true
         end
-      elsif purge_completion
-        self.delete
-      else
-        self.finished_at = Time.now
-        self.exception_executions = nil
-        self.save!
       end
-      self
+      purging ? self.delete : self.update_columns(self.attributes.slice(*self.changes.keys))
     rescue Exception => e
-      Skiplock.logger.error(e)
-      File.write("tmp/skiplock/#{self.id}", [dup, ex].to_yaml)
+      File.write("tmp/skiplock/#{self.id}", yaml) rescue nil
+      if Skiplock.logger
+        Skiplock.logger.error(e.to_s)
+        Skiplock.logger.error(e.backtrace.join("\n"))
+      end
+      Skiplock.on_errors.each { |p| p.call(e) }
       nil
     end
+
+    def execute(purge_completion: true, max_retries: 20)
+      Skiplock.logger.info("[Skiplock] Performing #{self.job_class} (#{self.id}) from queue '#{self.queue_name || 'default'}'...") if Skiplock.logger
+      self.data ||= {}
+      self.exception_executions ||= {}
+      self.activejob_retry = false
+      job_data = self.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => self.id, 'enqueued_at' => self.updated_at, 'arguments' => (self.data['arguments'] || []))
+      self.executions = self.executions.to_i + 1
+      Thread.current[:skiplock_dispatch_job] = self
+      start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      begin
+        ActiveJob::Base.execute(job_data)
+        self.finished_at = Time.now unless self.activejob_retry
+      rescue Exception => ex
+      end
+      if Skiplock.logger
+        if ex || self.activejob_retry
+          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.activejob_retry }")
+          if ex
+            Skiplock.logger.error(ex.to_s)
+            Skiplock.logger.error(ex.backtrace.join("\n"))
+          end
+        else
+          end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          job_name = self.job_class
+          if self.job_class == 'Skiplock::Extension::ProxyJob'
+            target, method_name = ::YAML.load(self.data['arguments'].first)
+            job_name = "'#{target.name}.#{method_name}'"
+          end
+          Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{self.id}) from queue '#{self.queue_name || 'default'}' in #{end_time - start_time} seconds"
+        end
+      end
+    ensure
+      self.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
+    end
   end
 end

data/lib/skiplock/manager.rb

@@ -6,30 +6,59 @@ module Skiplock
       @config.symbolize_keys!
       @config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
       @config.merge!(config)
-      Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
-      return unless @config[:standalone] || (caller.any?{ |l| l =~ %r{/rack/} } && (@config[:workers] == 0 || Rails.env.development?))
       @config[:hostname] = `hostname -f`.strip
-      do_config
-      banner if @config[:standalone]
+      configure
+      setup_logger
+      Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
+      if (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
+        cleanup_workers
+        @worker = create_worker
+        @worker.start(**@config)
+        at_exit { @worker.shutdown }
+      end
+    rescue Exception => ex
+      @logger.error(ex.to_s)
+      @logger.error(ex.backtrace.join("\n"))
+    end
+
+    def standalone(**options)
+      @config.merge!(options)
+      Rails.logger.reopen('/dev/null')
+      Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
+      @config[:workers] = 1 if @config[:workers] <= 0
+      @config[:standalone] = true
+      banner
       cleanup_workers
-      create_worker
-      ActiveJob::Base.logger = nil
-      if @config[:standalone]
-        standalone
-      else
-        dispatcher = Dispatcher.new(worker: @worker, **@config)
-        thread = dispatcher.run
-        at_exit do
-          dispatcher.shutdown
-          thread.join(@config[:graceful_shutdown])
-          @worker.delete
+      @worker = create_worker
+      @parent_id = Process.pid
+      @shutdown = false
+      Signal.trap('INT') { @shutdown = true }
+      Signal.trap('TERM') { @shutdown = true }
+      Signal.trap('HUP') { setup_logger }
+      (@config[:workers] - 1).times do |n|
+        fork do
+          sleep 1
+          worker = create_worker(master: false)
+          worker.start(worker_num: n + 1, **@config)
+          loop do
+            sleep 0.5
+            break if @shutdown || Process.ppid != @parent_id
+          end
+          worker.shutdown
         end
       end
-    rescue Exception => ex
-      @logger.error(ex)
+      @worker.start(**@config)
+      loop do
+        sleep 0.5
+        break if @shutdown
+      end
+      @logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
+      Process.waitall
+      @worker.shutdown
+      @logger.info "[Skiplock] Shutdown completed."
     end
 
-  private
+    private
 
     def banner
       title = "Skiplock #{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
@@ -43,6 +72,7 @@ module Skiplock
       @logger.info "           Min threads: #{@config[:min_threads]}"
       @logger.info "           Max threads: #{@config[:max_threads]}"
       @logger.info "           Environment: #{Rails.env}"
+      @logger.info "              Loglevel: #{@config[:loglevel]}"
       @logger.info "               Logfile: #{@config[:logfile] || '(disabled)'}"
       @logger.info "               Workers: #{@config[:workers]}"
       @logger.info "                Queues: #{@config[:queues].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if @config[:queues].is_a?(Hash)
@@ -52,24 +82,22 @@ module Skiplock
     end
 
     def cleanup_workers
+      Rails.application.eager_load! if Rails.env.development?
       delete_ids = []
       Worker.where(hostname: @config[:hostname]).each do |worker|
         sid = Process.getsid(worker.pid) rescue nil
-        delete_ids << worker.id if worker.sid != sid || worker.updated_at < 30.minutes.ago
-      end
-      if delete_ids.count > 0
-        Job.where(running: true, worker_id: delete_ids).update_all(running: false, worker_id: nil)
-        Worker.where(id: delete_ids).delete_all
+        delete_ids << worker.id if worker.sid != sid || worker.updated_at < 10.minutes.ago
       end
+      Worker.where(id: delete_ids).delete_all if delete_ids.count > 0
     end
 
-    def create_worker(pid: Process.pid, sid: Process.getsid(), master: true)
-      @worker = Worker.create!(pid: pid, sid: sid, master: master, hostname: @config[:hostname], capacity: @config[:max_threads])
+    def create_worker(master: true)
+      Worker.create!(pid: Process.pid, sid: Process.getsid(), master: master, hostname: @config[:hostname], capacity: @config[:max_threads])
     rescue
-      @worker = Worker.create!(pid: pid, sid: sid, master: false, hostname: @config[:hostname], capacity: @config[:max_threads])
+      Worker.create!(pid: Process.pid, sid: Process.getsid(), master: false, hostname: @config[:hostname], capacity: @config[:max_threads])
     end
 
-    def do_config
+    def configure
       @config[:graceful_shutdown] = 300 if @config[:graceful_shutdown] > 300
       @config[:graceful_shutdown] = nil if @config[:graceful_shutdown] < 0
       @config[:max_retries] = 20 if @config[:max_retries] > 20
@@ -78,19 +106,6 @@ module Skiplock
       @config[:max_threads] = 20 if @config[:max_threads] > 20
       @config[:min_threads] = 0 if @config[:min_threads] < 0
       @config[:workers] = 0 if @config[:workers] < 0
-      @config[:workers] = 1 if @config[:standalone] && @config[:workers] <= 0
-      @logger = ActiveSupport::Logger.new(STDOUT)
-      @logger.level = Rails.logger.level
-      Skiplock.logger = @logger
-      raise "Cannot create logfile '#{@config[:logfile]}'" if @config[:logfile] && !File.writable?(File.dirname(@config[:logfile]))
-      @config[:logfile] = nil if @config[:logfile].to_s.length == 0
-      if @config[:logfile]
-        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(@config[:logfile])))
-        if @config[:standalone]
-          Rails.logger.reopen('/dev/null')
-          Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
-        end
-      end
       @config[:queues].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if @config[:queues].is_a?(Hash)
       if @config[:notification] == 'auto'
         if defined?(Airbrake)
@@ -125,39 +140,19 @@ module Skiplock
       Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
     end
 
-    def standalone
-      parent_id = Process.pid
-      shutdown = false
-      Signal.trap("INT") { shutdown = true }
-      Signal.trap("TERM") { shutdown = true }
-      (@config[:workers] - 1).times do |n|
-        fork do
-          sleep 1
-          worker = create_worker(master: false)
-          dispatcher = Dispatcher.new(worker: worker, worker_num: n + 1, **@config)
-          thread = dispatcher.run
-          loop do
-            sleep 0.5
-            break if shutdown || Process.ppid != parent_id
-          end
-          dispatcher.shutdown
-          thread.join(@config[:graceful_shutdown])
-          worker.delete
-          exit
-        end
-      end
-      dispatcher = Dispatcher.new(worker: @worker, **@config)
-      thread = dispatcher.run
-      loop do
-        sleep 0.5
-        break if shutdown
+    def setup_logger
+      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
+      @logger = ActiveSupport::Logger.new(STDOUT)
+      @logger.level = @config[:loglevel].to_sym
+      Skiplock.logger = @logger
+      if @config[:logfile].to_s.length > 0
+        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(File.join(Rails.root, 'log', @config[:logfile].to_s), 'daily')))
+        ActiveJob::Base.logger = nil
       end
-      @logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
-      Process.waitall
-      dispatcher.shutdown
-      thread.join(@config[:graceful_shutdown])
-      @worker.delete
-      @logger.info "[Skiplock] Shutdown completed."
+    rescue Exception => ex
+      puts "Exception with logger: #{ex.to_s}"
+      puts ex.backtrace.join("\n")
+      Skiplock.on_errors.each { |p| p.call(ex) }
     end
   end
 end

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.11'
+  VERSION = Version = '1.0.15'
 end
 

data/lib/skiplock/worker.rb

@@ -1,5 +1,110 @@
 module Skiplock
   class Worker < ActiveRecord::Base
     self.implicit_order_column = 'created_at'
+    has_many :jobs, inverse_of: :worker
+
+    def start(worker_num: 0, **config)
+      @config = config
+      @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
+      @next_schedule_at = Time.now.to_f
+      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
+      if self.master
+        Job.cleanup(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
+        Cron.setup
+      end
+      @running = true
+      Process.setproctitle("skiplock-#{self.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
+      @executor.post { run }
+    end
+
+    def shutdown
+      @running = false
+      @executor.shutdown
+      @executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
+      self.delete
+    end
+
+    private
+
+    def get_next_available_job
+      @connection.transaction do
+        job = Job.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+        if job && job.scheduled_at.to_f <= Time.now.to_f
+          job = Job.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
+        end
+        job
+      end
+    end
+
+    def run
+      error = false
+      listen = false
+      timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      while @running
+        Rails.application.reloader.wrap do
+          begin
+            unless listen
+              @connection = self.class.connection
+              @connection.exec_query('LISTEN "skiplock::jobs"')
+              listen = true
+            end
+            if error
+              unless @connection.active?
+                @connection.reconnect!
+                sleep(0.5)
+                @connection.exec_query('LISTEN "skiplock::jobs"')
+                @next_schedule_at = Time.now.to_f
+              end
+              Job.cleanup if self.master
+              error = false
+            end
+            if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
+              job = get_next_available_job
+              if job.try(:running)
+                @executor.post { Rails.application.reloader.wrap { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) } }
+              else
+                @next_schedule_at = (job ? job.scheduled_at.to_f : Float::INFINITY)
+              end
+            end
+            job_notifications = []
+            @connection.raw_connection.wait_for_notify(0.4) do |channel, pid, payload|
+              job_notifications << payload if payload
+              loop do
+                payload = @connection.raw_connection.notifies
+                break unless @running && payload
+                job_notifications << payload[:extra]
+              end
+              job_notifications.each do |n|
+                op, id, worker_id, job_class, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
+                next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0
+                @next_schedule_at = scheduled_at.to_f if scheduled_at.to_f < @next_schedule_at
+              end
+            end
+            if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
+              self.touch
+              timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            end
+          rescue Exception => ex
+            # most likely error with database connection
+            Skiplock.logger.error(ex.to_s)
+            Skiplock.logger.error(ex.backtrace.join("\n"))
+            Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
+            error = true
+            wait(5)
+            @last_exception = ex
+          end
+          sleep(0.3)
+        end
+      end
+      @connection.exec_query('UNLISTEN *')
+    end
+
+    def wait(timeout)
+      t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      while @running
+        sleep(0.5)
+        break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > timeout
+      end
+    end
   end
 end

data/lib/skiplock.rb

@@ -3,7 +3,6 @@ require 'active_job/queue_adapters/skiplock_adapter'
 require 'active_record'
 require 'skiplock/counter'
 require 'skiplock/cron'
-require 'skiplock/dispatcher'
 require 'skiplock/extension'
 require 'skiplock/job'
 require 'skiplock/manager'
@@ -11,7 +10,7 @@ require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
-  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'log/skiplock.log', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
+  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
 
   def self.logger=(l)
     @logger = l

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.11
+  version: 1.0.15
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-08-30 00:00:00.000000000 Z
+date: 2021-09-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -84,7 +84,6 @@ files:
 - lib/skiplock.rb
 - lib/skiplock/counter.rb
 - lib/skiplock/cron.rb
-- lib/skiplock/dispatcher.rb
 - lib/skiplock/extension.rb
 - lib/skiplock/job.rb
 - lib/skiplock/manager.rb

data/lib/skiplock/dispatcher.rb

@@ -1,116 +0,0 @@
-module Skiplock
-  class Dispatcher
-    def initialize(worker:, worker_num: nil, **config)
-      @config = config
-      @worker = worker
-      @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
-      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads], max_threads: @config[:max_threads], max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
-      @last_dispatch_at = 0
-      @next_schedule_at = Time.now.to_f
-      Process.setproctitle("skiplock-#{@worker.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
-    end
-    
-    def run
-      @running = true
-      Thread.new do
-        ActiveRecord::Base.connection_pool.with_connection do |connection|
-          connection.exec_query('LISTEN "skiplock::jobs"')
-          if @worker.master
-            Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
-            check_sync_errors
-            Cron.setup
-          end
-          error = false
-          timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-          while @running
-            begin
-              if error
-                unless connection.active?
-                  connection.reconnect!
-                  sleep(0.5)
-                  connection.exec_query('LISTEN "skiplock::jobs"')
-                  @next_schedule_at = Time.now.to_f
-                end
-                check_sync_errors
-                error = false
-              end
-              job_notifications = []
-              connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
-                job_notifications << payload if payload
-                loop do
-                  payload = connection.raw_connection.notifies
-                  break unless @running && payload
-                  job_notifications << payload[:extra]
-                end
-                job_notifications.each do |n|
-                  op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
-                  next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0  || scheduled_at.to_f < @last_dispatch_at
-                  if scheduled_at.to_f <= Time.now.to_f
-                    @next_schedule_at = Time.now.to_f
-                  elsif scheduled_at.to_f < @next_schedule_at
-                    @next_schedule_at = scheduled_at.to_f
-                  end
-                end
-              end
-              if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
-                @executor.post { do_work }
-              end
-              if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
-                @worker.touch
-                timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-              end
-            rescue Exception => ex
-              # most likely error with database connection
-              Skiplock.logger.error(ex)
-              Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
-              error = true
-              t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-              while @running
-                sleep(0.5)
-                break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > 5
-              end
-              @last_exception = ex
-            end
-            sleep(0.2)
-          end
-          connection.exec_query('UNLISTEN *')
-          @executor.shutdown
-          @executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
-        end
-      end
-    end
-
-    def shutdown
-      @running = false
-    end
-
-    private
-
-    def check_sync_errors
-      # get performed jobs that could not sync with database
-      Dir.glob('tmp/skiplock/*').each do |f|
-        job_from_db = Job.find_by(id: File.basename(f), running: true)
-        disposed = true
-        if job_from_db
-          job, ex = YAML.load_file(f) rescue nil
-          disposed = job.dispose(ex, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
-        end
-        File.delete(f) if disposed
-      end
-    end
-
-    def do_work
-      while @running
-        @last_dispatch_at = Time.now.to_f - 1  # 1 second allowance for time drift
-        result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
-        next if result.is_a?(Job) && Time.now.to_f >= @next_schedule_at
-        @next_schedule_at = result if result.is_a?(Float)
-        break
-      end
-    rescue Exception => ex
-      Skiplock.logger.error(ex)
-      Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
-      @last_exception = ex
-    end
-  end
-end