skiplock 1.0.14 → 1.0.15

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55c98833e2792a216b6389d399e992aae9660f23ea076fcf852fb164cbc32d55
-  data.tar.gz: 43c8c2f90ee55ec423242042d1ab2986c0cb9f70fcc04db56aaf2c566bbb0382
+  metadata.gz: 5c965cac0fea118b7a8295bd82999bd33cd2807829dee639b6b782279a27697d
+  data.tar.gz: f5357c225546ef4fcec74a10b3180d07950c7f7f3308a39b2fe19c5668677a0f
 SHA512:
-  metadata.gz: 873492b815e6b5fc653630fed3bc64c5a363339551bbad304ecbad9453ea3738c24f3f0b874f4d6fe47cd4cc0ea5d46fb61b4c0dc322814d08dbc11002c24443
-  data.tar.gz: e84f7081e07ba0aff19834251de09e08df573d45ff159299fe08baf621b76845cd2fdf0461229da77b07cf707f151e7d8fc00d97b19a9bf30f1129376c2dd053
+  metadata.gz: d42f8d156e11c25d117a246f30cee23e7c487ecfa80e74e0a10866013f2a6145afd42bb55e3f057ff1687f686ee4e030f851d88fd3c2d3a99b33df745c4617f0
+  data.tar.gz: 981e7adad4bae9281e40bda3e049f68b32458078b98f192f5a9721ba31b40d82347c19a4da1876b210826a066f49f3ff0f2b8c966e9708b2d5b7d7bb890ff8b4

data/README.md

@@ -4,7 +4,7 @@
 
 It only uses the `LISTEN/NOTIFY/SKIP LOCKED` features provided natively on PostgreSQL 9.5+ to efficiently and reliably dispatch jobs to worker processes and threads ensuring that each job can be completed successfully **only once**.  No other polling or timer is needed.
 
-The library is quite small compared to other PostgreSQL job queues (eg. *delay_job*, *queue_classic*, *que*, *good_job*) with less than 400 lines of codes; and it still provides similar set of features and more...
+The library is quite small compared to other PostgreSQL job queues (eg. *delay_job*, *queue_classic*, *que*, *good_job*) with less than 500 lines of codes; and it still provides similar set of features and more...
 
 #### Compatibility:
 
@@ -187,6 +187,37 @@ If the `retry_on` block is not defined, then the built-in retry system of `skipl
   Subscription.skiplock(wait_until: Date.tomorrow.noon).charge(amount: 100)
   ```
 
+## Fault tolerant
+`Skiplock` ensures that jobs will be executed sucessfully only once even if database connection is lost during or after the job was dispatched.  Successful jobs are marked as completed or removed (with `purge_completion` turned on), and failed or interrupted jobs are marked for retry; however, when the database connection is dropped for any reasons and the commit is lost, `Skiplock` will then save the commit data to local disk (as `tmp/skiplock/<job_id>`) and synchronize with the database when the connection resumes.  This also protects in-progress jobs that were terminated abruptly during a graceful shutdown with timeout; they will be queued for retry.
+
+## Scalability
+`Skiplock` can scale both vertically and horizontally.  To scale vertically, simply increase the number of `Skiplock` workers per host.  To scale horizontally, simply deploy `Skiplock` to multiple hosts sharing the same PostgreSQL database.
+
+## Statistics and counters
+The `skiplock.workers` database table contains all the `Skiplock` workers running on all the hosts.  Active worker will update its timestamp column (`updated_at`) every minute; and dispatched jobs would be associated with the running workers.  At any given time, a list of active workers running a list of jobs can be determined using the database table.
+
+The `skiplock.counters` database table contains all historical job dispatches, completions, expiries, failures and retries.  The counters are recorded by dates; so it's possible to get statistical data for any given day or range of dates.
+
+  - **completions**: numbers of jobs completed successfully
+  - **dispatches**: number of jobs dispatched for the first time (**retries** are not counted here)
+  - **expiries**: number of jobs exceeded `max_retry` and still failed to complete
+  - **failures**: number of jobs interrupted by graceful shutdown or errors (exceptions)
+  - **retries**: number of jobs dispatched for retrying
+
+Code examples of gathering counters information:
+  - get counter information for today
+    ```ruby
+    Skiplock::Counter.where(day: Date.today).first
+    ```
+  - get total number of successfully completed jobs within the past 30 days
+    ```ruby
+    Skiplock::Counter.where("day >= ?", 30.days.ago).sum(:completions)
+    ```
+  - get total number of expired jobs
+    ```ruby
+    Skiplock::Counter.sum(:expiries)
+    ```
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/vtt/skiplock.

data/bin/skiplock

@@ -5,7 +5,7 @@ begin
   op = OptionParser.new do |opts|
     opts.banner = "Usage: #{File.basename($0)} [options]"
     opts.on('-e', '--environment STRING', String, 'Rails environment')
-    opts.on('-l', '--logfile STRING', String, 'Full path to logfile')
+    opts.on('-l', '--logfile STRING', String, 'Log filename')
     opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
     opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
     opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')
@@ -25,4 +25,4 @@ options.transform_keys! { |k| k.to_s.gsub('-', '_').to_sym }
 env = options.delete(:environment)
 ENV['RAILS_ENV'] = env if env
 require File.expand_path("config/environment.rb")
-Rails.application.config.skiplock.standalone(**options.merge(standalone: true))
+Rails.application.config.skiplock.standalone(**options)

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -43,21 +43,20 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
           record = NEW;
           IF (TG_OP = 'DELETE') THEN
             record = OLD;
-            IF (record.finished_at IS NOT NULL OR record.expired_at IS NOT NULL) THEN
-              RETURN NULL;
+            IF (record.running = TRUE) THEN
+              INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
             END IF;
-            INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
           ELSIF (record.running = TRUE) THEN
-            IF (record.executions IS NULL) THEN
-              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
-            ELSE
+            IF (record.executions > 0) THEN
               INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
+            ELSE
+              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
             END IF;
           ELSIF (record.finished_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
           ELSIF (record.expired_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
-          ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
+          ELSIF (record.executions > 0) THEN
             INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
           END IF;
           PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.job_class,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));

data/lib/skiplock/counter.rb

@@ -1,4 +1,5 @@
 module Skiplock
   class Counter < ActiveRecord::Base
+    self.implicit_order_column = 'day'
   end
 end

data/lib/skiplock/cron.rb

@@ -2,7 +2,6 @@ require 'cron_parser'
 module Skiplock
   class Cron
     def self.setup
-      Rails.application.eager_load! if Rails.env.development?
       cronjobs = []
       ActiveJob::Base.descendants.each do |j|
         next unless j.const_defined?('CRON')

data/lib/skiplock/job.rb

@@ -1,15 +1,32 @@
 module Skiplock
   class Job < ActiveRecord::Base
     self.implicit_order_column = 'created_at'
+    attr_accessor :activejob_retry
+    belongs_to :worker, inverse_of: :jobs, required: false
 
-    def self.dispatch(worker_id: nil, purge_completion: true, max_retries: 20)
+    # resynchronize jobs that could not commit to database and retry any abandoned jobs
+    def self.cleanup(purge_completion: true, max_retries: 20)
+      Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
+      Dir.glob('tmp/skiplock/*').each do |f|
+        job_from_db = self.find_by(id: File.basename(f), running: true)
+        disposed = true
+        if job_from_db
+          job, ex = YAML.load_file(f) rescue nil
+          disposed = job.dispose(ex, purge_completion: purge_completion, max_retries: max_retries) if job
+        end
+        (File.delete(f) rescue nil) if disposed
+      end
+      self.where(running: true).where.not(worker_id: Worker.ids).update_all(running: false, worker_id: nil)
+    end
+
+    def self.dispatch(purge_completion: true, max_retries: 20)
       job = nil
       self.connection.transaction do
         job = self.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
         return if job.nil? || job.scheduled_at.to_f > Time.now.to_f
-        job = self.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
+        job = self.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
       end
-      self.dispatch(worker_id: worker_id, purge_completion: purge_completion, max_retries: max_retries) if job.execute(purge_completion: purge_completion, max_retries: max_retries)
+      self.dispatch(purge_completion: purge_completion, max_retries: max_retries) if job.execute(purge_completion: purge_completion, max_retries: max_retries)
     end
 
     def self.enqueue(activejob)
@@ -19,8 +36,9 @@ module Skiplock
     def self.enqueue_at(activejob, timestamp)
       timestamp = Time.at(timestamp) if timestamp
       if Thread.current[:skiplock_dispatch_job].try(:id) == activejob.job_id
-        Thread.current[:skiplock_dispatch_job].exception_executions = activejob.exception_executions.merge('activejob_retry' => true)
+        Thread.current[:skiplock_dispatch_job].activejob_retry = true
         Thread.current[:skiplock_dispatch_job].executions = activejob.executions
+        Thread.current[:skiplock_dispatch_job].exception_executions = activejob.exception_executions
         Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
         Thread.current[:skiplock_dispatch_job]
       else
@@ -35,48 +53,45 @@ module Skiplock
 
     def dispose(ex, purge_completion: true, max_retries: 20)
       yaml = [self, ex].to_yaml
+      purging = false
       self.running = false
       self.worker_id = nil
-      self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
+      self.updated_at = Time.now > self.updated_at ? Time.now : self.updated_at + 1 # in case of clock drifting
       if ex
-        self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
-        if self.executions >= max_retries || self.exception_executions.key?('activejob_retry')
+        self.exception_executions ||= {}
+        self.exception_executions["[#{ex.class.name}]"] = self.exception_executions["[#{ex.class.name}]"].to_i + 1 unless self.activejob_retry
+        if self.executions.to_i >= max_retries || self.activejob_retry
           self.expired_at = Time.now
         else
-          self.scheduled_at = Time.now + (5 * 2**self.executions)
+          self.scheduled_at = Time.now + (5 * 2**self.executions.to_i)
         end
-        self.save!
         Skiplock.on_errors.each { |p| p.call(ex) }
-      elsif self.exception_executions.try(:key?, 'activejob_retry')
-        self.save!
-      elsif self.cron
-        self.data ||= {}
-        self.data['crons'] = (self.data['crons'] || 0) + 1
-        self.data['last_cron_at'] = Time.now.utc.to_s
-        next_cron_at = Cron.next_schedule_at(self.cron)
-        if next_cron_at
-          self.executions = nil
-          self.exception_executions = nil
-          self.scheduled_at = Time.at(next_cron_at)
-          self.save!
-        else
-          Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
-          self.delete
+      elsif self.finished_at
+        if self.cron
+          self.data ||= {}
+          self.data['crons'] = (self.data['crons'] || 0) + 1
+          self.data['last_cron_at'] = Time.now.utc.to_s
+          next_cron_at = Cron.next_schedule_at(self.cron)
+          if next_cron_at
+            self.executions = nil
+            self.exception_executions = nil
+            self.scheduled_at = Time.at(next_cron_at)
+          else
+            Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
+            purging = true
+          end
+        elsif purge_completion
+          purging = true
         end
-      elsif purge_completion
-        self.delete
-      else
-        self.finished_at = Time.now
-        self.exception_executions = nil
-        self.save!
       end
-      self
+      purging ? self.delete : self.update_columns(self.attributes.slice(*self.changes.keys))
     rescue Exception => e
+      File.write("tmp/skiplock/#{self.id}", yaml) rescue nil
       if Skiplock.logger
         Skiplock.logger.error(e.to_s)
         Skiplock.logger.error(e.backtrace.join("\n"))
-        File.write("tmp/skiplock/#{self.id}", yaml)
       end
+      Skiplock.on_errors.each { |p| p.call(e) }
       nil
     end
 
@@ -84,18 +99,19 @@ module Skiplock
       Skiplock.logger.info("[Skiplock] Performing #{self.job_class} (#{self.id}) from queue '#{self.queue_name || 'default'}'...") if Skiplock.logger
       self.data ||= {}
       self.exception_executions ||= {}
+      self.activejob_retry = false
       job_data = self.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => self.id, 'enqueued_at' => self.updated_at, 'arguments' => (self.data['arguments'] || []))
-      self.executions = (self.executions || 0) + 1
+      self.executions = self.executions.to_i + 1
       Thread.current[:skiplock_dispatch_job] = self
-      activejob = ActiveJob::Base.deserialize(job_data)
       start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       begin
-        activejob.perform_now
+        ActiveJob::Base.execute(job_data)
+        self.finished_at = Time.now unless self.activejob_retry
       rescue Exception => ex
       end
       if Skiplock.logger
-        if ex || self.exception_executions.key?('activejob_retry')
-          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.exception_executions.key?('activejob_retry') }")
+        if ex || self.activejob_retry
+          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.activejob_retry }")
           if ex
             Skiplock.logger.error(ex.to_s)
             Skiplock.logger.error(ex.backtrace.join("\n"))
@@ -110,6 +126,7 @@ module Skiplock
           Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{self.id}) from queue '#{self.queue_name || 'default'}' in #{end_time - start_time} seconds"
         end
       end
+    ensure
       self.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
     end
   end

data/lib/skiplock/manager.rb

@@ -8,6 +8,7 @@ module Skiplock
       @config.merge!(config)
       @config[:hostname] = `hostname -f`.strip
       configure
+      setup_logger
       Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
       if (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
         cleanup_workers
@@ -25,13 +26,15 @@ module Skiplock
       Rails.logger.reopen('/dev/null')
       Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
       @config[:workers] = 1 if @config[:workers] <= 0
+      @config[:standalone] = true
       banner
       cleanup_workers
       @worker = create_worker
       @parent_id = Process.pid
       @shutdown = false
-      Signal.trap("INT") { @shutdown = true }
-      Signal.trap("TERM") { @shutdown = true }
+      Signal.trap('INT') { @shutdown = true }
+      Signal.trap('TERM') { @shutdown = true }
+      Signal.trap('HUP') { setup_logger }
       (@config[:workers] - 1).times do |n|
         fork do
           sleep 1
@@ -79,13 +82,13 @@ module Skiplock
     end
 
     def cleanup_workers
+      Rails.application.eager_load! if Rails.env.development?
       delete_ids = []
       Worker.where(hostname: @config[:hostname]).each do |worker|
         sid = Process.getsid(worker.pid) rescue nil
-        delete_ids << worker.id if worker.sid != sid || worker.updated_at < 30.minutes.ago
+        delete_ids << worker.id if worker.sid != sid || worker.updated_at < 10.minutes.ago
       end
       Worker.where(id: delete_ids).delete_all if delete_ids.count > 0
-      Job.where(running: true).where.not(worker_id: Worker.ids).update_all(running: false, worker_id: nil)
     end
 
     def create_worker(master: true)
@@ -95,7 +98,6 @@ module Skiplock
     end
 
     def configure
-      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
       @config[:graceful_shutdown] = 300 if @config[:graceful_shutdown] > 300
       @config[:graceful_shutdown] = nil if @config[:graceful_shutdown] < 0
       @config[:max_retries] = 20 if @config[:max_retries] > 20
@@ -104,15 +106,6 @@ module Skiplock
       @config[:max_threads] = 20 if @config[:max_threads] > 20
       @config[:min_threads] = 0 if @config[:min_threads] < 0
       @config[:workers] = 0 if @config[:workers] < 0
-      @logger = ActiveSupport::Logger.new(STDOUT)
-      @logger.level = @config[:loglevel].to_sym
-      Skiplock.logger = @logger
-      raise "Cannot create logfile '#{@config[:logfile]}'" if @config[:logfile] && !File.writable?(File.dirname(@config[:logfile]))
-      @config[:logfile] = nil if @config[:logfile].to_s.length == 0
-      if @config[:logfile]
-        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(@config[:logfile])))
-        ActiveJob::Base.logger = nil
-      end
       @config[:queues].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if @config[:queues].is_a?(Hash)
       if @config[:notification] == 'auto'
         if defined?(Airbrake)
@@ -146,5 +139,20 @@ module Skiplock
       end
       Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
     end
+
+    def setup_logger
+      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
+      @logger = ActiveSupport::Logger.new(STDOUT)
+      @logger.level = @config[:loglevel].to_sym
+      Skiplock.logger = @logger
+      if @config[:logfile].to_s.length > 0
+        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(File.join(Rails.root, 'log', @config[:logfile].to_s), 'daily')))
+        ActiveJob::Base.logger = nil
+      end
+    rescue Exception => ex
+      puts "Exception with logger: #{ex.to_s}"
+      puts ex.backtrace.join("\n")
+      Skiplock.on_errors.each { |p| p.call(ex) }
+    end
   end
 end

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.14'
+  VERSION = Version = '1.0.15'
 end
 

data/lib/skiplock/worker.rb

@@ -1,17 +1,20 @@
 module Skiplock
   class Worker < ActiveRecord::Base
     self.implicit_order_column = 'created_at'
+    has_many :jobs, inverse_of: :worker
 
     def start(worker_num: 0, **config)
       @config = config
       @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
       @next_schedule_at = Time.now.to_f
       @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
+      if self.master
+        Job.cleanup(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries])
+        Cron.setup
+      end
       @running = true
       Process.setproctitle("skiplock-#{self.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
-      @executor.post do
-        Rails.application.reloader.wrap { run }
-      end
+      @executor.post { run }
     end
 
     def shutdown
@@ -23,19 +26,6 @@ module Skiplock
 
     private
 
-    def check_sync_errors
-      # get executed jobs that could not sync with database
-      Dir.glob('tmp/skiplock/*').each do |f|
-        job_from_db = Job.find_by(id: File.basename(f), running: true)
-        disposed = true
-        if job_from_db
-          job, ex = YAML.load_file(f) rescue nil
-          disposed = job.dispose(ex, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) if job
-        end
-        File.delete(f) if disposed
-      end
-    end
-
     def get_next_available_job
       @connection.transaction do
         job = Job.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
@@ -47,18 +37,17 @@ module Skiplock
     end
 
     def run
-      @connection = self.class.connection
-      @connection.exec_query('LISTEN "skiplock::jobs"')
-      if self.master
-        Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
-        check_sync_errors
-        Cron.setup
-      end
       error = false
+      listen = false
       timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       while @running
         Rails.application.reloader.wrap do
           begin
+            unless listen
+              @connection = self.class.connection
+              @connection.exec_query('LISTEN "skiplock::jobs"')
+              listen = true
+            end
             if error
               unless @connection.active?
                 @connection.reconnect!
@@ -66,17 +55,13 @@ module Skiplock
                 @connection.exec_query('LISTEN "skiplock::jobs"')
                 @next_schedule_at = Time.now.to_f
               end
-              check_sync_errors if self.master
+              Job.cleanup if self.master
               error = false
             end
             if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
               job = get_next_available_job
               if job.try(:running)
-                @executor.post do
-                  Rails.application.executor.wrap do
-                    ActiveSupport::Dependencies.interlock.permit_concurrent_loads { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) }
-                  end
-                end
+                @executor.post { Rails.application.reloader.wrap { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) } }
               else
                 @next_schedule_at = (job ? job.scheduled_at.to_f : Float::INFINITY)
               end
@@ -105,11 +90,7 @@ module Skiplock
             Skiplock.logger.error(ex.backtrace.join("\n"))
             Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
             error = true
-            t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-            while @running
-              sleep(0.5)
-              break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > 5
-            end
+            wait(5)
             @last_exception = ex
           end
           sleep(0.3)
@@ -117,5 +98,13 @@ module Skiplock
       end
       @connection.exec_query('UNLISTEN *')
     end
+
+    def wait(timeout)
+      t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      while @running
+        sleep(0.5)
+        break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > timeout
+      end
+    end
   end
 end

data/lib/skiplock.rb

@@ -10,7 +10,7 @@ require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
-  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'log/skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
+  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
 
   def self.logger=(l)
     @logger = l

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.14
+  version: 1.0.15
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-09-07 00:00:00.000000000 Z
+date: 2021-09-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob