skiplock 1.0.17 → 1.0.22

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d9533e3df15a24e784000c160588b96207271986bf927bff095752e556c3376a
-  data.tar.gz: dcc40ab796f55fc41363d671d5e4f73600c58c468d7c4543e496fad38c3e9a8a
+  metadata.gz: 3b99466d7f1848f9dcd750a4dceb6a39d7124707e941c74b6c75f0b3a57944ff
+  data.tar.gz: 3f3ed1c9327d41e675d5c170590b52fe32f1facc5b57667dba513e5324e6ee20
 SHA512:
-  metadata.gz: 0a0c21835b52f749022d7f3a135f041d122425defffaf6aebb6d8510d4dd0836aed2fb9977daff9e7d3bf2547e139d2dd3ea49f5f55a869ce851c93aaeb56a12
-  data.tar.gz: d94c6801c82e1bddb12cff0152b71e8554d03de4d28f51ec0d742807311adecca1b3b142aeadf3cd5e81336416b03206d81848f6da6abf8e0f8c726631a5d898
+  metadata.gz: '0129296507b0479ba28a6e4e0bce810340f0ff2559004da1385cf02b6d6023f7be5cb1543f3fb24cebdeb579d6be6ed1e5d7d72d43245de5374926607b23d7df'
+  data.tar.gz: 163faf212d06b6d7f8045eb347ccab77de5b16bfe77fe8780a5d73176ce3c5804cc25cbca8f962d92bc5dfccc1ada223d03fc0b443d0df48125a12bf4ec7ddeb

data/README.md

@@ -106,7 +106,7 @@ Inside the Rails application:
     MyJob.set(wait_until: Day.tomorrow.noon).perform_later(1,2,3)
     ```
 - Skiplock supports custom options which override the global `Skiplock` configuration options for specified jobs
-    - **purge** (*boolean*): whether to remove this job after it has ran successfully
+    - **purge** (*boolean*): whether to remove this job after it has completed successfully
     - **max_retries** (*integer*): set maximum retry attempt for this job
     ```ruby
     MyJob.set(purge: false, max_retries: 5).perform_later(1,2,3)
@@ -119,7 +119,8 @@ Outside the Rails application:
 - with scheduling, priority, queue, arguments and custom options
     ```sql
     INSERT INTO skiplock.jobs(job_class, queue_name, priority, scheduled_at, data)
-      VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min', '{"arguments":[1,2,3],"options":{"purge":false,"max_retries":5}}');
+      VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min',
+      '{"arguments":[1,2,3],"options":{"purge":false,"max_retries":5}}');
     ```
 ## Queue priority vs Job priority
 *Why do queues use priorities when jobs already have priorities?*
@@ -171,22 +172,22 @@ If the `retry_on` block is not defined, then the built-in retry system of `Skipl
 `Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  Custom notification can also be called whenever an exception occurs; it can be configured in an initializer like below:
 ```ruby
   # config/initializers/skiplock.rb
-  Skiplock.on_error do |ex, previous|
-    if ex.backtrace != previous.try(:backtrace)
-      # sends text message using Amazon SNS on new exceptions only
-      # the same repeated exceptions will only be sent once to avoid SPAM
-      # NOTE: exceptions generated from Job executions will not provide 'previous' exceptions
-      sms = Aws::SNS::Client.new(region: 'us-west-2', access_key_id: Rails.application.credentials[:aws][:access_key_id], secret_access_key: Rails.application.credentials[:aws][:secret_access_key])
-      sms.publish({ phone_number: '+122233334444', message: "Exception: #{ex.message}"[0..130] })
-    end
+  Skiplock.on_error do |ex|
+    # sends text message using Amazon SNS
+    sms = Aws::SNS::Client.new(region: 'us-west-2', access_key_id: Rails.application.credentials[:aws][:access_key_id], secret_access_key: Rails.application.credentials[:aws][:secret_access_key])
+    sms.publish(phone_number: '+12223334444', message: "Exception: #{ex.message}"[0..130])
   end
   # supports multiple 'on_error' event callbacks
 ```
 ## ClassMethod extension
-`Skiplock` can add extension to allow class methods to be performed as a background job; it is disabled in the default configuration.  To enable globally for all classes and modules, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`; this can expose remote execution if the `skiplock.jobs` database table is not secured properly.  To enable extension for specific classes and modules only then set the configuration to an array of names of the classes and modules eg. `['MyClass', 'MyModule']`
-- An example of remote execution if the extension is enabled globally (ie: configuration is set to `true`) and attacker can insert `skiplock.jobs`
+`Skiplock` can add extension to allow class methods to be performed as a background job; it is disabled in the default configuration.  To enable globally for all classes and modules, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`; this can expose remote code execution if the `skiplock.jobs` database table is not secured properly.
+
+To enable extension for specific classes and modules only then set the configuration to an array of names of the classes and modules eg. `['MyClass', 'MyModule']`
+- An example of remote code execution if the extension is enabled globally (ie: configuration is set to `true`) and attacker can insert `skiplock.jobs`
   ```sql
-  INSERT INTO skiplock.jobs(job_class, data) VALUES ('Skiplock::Extension::ProxyJob', '{"arguments":["---\n- !ruby/module ''Kernel''\n- :system\n- - rm -rf /tmp/*\n"]}');
+  INSERT INTO skiplock.jobs(job_class, data)
+    VALUES ('Skiplock::Extension::ProxyJob',
+    '{"arguments":["---\n- !ruby/module ''Kernel''\n- :system\n- - rm -rf /tmp/*\n"]}');
   ```
 - Queue class method `generate_thumbnails` of class `Image` as background job to run as soon as possible
   ```ruby
@@ -196,9 +197,9 @@ If the `retry_on` block is not defined, then the built-in retry system of `Skipl
   ```ruby
   Session.skiplock(wait: 5.minutes, queue: 'maintenance').cleanup
   ```
-- Queue class method `charge` of class `Subscription` as background job to run tomorrow at noon
+- Queue class method `charge` of class `Subscription` as background job to run tomorrow at noon without purging
   ```ruby
-  Subscription.skiplock(wait_until: Date.tomorrow.noon).charge(amount: 100)
+  Subscription.skiplock(purge: false, wait_until: Date.tomorrow.noon).charge(amount: 100)
   ```
 
 ## Fault tolerant

data/bin/skiplock

@@ -7,6 +7,7 @@ begin
     opts.banner = "Usage: #{File.basename($0)} [options]"
     opts.on('-e', '--environment STRING', String, 'Rails environment')
     opts.on('-l', '--logfile STRING', String, 'Log filename')
+    opts.on('-L', '--loglevel STRING', String, 'Log level (debug, info, warn, error, fatal, unknown)')
     opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
     opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
     opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -46,18 +46,20 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
             IF (record.running = TRUE) THEN
               INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
             END IF;
-          ELSIF (record.running = TRUE) THEN
-            IF (record.executions > 0) THEN
-              INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
-            ELSE
-              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
+          ELSIF (TG_OP = 'UPDATE') THEN
+            IF (OLD.running = FALSE AND record.running = TRUE) THEN
+              IF (record.executions > 0) THEN
+                INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
+              ELSE
+                INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
+              END IF;
+            ELSIF (OLD.finished_at IS NULL AND record.finished_at IS NOT NULL) THEN
+              INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
+            ELSIF (OLD.running = TRUE AND record.running = FALSE AND record.expired_at IS NOT NULL) THEN
+              INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
+            ELSIF (OLD.running = TRUE AND record.running = FALSE AND record.expired_at IS NULL AND record.finished_at IS NULL) THEN
+              INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
             END IF;
-          ELSIF (record.finished_at IS NOT NULL) THEN
-            INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
-          ELSIF (record.expired_at IS NOT NULL) THEN
-            INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
-          ELSIF (TG_OP = 'UPDATE' AND OLD.running = TRUE AND NEW.running = FALSE) THEN
-            INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
           END IF;
           PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.job_class,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
           RETURN NULL;
@@ -85,7 +87,7 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
     execute "CREATE INDEX jobs_retry_index ON skiplock.jobs(scheduled_at) WHERE running = FALSE AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL"
     execute "CREATE INDEX jobs_cron_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE cron IS NOT NULL AND finished_at IS NULL"
     execute "CREATE UNIQUE INDEX jobs_unique_cron_index ON skiplock.jobs (job_class) WHERE cron IS NOT NULL"
-    execute "CREATE UNIQUE INDEX workers_unique_master_index ON skiplock.workers(hostname) WHERE master = 't'"
+    execute "CREATE UNIQUE INDEX workers_unique_master_index ON skiplock.workers(hostname) WHERE master = TRUE"
   end
 
   def down

data/lib/skiplock/cron.rb

@@ -11,7 +11,7 @@ module Skiplock
         if time
           job.cron = cron
           job.running = false
-          job.scheduled_at = Time.at(time)
+          job.scheduled_at = Time.at(time) unless job.try(:executions).to_i > 0  # do not update schedule of retrying cron jobs
           job.save
           cronjobs << j.name
         end

data/lib/skiplock/extension.rb

@@ -11,9 +11,12 @@ module Skiplock
       end
     end
 
+    class ProxyError < StandardError; end
+
     class ProxyJob < ActiveJob::Base
       def perform(yml)
-        target, method_name, args = ::YAML.load(yml)
+        target, method_name, args = ::YAML.load(yml) rescue nil
+        raise ProxyError, "Skiplock extension is not allowed for:\n#{yml}" unless target.respond_to?(:skiplock)
         target.__send__(method_name, *args)
       end
     end

data/lib/skiplock/job.rb

@@ -1,7 +1,10 @@
 module Skiplock
   class Job < ActiveRecord::Base
-    self.implicit_order_column = 'created_at'
-    attr_accessor :activejob_retry
+    self.implicit_order_column = 'updated_at'
+    attribute :activejob_error
+    attribute :exception
+    attribute :max_retries, :integer
+    attribute :purge, :boolean
     belongs_to :worker, inverse_of: :jobs, required: false
 
     def self.dispatch(purge_completion: true, max_retries: 20)
@@ -19,27 +22,28 @@ module Skiplock
     end
 
     def self.enqueue_at(activejob, timestamp)
+      options = activejob.instance_variable_get('@skiplock_options') || {}
       timestamp = Time.at(timestamp) if timestamp
       if Thread.current[:skiplock_job].try(:id) == activejob.job_id
-        Thread.current[:skiplock_job].activejob_retry = true
+        Thread.current[:skiplock_job].activejob_error = options[:error]
+        Thread.current[:skiplock_job].data['activejob_error'] = true
         Thread.current[:skiplock_job].executions = activejob.executions
         Thread.current[:skiplock_job].exception_executions = activejob.exception_executions
         Thread.current[:skiplock_job].scheduled_at = timestamp
         Thread.current[:skiplock_job]
       else
-        options = activejob.instance_variable_get('@skiplock_options') || {}
         serialize = activejob.serialize
         self.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'], 'options' => options }, 'scheduled_at' => timestamp))
       end
     end
 
-    # resynchronize jobs that could not commit to database and retry any abandoned jobs
+    # resynchronize jobs that could not commit to database and reset any abandoned jobs for retry
     def self.flush
       Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
       Dir.glob('tmp/skiplock/*').each do |f|
         disposed = true
         if self.exists?(id: File.basename(f), running: true)
-          job = Marshal.load(File.binread(f)) rescue nil
+          job = YAML.load_file(f) rescue nil
           disposed = job.dispose if job.is_a?(Skiplock::Job)
         end
         (File.delete(f) rescue nil) if disposed
@@ -53,15 +57,15 @@ module Skiplock
     end
 
     def dispose
-      return unless @max_retries
-      dump = Marshal.dump(self)
+      return unless self.max_retries
+      yaml = self.to_yaml
       purging = false
       self.running = false
       self.worker_id = nil
       self.updated_at = Time.now > self.updated_at ? Time.now : self.updated_at + 1 # in case of clock drifting
-      if @exception
-        self.exception_executions["[#{@exception.class.name}]"] = self.exception_executions["[#{@exception.class.name}]"].to_i + 1 unless self.activejob_retry
-        if (self.executions.to_i >= @max_retries + 1) || self.activejob_retry
+      if self.exception
+        self.exception_executions["[#{self.exception.class.name}]"] = self.exception_executions["[#{self.exception.class.name}]"].to_i + 1 unless self.data.key?('activejob_error')
+        if (self.executions.to_i >= self.max_retries + 1) || self.data.key?('activejob_error') || self.exception.is_a?(Skiplock::Extension::ProxyError)
           self.expired_at = Time.now
         else
           self.scheduled_at = Time.now + (5 * 2**self.executions.to_i)
@@ -71,25 +75,27 @@ module Skiplock
           self.data['cron'] ||= {}
           self.data['cron']['executions'] = self.data['cron']['executions'].to_i + 1
           self.data['cron']['last_finished_at'] = self.finished_at.utc.to_s
+          self.data['cron']['last_result'] = self.data['result']
           next_cron_at = Cron.next_schedule_at(self.cron)
           if next_cron_at
             # update job to record completions counter before resetting finished_at to nil
-            self.update_columns(self.attributes.slice(*self.changes.keys))
+            self.update_columns(self.attributes.slice(*(self.changes.keys & self.class.column_names)))
             self.finished_at = nil
             self.executions = nil
             self.exception_executions = nil
+            self.data.delete('result')
             self.scheduled_at = Time.at(next_cron_at)
           else
             Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
             purging = true
           end
-        elsif @purge == true
+        elsif self.purge == true
           purging = true
         end
       end
-      purging ? self.delete : self.update_columns(self.attributes.slice(*self.changes.keys))
+      purging ? self.delete : self.update_columns(self.attributes.slice(*(self.changes.keys & self.class.column_names)))
     rescue Exception => e
-      File.binwrite("tmp/skiplock/#{self.id}", dump) rescue nil
+      File.write("tmp/skiplock/#{self.id}", yaml) rescue nil
       if Skiplock.logger
         Skiplock.logger.error(e.to_s)
         Skiplock.logger.error(e.backtrace.join("\n"))
@@ -100,14 +106,15 @@ module Skiplock
 
     def execute(purge_completion: true, max_retries: 20)
       raise 'Job has already been completed' if self.finished_at
+      self.update_columns(running: true, updated_at: Time.now) unless self.running
       Skiplock.logger.info("[Skiplock] Performing #{self.job_class} (#{self.id}) from queue '#{self.queue_name || 'default'}'...") if Skiplock.logger
       self.data ||= {}
       self.data.delete('result')
       self.exception_executions ||= {}
-      self.activejob_retry = false
-      @max_retries = (self.data['options'].key?('max_retries') ? self.data['options']['max_retries'].to_i : max_retries) rescue max_retries
-      @max_retries = 20 if @max_retries < 0 || @max_retries > 20
-      @purge = (self.data['options'].key?('purge') ? self.data['options']['purge'] : purge_completion) rescue purge_completion
+      self.activejob_error = nil
+      self.max_retries = (self.data['options'].key?('max_retries') ? self.data['options']['max_retries'].to_i : max_retries) rescue max_retries
+      self.max_retries = 20 if self.max_retries < 0 || self.max_retries > 20
+      self.purge = (self.data['options'].key?('purge') ? self.data['options']['purge'] : purge_completion) rescue purge_completion
       job_data = self.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => self.id, 'enqueued_at' => self.updated_at, 'arguments' => (self.data['arguments'] || []))
       self.executions = self.executions.to_i + 1
       Thread.current[:skiplock_job] = self
@@ -115,15 +122,15 @@ module Skiplock
       begin
         self.data['result'] = ActiveJob::Base.execute(job_data)
       rescue Exception => ex
-        @exception = ex
-        Skiplock.on_errors.each { |p| p.call(@exception) }
+        self.exception = ex
+        Skiplock.on_errors.each { |p| p.call(ex) }
       end
       if Skiplock.logger
-        if @exception || self.activejob_retry
-          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.activejob_retry }")
-          if @exception
-            Skiplock.logger.error(@exception.to_s)
-            Skiplock.logger.error(@exception.backtrace.join("\n"))
+        if self.exception || self.activejob_error
+          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.activejob_error }")
+          if self.exception
+            Skiplock.logger.error(self.exception.to_s)
+            Skiplock.logger.error(self.exception.backtrace.join("\n"))
           end
         else
           end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
@@ -135,8 +142,9 @@ module Skiplock
           Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{self.id}) from queue '#{self.queue_name || 'default'}' in #{end_time - start_time} seconds"
         end
       end
+      self.exception || self.activejob_error || self.data['result']
     ensure
-      self.finished_at ||= Time.now if self.data.key?('result') && !self.activejob_retry
+      self.finished_at ||= Time.now if self.data.key?('result') && !self.activejob_error
       self.dispose
     end
   end

data/lib/skiplock/manager.rb

@@ -1,20 +1,25 @@
 module Skiplock
   class Manager
-    def initialize(**config)
+    def initialize
       @config = Skiplock::DEFAULT_CONFIG.dup
       @config.merge!(YAML.load_file('config/skiplock.yml')) rescue nil
       @config.symbolize_keys!
-      @config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
-      @config.merge!(config)
-      @hostname = Socket.gethostname
-      configure
-      setup_logger
-      if (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
-        Worker.cleanup(@hostname)
-        @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
-        @worker.start(**@config)
-        at_exit { @worker.shutdown }
+      Rails.application.eager_load! if Rails.env.development?
+      if @config[:extensions] == true
+        Module.__send__(:include, Skiplock::Extension)
+      elsif @config[:extensions].is_a?(Array)
+        @config[:extensions].each { |n| n.constantize.__send__(:extend, Skiplock::Extension) if n.safe_constantize }
       end
+      async if (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
+    end
+
+    def async
+      setup_logger
+      configure
+      Worker.cleanup(@hostname)
+      @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
+      @worker.start(**@config)
+      at_exit { @worker.shutdown }
     rescue Exception => ex
       @logger.error(ex.to_s)
       @logger.error(ex.backtrace.join("\n"))
@@ -22,18 +27,19 @@ module Skiplock
 
     def standalone(**options)
       @config.merge!(options)
-      Rails.logger.reopen('/dev/null')
-      Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
-      @config[:workers] = 1 if @config[:workers] <= 0
       @config[:standalone] = true
+      @config[:workers] = 1 if @config[:workers] <= 0
+      setup_logger
+      configure
       banner
-      Worker.cleanup(@hostname)
-      @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
       @parent_id = Process.pid
       @shutdown = false
       Signal.trap('INT') { @shutdown = true }
       Signal.trap('TERM') { @shutdown = true }
       Signal.trap('HUP') { setup_logger }
+      Worker.cleanup(@hostname)
+      @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
+      ActiveRecord::Base.connection.disconnect! if @config[:workers] > 1
       (@config[:workers] - 1).times do |n|
         fork do
           sleep 1
@@ -46,6 +52,7 @@ module Skiplock
           worker.shutdown
         end
       end
+      ActiveRecord::Base.establish_connection if @config[:workers] > 1
       @worker.start(**@config)
       loop do
         sleep 0.5
@@ -54,7 +61,9 @@ module Skiplock
       @logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
       Process.waitall
       @worker.shutdown
-      @logger.info "[Skiplock] Shutdown completed."
+    rescue Exception => ex
+      @logger.error(ex.to_s)
+      @logger.error(ex.backtrace.join("\n"))
     end
 
     private
@@ -81,6 +90,8 @@ module Skiplock
     end
 
     def configure
+      @hostname = Socket.gethostname
+      @config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
       @config[:graceful_shutdown] = 300 if @config[:graceful_shutdown] > 300
       @config[:graceful_shutdown] = nil if @config[:graceful_shutdown] < 0
       @config[:max_retries] = 20 if @config[:max_retries] > 20
@@ -104,40 +115,38 @@ module Skiplock
       case @config[:notification]
       when 'airbrake'
         raise 'airbrake gem not found' unless defined?(Airbrake)
-        Skiplock.on_error do |ex, previous|
-          Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace)
+        Skiplock.on_error do |ex|
+          Airbrake.notify_sync(ex)
         end
       when 'bugsnag'
         raise 'bugsnag gem not found' unless defined?(Bugsnag)
-        Skiplock.on_error do |ex, previous|
-          Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace)
+        Skiplock.on_error do |ex|
+          Bugsnag.notify(ex)
         end
       when 'exception_notification'
         raise 'exception_notification gem not found' unless defined?(ExceptionNotifier)
-        Skiplock.on_error do |ex, previous|
-          ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace)
+        Skiplock.on_error do |ex|
+          ExceptionNotifier.notify_exception(ex)
         end
       else
         @config[:notification] = 'custom'
       end
-      Rails.application.eager_load! if Rails.env.development?
-      if @config[:extensions] == true
-        Module.__send__(:include, Skiplock::Extension)
-      elsif @config[:extensions].is_a?(Array)
-        @config[:extensions].each { |n| n.constantize.__send__(:extend, Skiplock::Extension) if n.safe_constantize }
-      end
-      Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
+      Skiplock.on_errors.freeze
     end
 
     def setup_logger
-      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s.downcase)
+      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
       @logger = ActiveSupport::Logger.new(STDOUT)
-      @logger.level = @config[:loglevel].downcase.to_sym
+      @logger.level = @config[:loglevel].to_sym
       Skiplock.logger = @logger
       if @config[:logfile].to_s.length > 0
         @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(File.join(Rails.root, 'log', @config[:logfile].to_s), 'daily')))
         ActiveJob::Base.logger = nil
       end
+      if @config[:standalone]
+        Rails.logger.reopen('/dev/null') rescue Rails.logger.reopen('NUL') # supports Windows NUL device
+        Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
+      end
     rescue Exception => ex
       @logger.error "Exception with logger: #{ex.to_s}"
       @logger.error ex.backtrace.join("\n")

data/lib/skiplock/patch.rb

@@ -1,7 +1,7 @@
 module Skiplock
   module Patch
     def enqueue(options = {})
-      self.instance_variable_set('@skiplock_options', options)
+      @skiplock_options = options
       super
     end
   end

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.17'
+  VERSION = Version = '1.0.22'
 end
 

data/lib/skiplock/worker.rb

@@ -1,6 +1,6 @@
 module Skiplock
   class Worker < ActiveRecord::Base
-    self.implicit_order_column = 'created_at'
+    self.implicit_order_column = 'updated_at'
     has_many :jobs, inverse_of: :worker
 
     def self.cleanup(hostname = nil)
@@ -18,74 +18,77 @@ module Skiplock
       self.create!(pid: Process.pid, sid: Process.getsid(), master: false, hostname: hostname, capacity: capacity)
     end
 
-    def start(worker_num: 0, **config)
-      if self.master
-        Job.flush
-        Cron.setup
-      end
-      @config = config
-      @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
-      @running = true
-      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
-      @executor.post { run }
-      Process.setproctitle("skiplock-#{self.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
-    end
-
     def shutdown
       @running = false
       @executor.shutdown
       @executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
       self.delete
+      Skiplock.logger.info "[Skiplock] Shutdown of #{self.master ? 'master' : 'cluster'} worker#{(' ' + @num.to_s) if @num > 0 && @config[:workers] > 2} (PID: #{self.pid}) was completed."
     end
 
-    private
-
-    def get_next_available_job
-      @connection.transaction do
-        job = Job.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
-        if job && job.scheduled_at.to_f <= Time.now.to_f
-          job = Job.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
-        end
-        job
-      end
+    def start(worker_num: 0, **config)
+      @num = worker_num
+      @config = config
+      @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
+      @running = true
+      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads] + 1, idletime: 60, auto_terminate: true, fallback_policy: :discard)
+      @executor.post { run }
+      Process.setproctitle("skiplock: #{self.master ? 'master' : 'cluster'} worker#{(' ' + @num.to_s) if @num > 0 && @config[:workers] > 2} [#{Rails.application.class.name.deconstantize.downcase}:#{Rails.env}]") if @config[:standalone]
     end
 
+    private
+
     def run
+      sleep 3
+      Skiplock.logger.info "[Skiplock] Starting in #{@config[:standalone] ? 'standalone' : 'async'} mode (PID: #{self.pid}) with #{@config[:max_threads]} max threads as #{self.master ? 'master' : 'cluster'} worker#{(' ' + @num.to_s) if @num > 0 && @config[:workers] > 2}..."
+      connection = nil
       error = false
       listen = false
       next_schedule_at = Time.now.to_f
+      pg_exception_timestamp = nil
       timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       while @running
         Rails.application.reloader.wrap do
           begin
             unless listen
-              @connection = self.class.connection
-              @connection.exec_query('LISTEN "skiplock::jobs"')
+              connection = self.class.connection
+              connection.exec_query('LISTEN "skiplock::jobs"')
+              if self.master
+                Job.flush
+                Cron.setup
+              end
               listen = true
             end
             if error
-              unless @connection.active?
-                @connection.reconnect!
+              unless connection.active?
+                connection.reconnect!
                 sleep(0.5)
-                @connection.exec_query('LISTEN "skiplock::jobs"')
+                connection.exec_query('LISTEN "skiplock::jobs"')
+                Job.flush if self.master
+                pg_exception_timestamp = nil
                 next_schedule_at = Time.now.to_f
               end
-              Job.flush if self.master
               error = false
             end
-            if Time.now.to_f >= next_schedule_at && @executor.remaining_capacity > 0
-              job = get_next_available_job
-              if job.try(:running)
-                @executor.post { Rails.application.reloader.wrap { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) } }
+            if Time.now.to_f >= next_schedule_at && @executor.remaining_capacity > 1  # reserves 1 slot in queue for Job.flush in case of pg_connection error
+              result = nil
+              connection.transaction do
+                result = connection.select_all("SELECT id, running, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+                result = connection.select_all("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{result['id']}' RETURNING *").first if result && result['scheduled_at'].to_f <= Time.now.to_f
+              end
+              if result && result['running']
+                @executor.post do
+                  Rails.application.executor.wrap { Job.instantiate(result).execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) }
+                end
               else
-                next_schedule_at = (job ? job.scheduled_at.to_f : Float::INFINITY)
+                next_schedule_at = (result ? result['scheduled_at'].to_f : Float::INFINITY)
               end
             end
             job_notifications = []
-            @connection.raw_connection.wait_for_notify(0.4) do |channel, pid, payload|
+            connection.raw_connection.wait_for_notify(0.2) do |channel, pid, payload|
               job_notifications << payload if payload
               loop do
-                payload = @connection.raw_connection.notifies
+                payload = connection.raw_connection.notifies
                 break unless @running && payload
                 job_notifications << payload[:extra]
               end
@@ -96,22 +99,26 @@ module Skiplock
               end
             end
             if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
-              self.touch
+              connection.exec_query("UPDATE skiplock.workers SET updated_at = NOW() WHERE id = '#{self.id}'")
               timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
             end
           rescue Exception => ex
-            # most likely error with database connection
             Skiplock.logger.error(ex.to_s)
             Skiplock.logger.error(ex.backtrace.join("\n"))
-            Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
+            report_exception = true
+            # if error is with database connection then only report if it persists longer than 1 minute
+            if ex.is_a?(::PG::ConnectionBad) ||ex.is_a?(::PG::UnableToSend) || ex.message.include?('Bad file descriptor')
+              report_exception = false if pg_exception_timestamp.nil? || Process.clock_gettime(Process::CLOCK_MONOTONIC) - pg_exception_timestamp <= 60
+              pg_exception_timestamp ||= Process.clock_gettime(Process::CLOCK_MONOTONIC)
+            end
+            Skiplock.on_errors.each { |p| p.call(ex) } if report_exception
             error = true
             wait(5)
-            @last_exception = ex
           end
           sleep(0.3)
         end
       end
-      @connection.exec_query('UNLISTEN *')
+      connection.exec_query('UNLISTEN *')
     end
 
     def wait(timeout)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.17
+  version: 1.0.22
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-09-12 00:00:00.000000000 Z
+date: 2021-09-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -109,7 +109,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.2.22
 signing_key: 
 specification_version: 4
 summary: ActiveJob Queue Adapter for PostgreSQL