skiplock 1.0.10 → 1.0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 681b41b73f6ddf95620515f3d83454ba1f623d57740248bf84041829c225318d
-  data.tar.gz: 0433b2e635174ae052b67d7502e4b2477c54736f79ec0acf0ad369a1be976256
+  metadata.gz: 432874dc801864f01a8c4de896f330ec62cecdb43d71ededc622ece7d9a17402
+  data.tar.gz: 6034dcb3cfa194b186465a5a84227899c8f00fdf4d62d5b20e0370c9c122cde5
 SHA512:
-  metadata.gz: 6860b6cff70a8881277fd0ce7d2d76d211723e8c1974a1e2b70db7ea31b05268af6d38b68f2437cc9144d0d896751de205e21890219d3664b0f5ae128e1c6ab0
-  data.tar.gz: ddf11fbb1b030dc437254f87142127fd135bacbb84301d03fbcb14e3029bc05736e9c02b1d08b6be8f2d6b90e3fbfdf8eff568c63c8d8c9b451c730a1a125b39
+  metadata.gz: 89f0ef53c0740cf5522e20aa1473ed9020bf5d116054acbb77f451472c1d9af2b123b47cdc593efb4545383829609537a4e2395b4c23c7482f04cff04b11ceab
+  data.tar.gz: 640fa077844855e312b6018d2858cb495915fba0f167c711540ff3b713c66dfc5aa76ca283d9dffaf3d778f37c7025a1f3f7ee3728d4120d87bc25079717bc60

data/README.md

@@ -50,12 +50,12 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     ```yaml
     # config/skiplock.yml (default settings)
     ---
-    extensions: false
-    logging: true
     min_threads: 1
     max_threads: 5
     max_retries: 20
-    notification: none
+    logfile: log/skiplock.log
+    notification: custom
+    extensions: false
     purge_completion: true
     queues:
       default: 200
@@ -63,12 +63,12 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     workers: 0
     ```
     Available configuration options are:
-    - **extensions** (*boolean*): enable or disable the class method extension.  See `ClassMethod extension` for more details
-    - **logging** (*boolean*): enable or disable file logging capability; the log file is stored at log/skiplock.log
     - **min_threads** (*integer*): sets minimum number of threads staying idle
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
-    - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired.  See `Retry System` for more details
-    - **notification** (*enumeration*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, none`).  Using `auto` will attempt to detect available gems in the application
+    - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired.  See `Retry system` for more details
+    - **logfile** (*string*): path filename for skiplock logs; empty logfile will disable logging
+    - **notification** (*string*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, custom`); using `auto` will detect library if available.  See `Notification system` for more details
+    - **extensions** (*boolean*): enable or disable the class method extension.  See `ClassMethod extension` for more details
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
     - **queues** (*hash*): defines the set of queues with priorities; lower priority takes precedence
     - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
@@ -82,7 +82,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     $ bundle exec skiplock -h
     Usage: skiplock [options]
       -e, --environment STRING         Rails environment
-      -l, --logging STRING             Possible values: true, false, timestamp
+      -l, --logfile STRING             Full path to logfile
       -s, --graceful-shutdown NUM      Number of seconds to wait for graceful shutdown
       -r, --max-retries NUM            Number of maxixum retries
       -t, --max-threads NUM            Number of maximum threads
@@ -158,17 +158,17 @@ If the retry attempt limit configured in ActiveJob has been reached, then the co
 If the `retry_on` block is not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
 
 ## Notification system
-`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification` as shown in the **Configuration** section above.  Custom function can also be called whenever an exception occurs; it can be configured in an initializer like below:
+`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  Custom notification can also be called whenever an exception occurs; it can be configured in an initializer like below:
 ```ruby
   # config/initializers/skiplock.rb
   Skiplock.on_error do |ex, previous|
     if ex.backtrace != previous.try(:backtrace)
       # sends custom email on new exceptions only
       # the same repeated exceptions will only be sent once to avoid SPAM
-      # NOTE: exceptions generated from Job perform method executions will not provide 'previous' exceptions
+      # NOTE: exceptions generated from Job executions will not provide 'previous' exceptions
     end
   end
-  # supports multiple on_error callbacks
+  # supports multiple 'on_error' event callbacks
 ```
 ## ClassMethod extension
 `Skiplock` can add extension to allow all class methods to be performed as a background job; it is disabled in the default configuration.  To enable, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`.

data/bin/skiplock

@@ -5,7 +5,7 @@ begin
   op = OptionParser.new do |opts|
     opts.banner = "Usage: #{File.basename($0)} [options]"
     opts.on('-e', '--environment STRING', String, 'Rails environment')
-    opts.on('-l', '--logging STRING', String, 'Possible values: true, false, timestamp')
+    opts.on('-l', '--logfile STRING', String, 'Full path to logfile')
     opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
     opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
     opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')
@@ -25,4 +25,4 @@ options.transform_keys! { |k| k.to_s.gsub('-', '_').to_sym }
 env = options.delete(:environment)
 ENV['RAILS_ENV'] = env if env
 require File.expand_path("config/environment.rb")
-Skiplock::Manager.new(options.merge(standalone: true))
+Skiplock::Manager.new(**options.merge(standalone: true))

data/lib/skiplock/cron.rb

@@ -2,6 +2,7 @@ require 'cron_parser'
 module Skiplock
   class Cron
     def self.setup
+      Rails.application.eager_load! if Rails.env.development?
       cronjobs = []
       ActiveJob::Base.descendants.each do |j|
         next unless j.const_defined?('CRON')

data/lib/skiplock/dispatcher.rb

@@ -7,7 +7,7 @@ module Skiplock
       @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads], max_threads: @config[:max_threads], max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
       @last_dispatch_at = 0
       @next_schedule_at = Time.now.to_f
-      Process.setproctitle("skiplock-#{@worker.master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if @config[:standalone]
+      Process.setproctitle("skiplock-#{@worker.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
     end
     
     def run
@@ -16,7 +16,6 @@ module Skiplock
         ActiveRecord::Base.connection_pool.with_connection do |connection|
           connection.exec_query('LISTEN "skiplock::jobs"')
           if @worker.master
-            Rails.application.eager_load! if Rails.env.development?
             Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
             check_sync_errors
             Cron.setup

data/lib/skiplock/job.rb

@@ -21,13 +21,15 @@ module Skiplock
       rescue Exception => ex
         Skiplock.logger.error(ex)
       end
-      end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-      job_name = job.job_class
-      if job.job_class == 'Skiplock::Extension::ProxyJob'
-        target, method_name = ::YAML.load(job.data['arguments'].first)
-        job_name = "'#{target.name}.#{method_name}'"
+      unless ex
+        end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        job_name = job.job_class
+        if job.job_class == 'Skiplock::Extension::ProxyJob'
+          target, method_name = ::YAML.load(job.data['arguments'].first)
+          job_name = "'#{target.name}.#{method_name}'"
+        end
+        Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{job.id}) from queue '#{job.queue_name || 'default'}' in #{end_time - start_time} seconds"
       end
-      Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{job.id}) from queue '#{job.queue_name || 'default'}' in #{end_time - start_time} seconds"
       job.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
     ensure
       Thread.current[:skiplock_dispatch_job] = nil
@@ -57,6 +59,7 @@ module Skiplock
     def dispose(ex, purge_completion: true, max_retries: 20)
       dup = self.dup
       self.running = false
+      self.worker_id = nil
       self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
       if ex
         self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
@@ -80,6 +83,7 @@ module Skiplock
           self.scheduled_at = Time.at(next_cron_at)
           self.save!
         else
+          Skiplock.logger.error "[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}"
           self.delete
         end
       elsif purge_completion
@@ -90,7 +94,8 @@ module Skiplock
         self.save!
       end
       self
-    rescue
+    rescue Exception => e
+      Skiplock.logger.error(e)
       File.write("tmp/skiplock/#{self.id}", [dup, ex].to_yaml)
       nil
     end

data/lib/skiplock/manager.rb

@@ -7,10 +7,10 @@ module Skiplock
       @config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
       @config.merge!(config)
       Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
-      return unless @config[:standalone] || (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
+      return unless @config[:standalone] || (caller.any?{ |l| l =~ %r{/rack/} } && (@config[:workers] == 0 || Rails.env.development?))
       @config[:hostname] = `hostname -f`.strip
       do_config
-      banner
+      banner if @config[:standalone]
       cleanup_workers
       create_worker
       ActiveJob::Base.logger = nil
@@ -25,27 +25,30 @@ module Skiplock
           @worker.delete
         end
       end
+    rescue Exception => ex
+      @logger.error(ex)
     end
 
   private
 
     def banner
-      title = "[Skiplock] V#{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
+      title = "Skiplock #{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
       @logger.info "-"*(title.length)
       @logger.info title
       @logger.info "-"*(title.length)
-      @logger.info "ClassMethod Extensions: #{@config[:extensions]}"
+      @logger.info "ClassMethod extensions: #{@config[:extensions]}"
       @logger.info "      Purge completion: #{@config[:purge_completion]}"
-      @logger.info "          Notification: #{@config[:notification]}#{(' (' + @notification + ')') if @config[:notification] == 'auto'}"
+      @logger.info "          Notification: #{@config[:notification]}"
       @logger.info "           Max retries: #{@config[:max_retries]}"
       @logger.info "           Min threads: #{@config[:min_threads]}"
       @logger.info "           Max threads: #{@config[:max_threads]}"
       @logger.info "           Environment: #{Rails.env}"
-      @logger.info "               Logging: #{@config[:logging]}"
+      @logger.info "               Logfile: #{@config[:logfile] || '(disabled)'}"
       @logger.info "               Workers: #{@config[:workers]}"
       @logger.info "                Queues: #{@config[:queues].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if @config[:queues].is_a?(Hash)
       @logger.info "                   PID: #{Process.pid}"
       @logger.info "-"*(title.length)
+      @logger.warn "[Skiplock] Custom notification has no registered 'on_error' callback" if Skiplock.on_errors.count == 0
     end
 
     def cleanup_workers
@@ -76,21 +79,31 @@ module Skiplock
       @config[:min_threads] = 0 if @config[:min_threads] < 0
       @config[:workers] = 0 if @config[:workers] < 0
       @config[:workers] = 1 if @config[:standalone] && @config[:workers] <= 0
+      @logger = ActiveSupport::Logger.new(STDOUT)
+      @logger.level = Rails.logger.level
+      Skiplock.logger = @logger
+      raise "Cannot create logfile '#{@config[:logfile]}'" if @config[:logfile] && !File.writable?(File.dirname(@config[:logfile]))
+      @config[:logfile] = nil if @config[:logfile].to_s.length == 0
+      if @config[:logfile]
+        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(@config[:logfile])))
+        if @config[:standalone]
+          Rails.logger.reopen('/dev/null')
+          Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
+        end
+      end
       @config[:queues].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if @config[:queues].is_a?(Hash)
-      @notification = @config[:notification]
-      if @notification == 'auto'
+      if @config[:notification] == 'auto'
         if defined?(Airbrake)
-          @notification = 'airbrake'
+          @config[:notification] = 'airbrake'
         elsif defined?(Bugsnag)
-          @notification = 'bugsnag'
+          @config[:notification] = 'bugsnag'
         elsif defined?(ExceptionNotifier)
-          @notification = 'exception_notification'
+          @config[:notification] = 'exception_notification'
         else
-          @logger.info "Unable to detect any known exception notification gem. Please define custom 'on_error' callback function and disable 'auto' notification in 'config/skiplock.yml'"
-          exit
+          raise "Unable to detect any known exception notification library. Please define custom 'on_error' event callbacks and change to 'custom' notification in 'config/skiplock.yml'"
         end
       end
-      case @notification
+      case @config[:notification]
       when 'airbrake'
         raise 'airbrake gem not found' unless defined?(Airbrake)
         Skiplock.on_error do |ex, previous|
@@ -106,16 +119,8 @@ module Skiplock
         Skiplock.on_error do |ex, previous|
           ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace)
         end
-      end
-      Skiplock.logger = ActiveSupport::Logger.new(STDOUT)
-      Skiplock.logger.level = Rails.logger.level
-      @logger = Skiplock.logger
-      if @config[:logging]
-        Skiplock.logger.extend(ActiveSupport::Logger.broadcast(::Logger.new('log/skiplock.log')))
-        if @config[:standalone]
-          Rails.logger.reopen('/dev/null')
-          Rails.logger.extend(ActiveSupport::Logger.broadcast(Skiplock.logger))
-        end
+      else
+        @config[:notification] = 'custom'
       end
       Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
     end

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.10'
+  VERSION = Version = '1.0.11'
 end
 

data/lib/skiplock.rb

@@ -11,7 +11,7 @@ require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
-  DEFAULT_CONFIG = { 'extensions' => false, 'logging' => true, 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'none', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
+  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'log/skiplock.log', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
 
   def self.logger=(l)
     @logger = l
@@ -28,7 +28,7 @@ module Skiplock
   end
 
   def self.on_errors
-    @on_errors || [].freeze
+    @on_errors || []
   end
 
   def self.table_name_prefix

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.10
+  version: 1.0.11
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-08-29 00:00:00.000000000 Z
+date: 2021-08-30 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob