skiplock 1.0.5 → 1.0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 572dcd7b3a55c259ed181c8d893b38b0d27e682d7647347fd020e1f4cc3b7cd8
-  data.tar.gz: 227e1f528be6ee5c86eb79173f0922fbcf5e436caab3158eafa5c40ed7e658c3
+  metadata.gz: 8155870e392d0172d914c5e10d981560eb05af653603d331ce46cdd3419b2efa
+  data.tar.gz: 940699344273b928820c22307d6cea654d4bdad6debfa80c67fbb28dc8490ea9
 SHA512:
-  metadata.gz: 3ddaba15001823730cf79cd644398ec14db0997d16413e694b4578ac599099e3ff39f754b18b103f6ed0005803e718b050ac2b9b45bb3b095977fef1f1fe46ef
-  data.tar.gz: 5ff4be776646109fad4ffe1476ee982f8fb47ed111e13cb836bb7c07988c9c4a8753ae43cfbd05f8694a2759264a59daedf094cf645325bada99dfa930d5d333
+  metadata.gz: aa423c44556fb2a31cc2ec6cf9f25d6c9712b460920a0d39d0108ade11fa7738fe24e8f1857236cd0a67042ac6ced5cb0b1c34774e8946b72fe971e3202426f7
+  data.tar.gz: ac5456a0613ad3d94e574de100ee26dd018bee39c2a74c16237c798b873a45798519c33b08cb05a26ab18415e15b87252699e2b4bf1978fffbc521008fccdffe

data/README.md

@@ -54,7 +54,11 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     min_threads: 1
     max_threads: 5
     max_retries: 20
+    notification: auto
     purge_completion: true
+    queues:
+      default: 200
+      mailers: 100
     workers: 0
     ```
     Available configuration options are:
@@ -62,8 +66,10 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     - **min_threads** (*integer*): sets minimum number of threads staying idle
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
     - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired (see Retry System for more details)
+    - **notification** (*enumeration*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, false`)
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
-    - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to 0 will enable **async mode**
+    - **queues** (*hash*): defines the set of queues with priorites; lower priority takes precedence
+    - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
     
     #### Async mode
     When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads.  If using multi-worker cluster web server like Puma, then it should be configured as below:
@@ -93,16 +99,22 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     ```
 - Skiplock supports all ActiveJob features:
     ```ruby
-    MyJob.set(wait: 5.minutes, priority: 10).perform_later(1,2,3)
+    MyJob.set(queue: 'my_queue', wait: 5.minutes, priority: 10).perform_later(1,2,3)
     ```
 - Outside of Rails application, queue the jobs by inserting the job records directly to the database table eg:
     ```sql
     INSERT INTO skiplock.jobs(job_class) VALUES ('MyJob');
     ```
-- Or with scheduling, priority and arguments:
+- Or with scheduling, priority, queue and arguments:
     ```sql
-    INSERT INTO skiplock.jobs(job_class,priority,scheduled_at,data) VALUES ('MyJob',10,NOW()+INTERVAL '5 min','{"arguments":[1,2,3]}');
+    INSERT INTO skiplock.jobs(job_class,queue_name,priority,scheduled_at,data) VALUES ('MyJob','my_queue',10,NOW()+INTERVAL '5 min','{"arguments":[1,2,3]}');
     ```
+## Queues priority vs Job priority
+*Why do queues have priorities when jobs already have priorities?*
+- Jobs are only prioritized with other jobs from the same queue
+- Queues, on the other hand, are prioritized with other queues
+- Rails has built-in queues that dispatches jobs without priorities (eg. Mail Delivery will queue as **mailers** with no priority)
+
 ## Cron system
 `Skiplock` provides the capability to setup cron jobs for running tasks periodically.  It fully supports the cron syntax to specify the frequency of the jobs.  To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
 - setup `MyJob` to run as cron job every hour at 30 minutes past
@@ -144,7 +156,16 @@ Once the retry attempt limit configured in ActiveJob has been reached, the contr
 If the rescue blocks are not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
 
 ## Notification system
-...
+`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  A customized function can also be called whenever an exception occurs; this can be configured in an initializer as below:
+```ruby
+  # config/initializers/skiplock.rb
+  Skiplock.on_error = -> (ex, previous = nil) do
+    if ex.backtrace != previous.try(:backtrace)
+      # sends custom email on new exceptions only
+      # the same repeated exceptions will only be sent once to avoid SPAM
+    end
+  end
+```
 
 ## Contributing
 

data/lib/skiplock.rb

@@ -4,22 +4,23 @@ require 'active_record'
 require 'skiplock/cron'
 require 'skiplock/dispatcher'
 require 'skiplock/manager'
-require 'skiplock/notification'
 require 'skiplock/job'
 require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
   Settings = { 
-    'logging' => :timestamp,
+    'logging' => 'timestamp',
     'min_threads' => 1,
     'max_threads' => 5,
     'max_retries' => 20,
+    'notification' => 'auto',
     'purge_completion' => true,
     'queues' => {
-      'default' => 100,
-      'mailers' => 999
+      'default' => 200,
+      'mailers' => 100
     },
     'workers' => 0
   }
+  mattr_accessor :on_error
 end

data/lib/skiplock/dispatcher.rb

@@ -1,7 +1,7 @@
 module Skiplock
   class Dispatcher
     def initialize(master: true, worker_num: nil, worker_pids: [])
-      @queues_order_query = Skiplock::Settings['queues'].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ')
+      @queues_order_query = Settings['queues'].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if Settings['queues'].is_a?(Hash) && Settings['queues'].count > 0
       @executor = Concurrent::ThreadPoolExecutor.new(min_threads: Settings['min_threads'], max_threads: Settings['max_threads'], max_queue: Settings['max_threads'], idletime: 60, auto_terminate: true, fallback_policy: :discard)
       @master = master
       if @master
@@ -17,19 +17,28 @@ module Skiplock
       Thread.new do
         Rails.application.reloader.wrap do
           sleep(0.1) while @running && !Rails.application.initialized?
-          Process.setproctitle("skiplock-#{@master ? 'master' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0
+          Process.setproctitle("skiplock-#{@master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0
           ActiveRecord::Base.connection_pool.with_connection do |connection|
             connection.exec_query('LISTEN "skiplock::jobs"')
-            hostname = `hostname`.strip
+            hostname = `hostname -f`.strip
             @worker = Worker.create!(pid: Process.pid, ppid: (@master ? nil : Process.ppid), capacity: Settings['max_threads'], hostname: hostname)
             if @master
-              # get worker ids that were not shutdown properly on the host
-              dead_worker_ids = Worker.where(hostname: hostname).where.not(pid: @worker_pids).ids
-              if dead_worker_ids.count > 0
-                # reset orphaned jobs of the dead worker ids for retry
-                Job.where(running: true, worker_id: dead_worker_ids).update_all(running: false, worker_id: nil)
-                Worker.where(id: dead_worker_ids).delete_all
+              if File.exists?('tmp/cache/skiplock')
+                # get performed jobs that could not sync with database
+                job_ids = File.read('tmp/cache/skiplock').split("\n")
+                if Settings['purge_completion']
+                  Job.where(id: job_ids, running: true).delete_all
+                else
+                  Job.where(id: job_ids, running: true).update_all(running: false, finished_at: File.mtime('tmp/cache/skiplock'))
+                end
+                File.delete('tmp/cache/skiplock')
               end
+              # get current worker ids
+              worker_ids = Worker.where(hostname: hostname, pid: @worker_pids).ids
+              # reset orphaned jobs of the dead worker ids for retry
+              Job.where(running: true).where.not(worker_id: worker_ids).update_all(running: false, worker_id: nil)
+              # remove workers that were not shutdown properly on the host
+              Worker.where(hostname: hostname).where.not(pid: @worker_pids).delete_all
               # reset retries schedules on startup
               Job.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
               Cron.setup
@@ -46,6 +55,17 @@ module Skiplock
                   end
                   error = false
                 end
+                if Job::Errors.keys.count > 0
+                  completed_ids = Job::Errors.keys.map { |k| k if Job::Errors[k] }.compact
+                  if Settings['purge_completion'] && completed_ids.count > 0
+                    Job.where(id: completed_ids, running: true).delete_all
+                  elsif completed_ids.count > 0
+                    Job.where(id: completed_ids, running: true).update_all(running: false, finished_at: Time.now)
+                  end
+                  orphaned_ids = Job::Errors.keys.map { |k| k unless Job::Errors[k] }.compact
+                  Job.where(id: orphaned_ids, running: true).update_all(running: false, worker_id: nil, scheduled_at: (Time.now + 10)) if orphaned_ids.count > 0
+                  Job::Errors.clear
+                end
                 if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
                   @executor.post { do_work }
                 end
@@ -59,7 +79,7 @@ module Skiplock
                   end
                   notifications.each do |n|
                     op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
-                    next if op == 'DELETE' || running == 'true' || expired_at || finished_at
+                    next if op == 'DELETE' || running == 'true' || expired_at.to_s.length > 0 || finished_at.to_s.length > 0
                     if scheduled_at.to_f <= Time.now.to_f
                       @next_schedule_at = Time.now.to_f
                     elsif scheduled_at.to_f < @next_schedule_at
@@ -68,15 +88,18 @@ module Skiplock
                   end
                 end
               rescue Exception => ex
-                # TODO: Report exception
+                STDERR.puts ex.message
+                STDERR.puts ex.backtrace
+                Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
                 error = true
                 t = Time.now
                 while @running
                   sleep(0.5)
                   break if Time.now - t > 5
                 end              
+                @last_exception = ex
               end
-              sleep(0.1)
+              sleep(0.2)
             end
             connection.exec_query('UNLISTEN *')
           end
@@ -100,9 +123,11 @@ module Skiplock
         @next_schedule_at = result if result.is_a?(Float)
         break
       end
-    rescue Exception => e
-      puts "Exception => #{e.message}   #{e.backtrace}"
-      # TODO: Report exception
+    rescue Exception => ex
+      STDERR.puts ex.message
+      STDERR.puts ex.backtrace
+      Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
+      @last_exception = ex
     end
   end
 end

data/lib/skiplock/job.rb

@@ -1,11 +1,13 @@
 module Skiplock
   class Job < ActiveRecord::Base
     self.table_name = 'skiplock.jobs'
+    Errors = Concurrent::Map.new
 
     # Return: Skiplock::Job if it was executed; otherwise returns the next Job's schedule time in FLOAT
     def self.dispatch(queues_order_query: nil, worker_id: nil)
+      performed = false
       self.connection.exec_query('BEGIN')
-      job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? 'CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+      job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
       if job.nil? || job.scheduled_at.to_f > Time.now.to_f
         self.connection.exec_query('END')
         return (job ? job.scheduled_at.to_f : Float::INFINITY)
@@ -21,9 +23,9 @@ module Skiplock
         ActiveJob::Base.execute(job_data)
       rescue Exception => ex
       end
+      performed = true
       job.running = false
       if ex
-        # TODO: report exception
         job.exception_executions["[#{ex.class.name}]"] = (job.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless job.exception_executions.key?('activejob_retry')
         if job.executions >= Settings['max_retries'] || job.exception_executions.key?('activejob_retry')
           job.expired_at = Time.now
@@ -32,11 +34,12 @@ module Skiplock
           job.scheduled_at = Time.now + (5 * 2**job.executions)
           job.save!
         end
+        Skiplock.on_error.call(ex) if Skiplock.on_error.is_a?(Proc) && (job.executions % 3 == 1)
       elsif job.exception_executions.key?('activejob_retry')
         job.save!
-      elsif job['cron']
+      elsif job.cron
         job.data['last_cron_run'] = Time.now.utc.to_s
-        next_cron_at = Cron.next_schedule_at(job['cron'])
+        next_cron_at = Cron.next_schedule_at(job.cron)
         if next_cron_at
           job.executions = 1
           job.exception_executions = nil
@@ -53,6 +56,14 @@ module Skiplock
         job.save!
       end
       job
+    rescue Exception => ex
+      if performed
+        Errors[job.id] = true
+        File.write('tmp/cache/skiplock', job.id + "\n", mode: 'a')
+      else
+        Errors[job.id] = false
+      end
+      raise ex
     ensure
       Thread.current[:skiplock_dispatch_job] = nil
     end

data/lib/skiplock/manager.rb

@@ -1,7 +1,21 @@
 module Skiplock
   class Manager
-    def self.start(standalone: false)
-      load_settings
+    def self.start(standalone: false, workers: nil, max_retries: nil, max_threads: nil, min_threads: nil, logging: nil)
+      unless Settings.frozen?
+        load_settings
+        Settings['logging'] = logging if logging
+        Settings['max_retries'] = max_retries if max_retries
+        Settings['max_threads'] = max_threads if max_threads
+        Settings['min_threads'] = min_threads if min_threads
+        Settings['workers'] = workers if workers
+        Settings['max_retries'] = 20 if Settings['max_retries'] > 20
+        Settings['max_retries'] = 0 if Settings['max_retries'] < 0
+        Settings['max_threads'] = 1 if Settings['max_threads'] < 1
+        Settings['max_threads'] = 20 if Settings['max_threads'] > 20
+        Settings['min_threads'] = 0 if Settings['min_threads'] < 0
+        Settings['workers'] = 0 if Settings['workers'] < 0
+        Settings.freeze
+      end
       return unless standalone || (caller.any?{|l| l =~ %r{/rack/}} && (Settings['workers'] == 0 || Rails.env.development?))
       if standalone
         self.standalone
@@ -22,37 +36,37 @@ module Skiplock
   private
 
     def self.load_settings
-      return if Settings.frozen?
       config = YAML.load_file('config/skiplock.yml') rescue {}
       Settings.merge!(config)
-      Settings['max_retries'] = 20 if Settings['max_retries'] > 20
-      Settings['max_retries'] = 0 if Settings['max_retries'] < 0
-      Settings['max_threads'] = 1 if Settings['max_threads'] < 1
-      Settings['max_threads'] = 20 if Settings['max_threads'] > 20
-      Settings['min_threads'] = 0 if Settings['min_threads'] < 0
-      Settings['workers'] = 0 if Settings['workers'] < 0
-      Settings['queues'].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) }
-      Settings.freeze
+      Settings['queues'].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if Settings['queues'].is_a?(Hash)
+      case Settings['notification'].to_s.downcase
+      when 'auto'
+        if defined?(Airbrake)
+          Skiplock.on_error = -> (ex, previous = nil) { Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace) }
+        elsif defined?(Bugsnag)
+          Skiplock.on_error = -> (ex, previous = nil) { Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace) }
+        elsif defined?(ExceptionNotifier)
+          Skiplock.on_error = -> (ex, previous = nil) { ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace) }
+        else
+          puts "Unable to detect any known exception notification gem. Please define custom 'on_error' function and disable notification in 'config/skiplock.yml'"
+          exit
+        end
+      when 'airbrake'
+        raise 'airbrake gem not found' unless defined?(Airbrake)
+        Skiplock.on_error = -> (ex, previous = nil) { Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace) }
+      when 'bugsnag'
+        raise 'bugsnag gem not found' unless defined?(Bugsnag)
+        Skiplock.on_error = -> (ex, previous = nil) { Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace) }
+      when 'exception_notification'
+        raise 'exception_notification gem not found' unless defined?(ExceptionNotifier)
+        Skiplock.on_error = -> (ex, previous = nil) { ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace) }
+      end
     rescue Exception => e
       STDERR.puts "Invalid configuration 'config/skiplock.yml': #{e.message}"
       exit
     end
 
     def self.standalone
-      title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
-      puts "-"*(title.length)
-      puts title
-      puts "-"*(title.length)
-      puts "Additional workers: #{Settings['workers']}"
-      puts "  Purge completion: #{Settings['purge_completion']}"
-      puts "       Max retries: #{Settings['max_retries']}"
-      puts "       Min threads: #{Settings['min_threads']}"
-      puts "       Max threads: #{Settings['max_threads']}"
-      puts "       Environment: #{Rails.env}"
-      puts "           Logging: #{Settings['logging']}"
-      puts "            Queues: #{Settings['queues'].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}"
-      puts "               PID: #{Process.pid}"
-      puts "-"*(title.length)
       if Settings['logging']
         log_timestamp = (Settings['logging'].to_s == 'timestamp')
         logfile = File.open('log/skiplock.log', 'a')
@@ -66,14 +80,29 @@ module Skiplock
         Rails.logger.reopen('/dev/null')
         Rails.logger.extend(ActiveSupport::Logger.broadcast(logger))
       end
+      title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
+      puts "-"*(title.length)
+      puts title
+      puts "-"*(title.length)
+      puts "Purge completion: #{Settings['purge_completion']}"
+      puts "    Notification: #{Settings['notification']}"
+      puts "     Max retries: #{Settings['max_retries']}"
+      puts "     Min threads: #{Settings['min_threads']}"
+      puts "     Max threads: #{Settings['max_threads']}"
+      puts "     Environment: #{Rails.env}"
+      puts "         Logging: #{Settings['logging']}"
+      puts "         Workers: #{Settings['workers']}"
+      puts "          Queues: #{Settings['queues'].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if Settings['queues'].is_a?(Hash)
+      puts "             PID: #{Process.pid}"
+      puts "-"*(title.length)
       parent_id = Process.pid
       shutdown = false
       Signal.trap("INT") { shutdown = true }
       Signal.trap("TERM") { shutdown = true }
       worker_pids = []
-      Settings['workers'].times do |n|
+      (Settings['workers']-1).times do |n|
         worker_pids << fork do
-          dispatcher = Dispatcher.new(master: false, worker_num: n)
+          dispatcher = Dispatcher.new(master: false, worker_num: n+1)
           thread = dispatcher.run
           loop do
             sleep 0.5

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.5'
+  VERSION = Version = '1.0.6'
 end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.5
+  version: 1.0.6
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-03-22 00:00:00.000000000 Z
+date: 2021-03-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -86,7 +86,6 @@ files:
 - lib/skiplock/dispatcher.rb
 - lib/skiplock/job.rb
 - lib/skiplock/manager.rb
-- lib/skiplock/notification.rb
 - lib/skiplock/version.rb
 - lib/skiplock/worker.rb
 homepage: https://github.com/vtt/skiplock

data/lib/skiplock/notification.rb

@@ -1,5 +0,0 @@
-module Skiplock
-  class Notification
-  end
-end
-