skiplock 1.0.4 → 1.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c53cdb42608339ffd7ba7532393e933ce9128757ee35679a665fd2f1e1d0fba3
-  data.tar.gz: e7dfcc14eed82fef12747d9c0172de006cdb8823fdcaa1d7e944b8d0bde71be1
+  metadata.gz: 7a975a3bb22318c6203edd59428a93d6a4abfa5ea1382f35abd774760467dc23
+  data.tar.gz: 79b787d23af0db0188aebf1b0361649d906d5f0cca4488d7cfcb3f61927d4781
 SHA512:
-  metadata.gz: b32ac8c7b3cf90be24c86310d4e2b2fe250a5ec7b244aad267144f49e09f14d9a46e64214f45afde8093d4a867f1c93245c5e216ac214cacca555f3965770f7c
-  data.tar.gz: 0a61bf8d14f3162fe981420d43278642ac98a4ab5ac6705bc9a9f65ddd3330e26b6ec1d3ce9a196155e2316ed972954cd657f4d36a48bda83554ea97eb205237
+  metadata.gz: 407469700b1fab6c4ea702b83df0571fd46322ad32c8c8fe8ffc0269e5e93cd9f774b8bd4658021320732ca4de6d7609a64430cbc26bddc14c9f14cf184f92e3
+  data.tar.gz: 69a5fe7f5c88f21585a4b581d34f44183cb12f25bc0df6e7982497fb63c088316a251cc55c4e712b3cad799ec9aa758c8045d911fbeda393267ed39bc694e9a5

data/README.md

@@ -54,7 +54,11 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     min_threads: 1
     max_threads: 5
     max_retries: 20
+    notification: auto
     purge_completion: true
+    queues:
+      default: 200
+      mailers: 100
     workers: 0
     ```
     Available configuration options are:
@@ -62,47 +66,51 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     - **min_threads** (*integer*): sets minimum number of threads staying idle
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
     - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired (see Retry System for more details)
+    - **notification** (*enumeration*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, false`)
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
-    - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to 0 will enable **async mode**
+    - **queues** (*hash*): defines the set of queues with priorities; lower priority takes precedence
+    - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
     
     #### Async mode
-    When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads.  If using multi-worker cluster web server like Puma, then it should be configured as below:
+    When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads.  If using multi-worker cluster mode web server like Puma, then it should be configured as below:
     ```ruby
     # config/puma.rb
-    before_fork do
-      # ...
-      Skiplock::Manager.shutdown
-    end
-
-    on_worker_boot do
-      # ...
-      Skiplock::Manager.start
+    # ...
+    on_worker_fork do |worker_index|
+      Skiplock::Manager.shutdown if worker_index == 1
     end
 
-    on_worker_shutdown do
-      # ...
-      Skiplock::Manager.shutdown
+    after_worker_fork do |worker_index|
+      # restarts skiplock after all Puma workers have been started
+      Skiplock::Manager.start(restart: true) if defined?(Skiplock) && worker_index + 1 == @options[:workers]
     end
     ```
-
 ## Usage
-
-- Inside the Rails application, queue your job:
+Inside the Rails application:
+- queue your job
     ```ruby
     MyJob.perform_later
     ```
-- Skiplock supports all ActiveJob features:
+- Skiplock supports all ActiveJob features
     ```ruby
-    MyJob.set(wait: 5.minutes, priority: 10).perform_later(1,2,3)
+    MyJob.set(queue: 'my_queue', wait: 5.minutes, priority: 10).perform_later(1,2,3)
     ```
-- Outside of Rails application, queue the jobs by inserting the job records directly to the database table eg:
+Outside the Rails application:
+- queue the jobs by inserting the job records directly to the database table
     ```sql
     INSERT INTO skiplock.jobs(job_class) VALUES ('MyJob');
     ```
-- Or with scheduling, priority and arguments:
+- with scheduling, priority, queue and arguments
     ```sql
-    INSERT INTO skiplock.jobs(job_class,priority,scheduled_at,data) VALUES ('MyJob',10,NOW()+INTERVAL '5 min','{"arguments":[1,2,3]}');
+    INSERT INTO skiplock.jobs(job_class, queue_name, priority, scheduled_at, data)
+      VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min', '{"arguments":[1,2,3]}');
     ```
+## Queue priority vs Job priority
+*Why do queues use priorities when jobs already have priorities?*
+- Jobs are only prioritized with other jobs from the same queue
+- Queues, on the other hand, are prioritized with other queues
+- Rails has built-in queues that dispatch jobs without priorities (eg. Mail Delivery will queue as **mailers** with no priority)
+
 ## Cron system
 `Skiplock` provides the capability to setup cron jobs for running tasks periodically.  It fully supports the cron syntax to specify the frequency of the jobs.  To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
 - setup `MyJob` to run as cron job every hour at 30 minutes past
@@ -123,7 +131,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
 - to remove the cron schedule from the job, simply comment out the constant definition or delete the line then re-deploy the application. At startup, the cron jobs that were undefined will be removed automatically
 
 ## Retry system
-`Skiplock` fully supports ActiveJob built-in retry system.  It also has its own retry system for fallback.  To use ActiveJob retry system, define the rescue blocks per ActiveJob's documentation.
+`Skiplock` fully supports ActiveJob built-in retry system.  It also has its own retry system for fallback.  To use ActiveJob retry system, define the `retry_on` block per ActiveJob's documentation.
 - configures `MyJob` to retry at maximum 20 attempts on StandardError with fixed delay of 5 seconds
     ```ruby
     class MyJob < ActiveJob::Base
@@ -139,12 +147,22 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     # ...
     end
     ```
-Once the retry attempt limit configured in ActiveJob has been reached, the control will be passed back to `skiplock` to be marked as an expired job.
+If the retry attempt limit configured in ActiveJob has been reached, then the control will be passed back to `skiplock` to be marked as an expired job.
 
-If the rescue blocks are not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
+If the `retry_on` block is not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
 
 ## Notification system
-...
+`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification` as shown in the **Configuration** section above.  Custom function can also be called whenever an exception occurs; it can be configured in an initializer like below:
+```ruby
+  # config/initializers/skiplock.rb
+  Skiplock.on_error do |ex, previous|
+    if ex.backtrace != previous.try(:backtrace)
+      # sends custom email on new exceptions only
+      # the same repeated exceptions will only be sent once to avoid SPAM
+    end
+  end
+  # supports multiple on_error callbacks
+```
 
 ## Contributing
 

data/bin/skiplock

@@ -1,3 +1,26 @@
 #!/usr/bin/env ruby
+require 'optparse'
+options = {}
+begin
+  op = OptionParser.new do |opts|
+    opts.banner = "Usage: #{File.basename($0)} [options]"
+    opts.on('-e', '--environment STRING', String, 'Rails environment')
+    opts.on('-l', '--logging STRING', String, 'Possible values: true, false, timestamp')
+    opts.on('-r', '--max_retries NUM', Integer, 'Number of maxixum retries')
+    opts.on('-t', '--max_threads NUM', Integer, 'Number of maximum threads')
+    opts.on('-T', '--min_threads NUM', Integer, 'Number of minimum threads')
+    opts.on('-w', '--workers NUM', Integer, 'Number of workers')
+    opts.on_tail('-h', '--help', 'Show this message') do
+      exit
+    end
+  end
+  op.parse!(into: options)
+rescue Exception => e
+  puts "\n#{e.message}\n\n" unless e.is_a?(SystemExit)
+  puts op
+  exit
+end
+env = options.delete(:environment)
+ENV['RAILS_ENV'] = env if env
 require File.expand_path("config/environment.rb")
-Skiplock::Manager.start(standalone: true)
+Skiplock::Manager.start(standalone: true, **options)

data/lib/active_job/queue_adapters/skiplock_adapter.rb

@@ -6,7 +6,7 @@ module ActiveJob
       end
 
       def enqueue(job)
-        enqueue_at(job, nil)
+        Skiplock::Job.enqueue(job)
       end
 
       def enqueue_at(job, timestamp)

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -1,13 +1,21 @@
 class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSION::MAJOR}.#{ActiveRecord::VERSION::MINOR}]" %>
   def up
     execute 'CREATE SCHEMA skiplock'
+    create_table 'skiplock.counters', id: :uuid do |t|
+      t.integer   :completions, null: false, default: 0
+      t.integer   :dispatches, null: false, default: 0
+      t.integer   :expiries, null: false, default: 0
+      t.integer   :failures, null: false, default: 0
+      t.integer   :retries, null: false, default: 0
+      t.date      :day, null: false, index: { unique: true }
+    end
     create_table 'skiplock.jobs', id: :uuid do |t|
       t.uuid      :worker_id, index: true
       t.string    :job_class, null: false
-      t.string    :cron
-      t.string    :queue_name
+      t.string    :queue_name, index: true
       t.string    :locale
       t.string    :timezone
+      t.string    :cron
       t.integer   :priority
       t.integer   :executions
       t.jsonb     :exception_executions
@@ -18,26 +26,63 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
       t.timestamp :scheduled_at
       t.timestamps null: false, default: -> { 'now()' }
     end
-    create_table 'skiplob.workers', id: :uuid do |t|
+    create_table 'skiplock.workers', id: :uuid do |t|
       t.integer   :pid, null: false, index: true
       t.integer   :ppid, index: true
-      t.integer   :remaining_capacity, null: false
+      t.integer   :capacity, null: false
       t.string    :hostname, null: false, index: true
       t.jsonb     :data
       t.timestamps null: false, index: true, default: -> { 'now()' }
     end    
-    execute %(CREATE OR REPLACE FUNCTION skiplock.notify() RETURNS TRIGGER AS $$
-      BEGIN
-        IF (NEW.finished_at IS NULL AND NEW.expired_at IS NULL) THEN
-          PERFORM pg_notify('skiplock', CONCAT(TG_OP,',',NEW.id::TEXT,',',NEW.queue_name,',',NEW.priority,',',CAST(EXTRACT(EPOCH FROM NEW.scheduled_at) AS FLOAT)::text));
-        END IF;
-        RETURN NULL;
-      END;
+    execute <<~ENDFUNC
+      CREATE OR REPLACE FUNCTION skiplock.notify_jobs() RETURNS TRIGGER AS $$
+        DECLARE
+          record RECORD;
+        BEGIN
+          record = NEW;
+          IF (TG_OP = 'DELETE') THEN
+            record = OLD;
+            IF (record.finished_at IS NOT NULL OR record.expired_at IS NOT NULL) THEN
+              RETURN NULL;
+            END IF;
+            INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
+          ELSIF (record.running = TRUE) THEN
+            IF (record.executions IS NULL) THEN
+              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
+            ELSE
+              INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
+            END IF;
+          ELSIF (record.finished_at IS NOT NULL) THEN
+            INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
+          ELSIF (record.expired_at IS NOT NULL) THEN
+            INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
+          ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
+            INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
+          END IF;
+          PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
+          RETURN NULL;
+        END;
       $$ LANGUAGE plpgsql
-    )
-    execute "CREATE TRIGGER notify_job AFTER INSERT OR UPDATE ON skiplock.jobs FOR EACH ROW EXECUTE PROCEDURE skiplock.notify()"
-    execute "CREATE INDEX jobs_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE expired_at IS NULL AND finished_at IS NULL"
-    execute "CREATE INDEX jobs_retry_index ON skiplock.jobs(scheduled_at) WHERE executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL"
+    ENDFUNC
+    execute <<~ENDFUNC
+      CREATE OR REPLACE FUNCTION skiplock.notify_workers() RETURNS TRIGGER AS $$
+        DECLARE
+          record RECORD;
+        BEGIN
+          IF (TG_OP = 'DELETE') THEN
+            record = OLD;
+          ELSE
+            record = NEW;
+          END IF;
+          PERFORM pg_notify('skiplock::workers', CONCAT(TG_OP,',',record.id::TEXT,',',record.hostname,',',record.capacity,',',record.pid,',',record.ppid));
+          RETURN NULL;
+        END;
+      $$ LANGUAGE plpgsql;
+    ENDFUNC
+    execute "CREATE TRIGGER notify_job AFTER INSERT OR UPDATE OR DELETE ON skiplock.jobs FOR EACH ROW EXECUTE PROCEDURE skiplock.notify_jobs()"
+    execute "CREATE TRIGGER notify_worker AFTER INSERT OR UPDATE OR DELETE ON skiplock.workers FOR EACH ROW EXECUTE PROCEDURE skiplock.notify_workers()"
+    execute "CREATE INDEX jobs_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL"
+    execute "CREATE INDEX jobs_retry_index ON skiplock.jobs(scheduled_at) WHERE running = FALSE AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL"
     execute "CREATE INDEX jobs_cron_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE cron IS NOT NULL AND finished_at IS NULL"
     execute "CREATE UNIQUE INDEX jobs_unique_cron_index ON skiplock.jobs (job_class) WHERE cron IS NOT NULL"
   end

data/lib/skiplock.rb

@@ -1,21 +1,35 @@
 require 'active_job'
 require 'active_job/queue_adapters/skiplock_adapter'
 require 'active_record'
+require 'skiplock/counter'
 require 'skiplock/cron'
 require 'skiplock/dispatcher'
-require 'skiplock/manager'
-require 'skiplock/notification'
 require 'skiplock/job'
+require 'skiplock/manager'
 require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
   Settings = { 
-    'logging' => :timestamp,
+    'logging' => 'timestamp',
     'min_threads' => 1,
     'max_threads' => 5,
     'max_retries' => 20,
+    'notification' => 'auto',
     'purge_completion' => true,
+    'queues' => {
+      'default' => 200,
+      'mailers' => 100
+    },
     'workers' => 0
   }
+  mattr_reader :on_errors, default: []
+
+  def self.on_error(&block)
+    @@on_errors << block
+  end
+
+  def self.table_name_prefix
+    'skiplock.'
+  end
 end

data/lib/skiplock/counter.rb

@@ -0,0 +1,4 @@
+module Skiplock
+  class Counter < ActiveRecord::Base
+  end
+end

data/lib/skiplock/dispatcher.rb

@@ -1,6 +1,7 @@
 module Skiplock
   class Dispatcher
     def initialize(master: true, worker_num: nil, worker_pids: [])
+      @queues_order_query = Settings['queues'].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if Settings['queues'].is_a?(Hash) && Settings['queues'].count > 0
       @executor = Concurrent::ThreadPoolExecutor.new(min_threads: Settings['min_threads'], max_threads: Settings['max_threads'], max_queue: Settings['max_threads'], idletime: 60, auto_terminate: true, fallback_policy: :discard)
       @master = master
       if @master
@@ -8,6 +9,7 @@ module Skiplock
       else
         @worker_num = worker_num
       end
+      @last_dispatch_at = 0
       @next_schedule_at = Time.now.to_f
       @running = true
     end
@@ -15,18 +17,23 @@ module Skiplock
     def run
       Thread.new do
         Rails.application.reloader.wrap do
-          sleep(0.1) while @running && !Rails.application.initialized?
-          Process.setproctitle("skiplock-#{@master ? 'master' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0
+          sleep(1) while @running && !Rails.application.initialized?
+          Rails.application.eager_load!
+          Process.setproctitle("skiplock-#{@master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0 && !Rails.env.development?
           ActiveRecord::Base.connection_pool.with_connection do |connection|
             connection.exec_query('LISTEN "skiplock::jobs"')
-            hostname = `hostname`.strip
+            hostname = `hostname -f`.strip
             @worker = Worker.create!(pid: Process.pid, ppid: (@master ? nil : Process.ppid), capacity: Settings['max_threads'], hostname: hostname)
             if @master
-              # get worker ids that were not shutdown properly on the host
+              connection.exec_query('LISTEN "skiplock::workers"')
+              Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
+              check_sync_errors
+              # get dead worker ids
               dead_worker_ids = Worker.where(hostname: hostname).where.not(pid: @worker_pids).ids
               if dead_worker_ids.count > 0
                 # reset orphaned jobs of the dead worker ids for retry
-                Job.where(running: true, worker_id: dead_worker_ids).update_all(running: false, worker_id: nil)
+                Job.where(running: true).where(worker_id: dead_worker_ids).update_all(running: false, worker_id: nil)
+                # remove dead workers
                 Worker.where(id: dead_worker_ids).delete_all
               end
               # reset retries schedules on startup
@@ -41,43 +48,60 @@ module Skiplock
                     connection.reconnect!
                     sleep(0.5)
                     connection.exec_query('LISTEN "skiplock::jobs"')
-                    @next_schedule_at = Time.now
+                    connection.exec_query('LISTEN "skiplock::workers"') if @master
+                    @next_schedule_at = Time.now.to_f
                   end
+                  check_sync_errors
                   error = false
                 end
-                if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
-                  @executor.post { do_work }
-                end
-                notifications = []
+                notifications = { 'skiplock::jobs' => [], 'skiplock::workers' => [] }
                 connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
-                  notifications << payload if payload
+                  notifications[channel] << payload if payload
                   loop do
                     payload = connection.raw_connection.notifies
                     break unless @running && payload
-                    notifications << payload[:extra]
+                    notifications[payload[:relname]] << payload[:extra]
                   end
-                  notifications.each do |n|
+                  notifications['skiplock::jobs'].each do |n|
                     op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
-                    next if op == 'DELETE' || running == 'true' || expired_at || finished_at
-                    if scheduled_at.to_f <= Time.now.to_f
+                    if @master
+                      # TODO: report job status to action cable
+                    end
+                    next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0  || scheduled_at.to_f < @last_dispatch_at
+                    if scheduled_at.to_f < Time.now.to_f
                       @next_schedule_at = Time.now.to_f
                     elsif scheduled_at.to_f < @next_schedule_at
-                      @next_schedule_at = time.to_f
+                      @next_schedule_at = scheduled_at.to_f
                     end
                   end
+                  if @master
+                    # TODO: report worker status to action cable
+                    notifications['skiplock::workers'].each do |n|
+                    end
+                  end
+                end
+                if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
+                  @executor.post { do_work }
                 end
               rescue Exception => ex
-                # TODO: Report exception
+                # most likely error with database connection
+                STDERR.puts ex.message
+                STDERR.puts ex.backtrace
+                Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
                 error = true
                 t = Time.now
                 while @running
                   sleep(0.5)
                   break if Time.now - t > 5
-                end              
+                end
+                @last_exception = ex
               end
-              sleep(0.1)
+              sleep(0.2)
             end
             connection.exec_query('UNLISTEN *')
+            @executor.shutdown
+            @executor.wait_for_termination if @wait
+            @worker.delete if @worker
           end
         end
       end
@@ -85,23 +109,37 @@ module Skiplock
 
     def shutdown(wait: true)
       @running = false
-      @executor.shutdown
-      @executor.wait_for_termination if wait
-      @worker.delete if @worker
+      @wait = wait
     end
 
     private
 
+    def check_sync_errors
+      # get performed jobs that could not sync with database
+      Dir.glob('tmp/skiplock/*').each do |f|
+        job_from_db = Job.find_by(id: File.basename(f), running: true)
+        disposed = true
+        if job_from_db
+          job, ex = YAML.load_file(f) rescue nil
+          disposed = job.dispose(ex)
+        end
+        File.delete(f) if disposed
+      end
+    end
+
     def do_work
       while @running
-        result = Job.dispatch(worker_id: @worker.id)
-        next if result.is_a?(Hash)
+        @last_dispatch_at = Time.now.to_f - 1  # 1 second allowance for timedrift
+        result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id)
+        next if result.is_a?(Job)
         @next_schedule_at = result if result.is_a?(Float)
         break
       end
-    rescue Exception => e
-      puts "Exception => #{e.message}   #{e.backtrace}"
-      # TODO: Report exception
+    rescue Exception => ex
+      STDERR.puts ex.message
+      STDERR.puts ex.backtrace
+      Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
+      @last_exception = ex
     end
   end
 end

data/lib/skiplock/job.rb

@@ -1,16 +1,13 @@
 module Skiplock
   class Job < ActiveRecord::Base
-    self.table_name = 'skiplock.jobs'
-
-    # Return: Attributes hash of the Job if it was executed; otherwise returns the next Job's schedule time in FLOAT
-    def self.dispatch(worker_id: nil)
+    def self.dispatch(queues_order_query: nil, worker_id: nil)
       self.connection.exec_query('BEGIN')
-      job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+      job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
       if job.nil? || job.scheduled_at.to_f > Time.now.to_f
         self.connection.exec_query('END')
         return (job ? job.scheduled_at.to_f : Float::INFINITY)
       end
-      job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)} WHERE id = '#{job.id}' RETURNING *").first
+      job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
       self.connection.exec_query('END')
       job.data ||= {}
       job.exception_executions ||= {}
@@ -21,40 +18,13 @@ module Skiplock
         ActiveJob::Base.execute(job_data)
       rescue Exception => ex
       end
-      job.running = false
-      if ex
-        # TODO: report exception
-        job.exception_executions["[#{ex.class.name}]"] = (job.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless job.exception_executions.key?('activejob_retry')
-        if job.executions >= Settings['max_retries'] || job.exception_executions.key?('activejob_retry')
-          job.expired_at = Time.now
-          job.save!
-        else
-          job.scheduled_at = Time.now + (5 * 2**job.executions)
-          job.save!
-        end
-      elsif job.exception_executions.key?('activejob_retry')
-        job.save!
-      elsif job['cron']
-        job.data['last_cron_run'] = Time.now.utc.to_s
-        next_cron_at = Cron.next_schedule_at(job['cron'])
-        if next_cron_at
-          job.executions = 1
-          job.exception_executions = nil
-          job.scheduled_at = Time.at(next_cron_at)
-          job.save!
-        else
-          job.delete
-        end
-      elsif Settings['purge_completion']
-        job.delete
-      else
-        job.finished_at = Time.now
-        job.exception_executions = nil
-        job.save!
-      end
-      job
+      job.dispose(ex)
     ensure
-      Thread.current[:skiplock_dispatch_data] = nil
+      Thread.current[:skiplock_dispatch_job] = nil
+    end
+
+    def self.enqueue(activejob)
+      self.enqueue_at(activejob, nil)
     end
 
     def self.enqueue_at(activejob, timestamp)
@@ -65,8 +35,49 @@ module Skiplock
         Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
         Thread.current[:skiplock_dispatch_job]
       else
-        Job.create!(id: activejob.job_id, job_class: activejob.class.name, queue_name: activejob.queue_name, locale: activejob.locale, timezone: activejob.timezone, priority: activejob.priority, executions: activejob.executions, data: { 'arguments' => activejob.serialize['arguments'] }, scheduled_at: timestamp)
+        Job.create!(id: activejob.job_id, job_class: activejob.class.name, queue_name: activejob.queue_name, locale: activejob.locale, timezone: activejob.timezone, priority: activejob.priority, data: { 'arguments' => activejob.serialize['arguments'] }, scheduled_at: timestamp)
+      end
+    end
+
+    def dispose(ex)
+      self.running = false
+      self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
+      if ex
+        self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
+        if self.executions >= Settings['max_retries'] || self.exception_executions.key?('activejob_retry')
+          self.expired_at = Time.now
+          self.save!
+        else
+          self.scheduled_at = Time.now + (5 * 2**self.executions)
+          self.save!
+        end
+        Skiplock.on_errors.each { |p| p.call(ex) }
+      elsif self.exception_executions.try(:key?, 'activejob_retry')
+        self.save!
+      elsif self.cron
+        self.data ||= {}
+        self.data['crons'] = (self.data['crons'] || 0) + 1
+        self.data['last_cron_at'] = Time.now.utc.to_s
+        next_cron_at = Cron.next_schedule_at(self.cron)
+        if next_cron_at
+          self.executions = nil
+          self.exception_executions = nil
+          self.scheduled_at = Time.at(next_cron_at)
+          self.save!
+        else
+          self.delete
+        end
+      elsif Settings['purge_completion']
+        self.delete
+      else
+        self.finished_at = Time.now
+        self.exception_executions = nil
+        self.save!
       end
+      self
+    rescue
+      File.write("tmp/skiplock/#{self.id}", [self, ex].to_yaml)
+      nil
     end
   end
 end

data/lib/skiplock/manager.rb

@@ -1,8 +1,23 @@
 module Skiplock
   class Manager
-    def self.start(standalone: false)
-      load_settings
-      return unless standalone || (caller.any?{|l| l =~ %r{/rack/}} && (Settings['workers'] == 0 || Rails.env.development?))
+    def self.start(standalone: false, restart: false, workers: nil, max_retries: nil, max_threads: nil, min_threads: nil, logging: nil)
+      unless Settings.frozen?
+        load_settings
+        Settings['logging'] = logging if logging
+        Settings['max_retries'] = max_retries if max_retries
+        Settings['max_threads'] = max_threads if max_threads
+        Settings['min_threads'] = min_threads if min_threads
+        Settings['workers'] = workers if workers
+        Settings['max_retries'] = 20 if Settings['max_retries'] > 20
+        Settings['max_retries'] = 0 if Settings['max_retries'] < 0
+        Settings['max_threads'] = 1 if Settings['max_threads'] < 1
+        Settings['max_threads'] = 20 if Settings['max_threads'] > 20
+        Settings['min_threads'] = 0 if Settings['min_threads'] < 0
+        Settings['workers'] = 0 if Settings['workers'] < 0
+        Settings['workers'] = 1 if standalone && Settings['workers'] <= 0
+        Settings.freeze
+      end
+      return unless standalone || restart || (caller.any?{|l| l =~ %r{/rack/}} && Settings['workers'] == 0)
       if standalone
         self.standalone
       else
@@ -10,44 +25,60 @@ module Skiplock
         @thread = @dispatcher.run
         at_exit { self.shutdown }
       end
+      ActiveJob::Base.logger = nil
     end
     
     def self.shutdown(wait: true)
       if @dispatcher && @thread
         @dispatcher.shutdown(wait: wait)
         @thread.join
+        @dispatcher = nil
+        @thread = nil
       end
     end
 
   private
 
     def self.load_settings
-      return if Settings.frozen?
       config = YAML.load_file('config/skiplock.yml') rescue {}
       Settings.merge!(config)
-      Settings['max_retries'] = 20 if Settings['max_retries'] > 20
-      Settings['max_retries'] = 0 if Settings['max_retries'] < 0
-      Settings['max_threads'] = 1 if Settings['max_threads'] < 1
-      Settings['max_threads'] = 20 if Settings['max_threads'] > 20
-      Settings['min_threads'] = 0 if Settings['min_threads'] < 0
-      Settings['workers'] = 0 if Settings['workers'] < 0
-      Settings.freeze
+      Settings['queues'].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if Settings['queues'].is_a?(Hash)
+      @notification = Settings['notification'] = Settings['notification'].to_s.downcase
+      if @notification == 'auto'
+        if defined?(Airbrake)
+          @notification = 'airbrake'
+        elsif defined?(Bugsnag)
+          @notification = 'bugsnag'
+        elsif defined?(ExceptionNotifier)
+          @notification = 'exception_notification'
+        else
+          puts "Unable to detect any known exception notification gem. Please define custom 'on_error' callback function and disable 'auto' notification in 'config/skiplock.yml'"
+          exit
+        end
+      end
+      case @notification
+      when 'airbrake'
+        raise 'airbrake gem not found' unless defined?(Airbrake)
+        Skiplock.on_error do |ex, previous|
+          Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace)
+        end
+      when 'bugsnag'
+        raise 'bugsnag gem not found' unless defined?(Bugsnag)
+        Skiplock.on_error do |ex, previous|
+          Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace)
+        end
+      when 'exception_notification'
+        raise 'exception_notification gem not found' unless defined?(ExceptionNotifier)
+        Skiplock.on_error do |ex, previous|
+          ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace)
+        end
+      end
+    rescue Exception => e
+      STDERR.puts "Invalid configuration 'config/skiplock.yml': #{e.message}"
+      exit
     end
 
     def self.standalone
-      title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
-      puts "-"*(title.length)
-      puts title
-      puts "-"*(title.length)
-      puts "Additional workers: #{Settings['workers']}"
-      puts "  Purge completion: #{Settings['purge_completion']}"
-      puts "       Max retries: #{Settings['max_retries']}"
-      puts "       Min threads: #{Settings['min_threads']}"
-      puts "       Max threads: #{Settings['max_threads']}"
-      puts "       Environment: #{Rails.env}"
-      puts "           Logging: #{Settings['logging']}"
-      puts "               PID: #{Process.pid}"
-      puts "-"*(title.length)
       if Settings['logging']
         log_timestamp = (Settings['logging'].to_s == 'timestamp')
         logfile = File.open('log/skiplock.log', 'a')
@@ -61,14 +92,29 @@ module Skiplock
         Rails.logger.reopen('/dev/null')
         Rails.logger.extend(ActiveSupport::Logger.broadcast(logger))
       end
+      title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
+      puts "-"*(title.length)
+      puts title
+      puts "-"*(title.length)
+      puts "Purge completion: #{Settings['purge_completion']}"
+      puts "    Notification: #{Settings['notification']}#{(' (' + @notification + ')') if Settings['notification'] == 'auto'}"
+      puts "     Max retries: #{Settings['max_retries']}"
+      puts "     Min threads: #{Settings['min_threads']}"
+      puts "     Max threads: #{Settings['max_threads']}"
+      puts "     Environment: #{Rails.env}"
+      puts "         Logging: #{Settings['logging']}"
+      puts "         Workers: #{Settings['workers']}"
+      puts "          Queues: #{Settings['queues'].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if Settings['queues'].is_a?(Hash)
+      puts "             PID: #{Process.pid}"
+      puts "-"*(title.length)
       parent_id = Process.pid
       shutdown = false
       Signal.trap("INT") { shutdown = true }
       Signal.trap("TERM") { shutdown = true }
       worker_pids = []
-      Settings['workers'].times do |n|
+      (Settings['workers']-1).times do |n|
         worker_pids << fork do
-          dispatcher = Dispatcher.new(master: false, worker_num: n)
+          dispatcher = Dispatcher.new(master: false, worker_num: n+1)
           thread = dispatcher.run
           loop do
             sleep 0.5

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.4'
+  VERSION = Version = '1.0.9'
 end
 

data/lib/skiplock/worker.rb

@@ -1,5 +1,4 @@
 module Skiplock
   class Worker < ActiveRecord::Base
-    self.table_name = 'skiplock.workers'
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.4
+  version: 1.0.9
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-03-22 00:00:00.000000000 Z
+date: 2021-05-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -82,11 +82,11 @@ files:
 - lib/generators/skiplock/install_generator.rb
 - lib/generators/skiplock/templates/migration.rb.erb
 - lib/skiplock.rb
+- lib/skiplock/counter.rb
 - lib/skiplock/cron.rb
 - lib/skiplock/dispatcher.rb
 - lib/skiplock/job.rb
 - lib/skiplock/manager.rb
-- lib/skiplock/notification.rb
 - lib/skiplock/version.rb
 - lib/skiplock/worker.rb
 homepage: https://github.com/vtt/skiplock
@@ -108,8 +108,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.6
+rubygems_version: 3.0.3
 signing_key: 
 specification_version: 4
 summary: ActiveJob Queue Adapter for PostgreSQL

data/lib/skiplock/notification.rb

@@ -1,5 +0,0 @@
-module Skiplock
-  class Notification
-  end
-end
-