skiplock 1.0.3 → 1.0.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9731224db727e3ebe4b32b94b4edff6bf7a53faca14be5a6f5d7cc20a02836e9
-  data.tar.gz: 9c3cb7ccf0d98d7e8ec4bcd21e33269504d166804f0679fe7d286290527a8457
+  metadata.gz: 2ee6a8ff68af1a1029a26db98867457a798c4fb6aacf0a2a2816efd3ef2b977b
+  data.tar.gz: 2c75fb21c79346f4525b8634b28fb326505acb7ab61795fe7d291c3faa75244f
 SHA512:
-  metadata.gz: 560b30ea96a8f255811b1c8a0beb7e161ba1edd2dde238e52cd31625e1a4e4d729ff24f3d4d37e4a29fbb2c3e2543e86b9cb1720553cacab346ac30db2970aea
-  data.tar.gz: '093b4ee02119973dbd1c8c905d2ccbcc1d6d9efad440ba23ba8fc05a8a2a3580ec7a7d3c7c47d5f6077b789a440e4c46e455f81a5c10fd032b01ffefae906a0e'
+  metadata.gz: c34fa2c27f9fc8bcfbe6c54a30de8645418dfe7c13e8db9ea8c30352f75abd9ab9b04d1a294aea7f2a9f33772d9cc5ed96839bd63d1eacbda17749758b4755b2
+  data.tar.gz: 03cdbef1a70a64fb188258dc44049f9323f53cb9c2e64f8a79a9c8a4a17ca8aea2e7ca718ceb3bc78cc08fd61d6dbd5c3f104f653a9645132ef31f73d22883da

data/README.md

@@ -1,6 +1,6 @@
 # Skiplock
 
-Skiplock is a background job queuing system that improves the performance and reliability of the job executions while providing the same ACID guarantees as the rest of your data.  It is designed for Active Jobs with Ruby on Rails using PostgreSQL database adapter, but it can be modified to work with other frameworks easily.
+`Skiplock` is a background job queuing system that improves the performance and reliability of the job executions while providing the same ACID guarantees as the rest of your data.  It is designed for Active Jobs with Ruby on Rails using PostgreSQL database adapter, but it can be modified to work with other frameworks easily.
 
 It only uses the `LISTEN/NOTIFY/SKIP LOCKED` features provided natively on PostgreSQL 9.5+ to efficiently and reliably dispatch jobs to worker processes and threads ensuring that each job can be completed successfully **only once**.  No other polling or timer is needed.
 
@@ -14,7 +14,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
 
 ## Installation
 
-1. Add `skiplock` to your application's Gemfile:
+1. Add `Skiplock` to your application's Gemfile:
 
     ```ruby
     gem 'skiplock'
@@ -26,7 +26,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     $ bundle install
     ```
 
-3. Run the Skiplock install generator. This will generate a configuration file and database migration to store the job records:
+3. Run the `Skiplock` install generator. This will generate a configuration file and database migration to store the job records:
 
     ```bash
     $ rails g skiplock:install
@@ -46,7 +46,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     # config/application.rb
     config.active_job.queue_adapter = :skiplock
     ```
-2. Skiplock configuration
+2. `Skiplock` configuration
     ```yaml
     # config/skiplock.yml
     ---
@@ -54,7 +54,11 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     min_threads: 1
     max_threads: 5
     max_retries: 20
+    notification: auto
     purge_completion: true
+    queues:
+      default: 200
+      mailers: 100
     workers: 0
     ```
     Available configuration options are:
@@ -62,11 +66,13 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     - **min_threads** (*integer*): sets minimum number of threads staying idle
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
     - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired (see Retry System for more details)
+    - **notification** (*enumeration*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, false`)
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
-    - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to 0 will enable **async mode**
+    - **queues** (*hash*): defines the set of queues with priorites; lower priority takes precedence
+    - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
     
     #### Async mode
-    When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads.  If using multi-worker cluster web server like Puma, then it should be configured as below:
+    When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads.  If using multi-worker cluster mode web server like Puma, then it should be configured as below:
     ```ruby
     # config/puma.rb
     before_fork do
@@ -74,14 +80,10 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
       Skiplock::Manager.shutdown
     end
 
-    on_worker_boot do
-      # ...
-      Skiplock::Manager.start
-    end
-
-    on_worker_shutdown do
-      # ...
-      Skiplock::Manager.shutdown
+    after_worker_fork do |worker_index|
+      # restarts skiplock after all Puma workers have been started
+      # Skiplock runs in Puma master's process only
+      Skiplock::Manager.start if (worker_index + 1) == @options[:workers]
     end
     ```
 
@@ -93,18 +95,24 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     ```
 - Skiplock supports all ActiveJob features:
     ```ruby
-    MyJob.set(wait: 5.minutes, priority: 10).perform_later(1,2,3)
+    MyJob.set(queue: 'my_queue', wait: 5.minutes, priority: 10).perform_later(1,2,3)
     ```
 - Outside of Rails application, queue the jobs by inserting the job records directly to the database table eg:
     ```sql
     INSERT INTO skiplock.jobs(job_class) VALUES ('MyJob');
     ```
-- Or with scheduling, priority and arguments:
+- Or with scheduling, priority, queue and arguments:
     ```sql
-    INSERT INTO skiplock.jobs(job_class,priority,scheduled_at,data) VALUES ('MyJob',10,NOW()+INTERVAL '5 min','{"arguments":[1,2,3]}');
+    INSERT INTO skiplock.jobs(job_class,queue_name,priority,scheduled_at,data) VALUES ('MyJob','my_queue',10,NOW()+INTERVAL '5 min','{"arguments":[1,2,3]}');
     ```
+## Queues priority vs Job priority
+*Why do queues have priorities when jobs already have priorities?*
+- Jobs are only prioritized with other jobs from the same queue
+- Queues, on the other hand, are prioritized with other queues
+- Rails has built-in queues that dispatches jobs without priorities (eg. Mail Delivery will queue as **mailers** with no priority)
+
 ## Cron system
-Skiplock provides the capability to setup cron jobs for running tasks periodically.  It fully supports the cron syntax to specify the frequency of the jobs.  To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
+`Skiplock` provides the capability to setup cron jobs for running tasks periodically.  It fully supports the cron syntax to specify the frequency of the jobs.  To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
 - setup `MyJob` to run as cron job every hour at 30 minutes past
 
     ```ruby
@@ -123,7 +131,7 @@ Skiplock provides the capability to setup cron jobs for running tasks periodical
 - to remove the cron schedule from the job, simply comment out the constant definition or delete the line then re-deploy the application. At startup, the cron jobs that were undefined will be removed automatically
 
 ## Retry system
-Skiplock fully supports ActiveJob built-in retry system.  It also has its own retry system for fallback.  To use ActiveJob retry system, define the rescue blocks per ActiveJob's documentation.
+`Skiplock` fully supports ActiveJob built-in retry system.  It also has its own retry system for fallback.  To use ActiveJob retry system, define the rescue blocks per ActiveJob's documentation.
 - configures `MyJob` to retry at maximum 20 attempts on StandardError with fixed delay of 5 seconds
     ```ruby
     class MyJob < ActiveJob::Base
@@ -144,7 +152,16 @@ Once the retry attempt limit configured in ActiveJob has been reached, the contr
 If the rescue blocks are not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
 
 ## Notification system
-...
+`Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  A customized function can also be called whenever an exception occurs; this can be configured in an initializer as below:
+```ruby
+  # config/initializers/skiplock.rb
+  Skiplock.on_error = -> (ex, previous = nil) do
+    if ex.backtrace != previous.try(:backtrace)
+      # sends custom email on new exceptions only
+      # the same repeated exceptions will only be sent once to avoid SPAM
+    end
+  end
+```
 
 ## Contributing
 

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -2,33 +2,64 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
   def up
     execute 'CREATE SCHEMA skiplock'
     create_table 'skiplock.jobs', id: :uuid do |t|
+      t.uuid      :worker_id, index: true
       t.string    :job_class, null: false
-      t.string    :cron
-      t.string    :queue_name
+      t.string    :queue_name, index: true
       t.string    :locale
       t.string    :timezone
+      t.string    :cron
       t.integer   :priority
       t.integer   :executions
       t.jsonb     :exception_executions
       t.jsonb     :data
-      t.boolean   :running, null: false, default: false
+      t.boolean   :running, null: false, default: false, index: true
       t.timestamp :expired_at
       t.timestamp :finished_at
       t.timestamp :scheduled_at
       t.timestamps null: false, default: -> { 'now()' }
     end
-    execute %(CREATE OR REPLACE FUNCTION skiplock.notify() RETURNS TRIGGER AS $$
-      BEGIN
-        IF (NEW.finished_at IS NULL AND NEW.expired_at IS NULL) THEN
-          PERFORM pg_notify('skiplock', CONCAT(TG_OP,',',NEW.id::TEXT,',',NEW.queue_name,',',NEW.priority,',',CAST(EXTRACT(EPOCH FROM NEW.scheduled_at) AS FLOAT)::text));
-        END IF;
-        RETURN NULL;
-      END;
+    create_table 'skiplock.workers', id: :uuid do |t|
+      t.integer   :pid, null: false, index: true
+      t.integer   :ppid, index: true
+      t.integer   :capacity, null: false
+      t.string    :hostname, null: false, index: true
+      t.jsonb     :data
+      t.timestamps null: false, index: true, default: -> { 'now()' }
+    end    
+    execute <<~ENDFUNC
+      CREATE OR REPLACE FUNCTION skiplock.notify_jobs() RETURNS TRIGGER AS $$
+        DECLARE
+          record RECORD;
+        BEGIN
+          IF (TG_OP = 'DELETE') THEN
+            record = OLD;
+          ELSE
+            record = NEW;
+          END IF;
+          PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));
+          RETURN NULL;
+        END;
       $$ LANGUAGE plpgsql
-    )
-    execute "CREATE TRIGGER notify_job AFTER INSERT OR UPDATE ON skiplock.jobs FOR EACH ROW EXECUTE PROCEDURE skiplock.notify()"
-    execute "CREATE INDEX jobs_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE running = 'f' AND expired_at IS NULL AND finished_at IS NULL"
-    execute "CREATE INDEX jobs_retry_index ON skiplock.jobs(scheduled_at) WHERE running = 'f' AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL"
+    ENDFUNC
+    execute <<~ENDFUNC
+      CREATE OR REPLACE FUNCTION skiplock.notify_workers() RETURNS TRIGGER AS $$
+        DECLARE
+          record RECORD;
+        BEGIN
+          IF (TG_OP = 'DELETE') THEN
+            record = OLD;
+          ELSE
+            record = NEW;
+          END IF;
+          PERFORM pg_notify('skiplock::workers', CONCAT(TG_OP,',',record.id::TEXT,',',record.hostname,',',record.capacity,',',record.pid,',',record.ppid));
+          RETURN NULL;
+        END;
+      $$ LANGUAGE plpgsql;
+    ENDFUNC
+    execute "CREATE TRIGGER notify_job AFTER INSERT OR UPDATE OR DELETE ON skiplock.jobs FOR EACH ROW EXECUTE PROCEDURE skiplock.notify_jobs()"
+    execute "CREATE TRIGGER notify_worker AFTER INSERT OR UPDATE OR DELETE ON skiplock.workers FOR EACH ROW EXECUTE PROCEDURE skiplock.notify_workers()"
+    execute "CREATE INDEX jobs_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL"
+    execute "CREATE INDEX jobs_retry_index ON skiplock.jobs(scheduled_at) WHERE running = FALSE AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL"
     execute "CREATE INDEX jobs_cron_index ON skiplock.jobs(scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC) WHERE cron IS NOT NULL AND finished_at IS NULL"
     execute "CREATE UNIQUE INDEX jobs_unique_cron_index ON skiplock.jobs (job_class) WHERE cron IS NOT NULL"
   end

data/lib/skiplock.rb

@@ -4,17 +4,23 @@ require 'active_record'
 require 'skiplock/cron'
 require 'skiplock/dispatcher'
 require 'skiplock/manager'
-require 'skiplock/notification'
 require 'skiplock/job'
+require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
   Settings = { 
-    'logging' => :timestamp,
+    'logging' => 'timestamp',
     'min_threads' => 1,
     'max_threads' => 5,
     'max_retries' => 20,
+    'notification' => 'auto',
     'purge_completion' => true,
+    'queues' => {
+      'default' => 200,
+      'mailers' => 100
+    },
     'workers' => 0
   }
+  mattr_accessor :on_error
 end

data/lib/skiplock/dispatcher.rb

@@ -1,68 +1,120 @@
 module Skiplock
   class Dispatcher
-    def initialize(master: true)
+    def initialize(master: true, worker_num: nil, worker_pids: [])
+      @queues_order_query = Settings['queues'].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if Settings['queues'].is_a?(Hash) && Settings['queues'].count > 0
       @executor = Concurrent::ThreadPoolExecutor.new(min_threads: Settings['min_threads'], max_threads: Settings['max_threads'], max_queue: Settings['max_threads'], idletime: 60, auto_terminate: true, fallback_policy: :discard)
       @master = master
+      if @master
+        @worker_pids = worker_pids + [ Process.pid ]
+      else
+        @worker_num = worker_num
+      end
+      @last_dispatch_at = 0
       @next_schedule_at = Time.now.to_f
       @running = true
     end
     
     def run
       Thread.new do
-        Rails.application.reloader.wrap do
-          sleep(0.1) while @running && !Rails.application.initialized?
-          ActiveRecord::Base.connection_pool.with_connection do |connection|
-            connection.exec_query('LISTEN skiplock')
-            if @master
-              # reset retries schedules on startup
-              Job.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
-              Cron.setup
+        sleep(1) while @running && !Rails.application.initialized?
+        Process.setproctitle("skiplock-#{@master ? 'master[0]' : 'worker[' + @worker_num.to_s + ']'}") if Settings['workers'] > 0
+        ActiveRecord::Base.connection_pool.with_connection do |connection|
+          connection.exec_query('LISTEN "skiplock::jobs"')
+          hostname = `hostname -f`.strip
+          @worker = Worker.create!(pid: Process.pid, ppid: (@master ? nil : Process.ppid), capacity: Settings['max_threads'], hostname: hostname)
+          if @master
+            connection.exec_query('LISTEN "skiplock::workers"')
+            if File.exists?('tmp/cache/skiplock')
+              # get performed jobs that could not sync with database
+              job_ids = File.read('tmp/cache/skiplock').split("\n")
+              if Settings['purge_completion']
+                Job.where(id: job_ids).delete_all
+              else
+                Job.where(id: job_ids).update_all(running: false, finished_at: File.mtime('tmp/cache/skiplock'), updated_at: Time.now)
+              end
+              File.delete('tmp/cache/skiplock')
             end
-            error = false
-            while @running
-              begin
-                if error
-                  unless connection.active?
-                    connection.reconnect!
-                    sleep(0.5)
-                    connection.exec_query('LISTEN skiplock')
-                    @next_schedule_at = Time.now
-                  end
-                  error = false
+            # get dead worker ids
+            dead_worker_ids = Worker.where(hostname: hostname).where.not(pid: @worker_pids).ids
+            if dead_worker_ids.count > 0
+              # reset orphaned jobs of the dead worker ids for retry
+              Job.where(running: true).where(worker_id: dead_worker_ids).update_all(running: false, worker_id: nil)
+              # remove dead workers
+              Worker.where(id: dead_worker_ids).delete_all
+            end
+            # reset retries schedules on startup
+            Job.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
+            Cron.setup
+          end
+          error = false
+          while @running
+            begin
+              if error
+                unless connection.active?
+                  connection.reconnect!
+                  sleep(0.5)
+                  connection.exec_query('LISTEN "skiplock::jobs"')
+                  connection.exec_query('LISTEN "skiplock::workers"') if @master
+                  @next_schedule_at = Time.now.to_f
+                end
+                error = false
+              end
+              if Job::Errors.keys.count > 0
+                completed_ids = Job::Errors.keys.map { |k| k if Job::Errors[k] }.compact
+                if Settings['purge_completion'] && completed_ids.count > 0
+                  Job.where(id: completed_ids, running: true).delete_all
+                elsif completed_ids.count > 0
+                  Job.where(id: completed_ids, running: true).update_all(running: false, finished_at: Time.now, updated_at: Time.now)
                 end
-                if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
-                  @executor.post { do_work }
+                orphaned_ids = Job::Errors.keys.map { |k| k unless Job::Errors[k] }.compact
+                Job.where(id: orphaned_ids, running: true).update_all(running: false, worker_id: nil, scheduled_at: (Time.now + 10), updated_at: Time.now) if orphaned_ids.count > 0
+                Job::Errors.clear
+              end
+              notifications = { 'skiplock::jobs' => [], 'skiplock::workers' => [] }
+              connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
+                notifications[channel] << payload if payload
+                loop do
+                  payload = connection.raw_connection.notifies
+                  break unless @running && payload
+                  notifications[payload[:relname]] << payload[:extra]
                 end
-                notifications = []
-                connection.raw_connection.wait_for_notify(0.1) do |channel, pid, payload|
-                  notifications << payload if payload
-                  loop do
-                    payload = connection.raw_connection.notifies
-                    break unless @running && payload
-                    notifications << payload[:extra]
+                notifications['skiplock::jobs'].each do |n|
+                  op, id, worker_id, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
+                  if @master
+                    # TODO: report job status to action cable
                   end
-                  notifications.each do |n|
-                    op, id, queue, priority, time = n.split(',')
-                    if time.to_f <= Time.now.to_f
-                      @next_schedule_at = Time.now.to_f
-                    elsif time.to_f < @next_schedule_at
-                      @next_schedule_at = time.to_f
-                    end
+                  next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0  || scheduled_at.to_f < @last_dispatch_at
+                  if scheduled_at.to_f < Time.now.to_f
+                    @next_schedule_at = Time.now.to_f
+                  elsif scheduled_at.to_f < @next_schedule_at
+                    @next_schedule_at = scheduled_at.to_f
                   end
                 end
-              rescue Exception => ex
-                # TODO: Report exception
-                error = true
-                timestamp = Time.now
-                while @running
-                  sleep(0.5)
-                  break if Time.now - timestamp > 10
-                end              
+                if @master
+                  # TODO: report worker status to action cable
+                  notifications['skiplock::workers'].each do |n|
+                  end
+                end
+              end
+              if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
+                @executor.post { do_work }
+              end
+            rescue Exception => ex
+              # most likely error with database connection
+              STDERR.puts ex.message
+              STDERR.puts ex.backtrace
+              Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
+              error = true
+              t = Time.now
+              while @running
+                sleep(0.5)
+                break if Time.now - t > 5
               end
-              sleep(0.1)
+              @last_exception = ex
             end
-            connection.exec_query('UNLISTEN *')
+            sleep(0.2)
           end
+          connection.exec_query('UNLISTEN *')
         end
       end
     end
@@ -71,22 +123,24 @@ module Skiplock
       @running = false
       @executor.shutdown
       @executor.wait_for_termination if wait
+      @worker.delete if @worker
     end
 
     private
 
     def do_work
-      connection = ActiveRecord::Base.connection_pool.checkout
       while @running
-        result = Job.dispatch(connection: connection)
-        next if result.is_a?(Hash)
+        @last_dispatch_at = Time.now.to_f
+        result = Job.dispatch(queues_order_query: @queues_order_query, worker_id: @worker.id)
+        next if result.is_a?(Job)
         @next_schedule_at = result if result.is_a?(Float)
         break
       end
-    rescue Exception => e
-      # TODO: Report exception
-    ensure
-      ActiveRecord::Base.connection_pool.checkin(connection)
+    rescue Exception => ex
+      STDERR.puts ex.message
+      STDERR.puts ex.backtrace
+      Skiplock.on_error.call(ex, @last_exception) if Skiplock.on_error.is_a?(Proc)
+      @last_exception = ex
     end
   end
 end

data/lib/skiplock/job.rb

@@ -1,71 +1,82 @@
 module Skiplock
   class Job < ActiveRecord::Base
     self.table_name = 'skiplock.jobs'
+    Errors = Concurrent::Map.new
 
-    # Accept: An active ActiveRecord database connection (eg. ActiveRecord::Base.connection)
-    #         The connection should be checked out using ActiveRecord::Base.connection_pool.checkout, and be checked
-    #         in using ActiveRecord::Base.conection_pool.checkin once all of the job dispatches have been completed.
-    #         *** IMPORTANT: This connection cannot be shared with the job's execution
-    #
-    # Return: Attributes hash of the Job if it was executed; otherwise returns the next Job's schedule time in FLOAT
-    def self.dispatch(connection: ActiveRecord::Base.connection)
-      connection.exec_query('BEGIN')
-      job = connection.exec_query("SELECT * FROM #{self.table_name} WHERE running = 'f' AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
-      if job && job['scheduled_at'].to_f <= Time.now.to_f  # found job ready to perform
-        # update the job to mark it in progress in case database server goes down during job execution
-        connection.exec_query("UPDATE #{self.table_name} SET running = 't' WHERE id = '#{job['id']}'")
-        connection.exec_query('END') # close the transaction commit the state of job in progress
-        executions = (job['executions'] || 0) + 1
-        exceptions = job['exception_executions'] ? JSON.parse(job['exception_executions']) : {}
-        data = job['data'] ? JSON.parse(job['data']) : {}
-        job_data = job.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions').merge('job_id' => job['id'], 'exception_executions' => exceptions, 'enqueued_at' => job['updated_at']).merge(data)
-        Thread.current[:skiplock_dispatch_data] = job_data
-        begin
-          ActiveJob::Base.execute(job_data)
-        rescue Exception => ex
+    # Return: Skiplock::Job if it was executed; otherwise returns the next Job's schedule time in FLOAT
+    def self.dispatch(queues_order_query: nil, worker_id: nil)
+      performed = false
+      self.connection.exec_query('BEGIN')
+      job = self.find_by_sql("SELECT id, scheduled_at FROM #{self.table_name} WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{queues_order_query ? ' CASE ' + queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+      if job.nil? || job.scheduled_at.to_f > Time.now.to_f
+        self.connection.exec_query('END')
+        return (job ? job.scheduled_at.to_f : Float::INFINITY)
+      end
+      job = Skiplock::Job.find_by_sql("UPDATE #{self.table_name} SET running = TRUE, worker_id = #{self.connection.quote(worker_id)} WHERE id = '#{job.id}' RETURNING *").first
+      self.connection.exec_query('END')
+      job.data ||= {}
+      job.exception_executions ||= {}
+      job_data = job.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => job.id, 'enqueued_at' => job.updated_at, 'arguments' => (job.data['arguments'] || []))
+      job.executions = (job.executions || 0) + 1
+      Thread.current[:skiplock_dispatch_job] = job
+      begin
+        ActiveJob::Base.execute(job_data)
+      rescue Exception => ex
+      end
+      performed = true
+      job.running = false
+      if ex
+        job.exception_executions["[#{ex.class.name}]"] = (job.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless job.exception_executions.key?('activejob_retry')
+        if job.executions >= Settings['max_retries'] || job.exception_executions.key?('activejob_retry')
+          job.expired_at = Time.now
+          job.save!
+        else
+          job.scheduled_at = Time.now + (5 * 2**job.executions)
+          job.save!
         end
-        if ex
-          # TODO: report exception
-          exceptions["[#{ex.class.name}]"] = (exceptions["[#{ex.class.name}]"] || 0) + 1 unless exceptions.key?('activejob_retry')
-          if executions >= Settings['max_retries'] || exceptions.key?('activejob_retry')
-            connection.exec_query("UPDATE #{self.table_name} SET running = 'f', executions = #{executions}, exception_executions = '#{connection.quote_string(exceptions.to_json.to_s)}', expired_at = NOW(), updated_at = NOW() WHERE id = '#{job['id']}' RETURNING *").first
-          else
-            timestamp = Time.now + (5 * 2**executions)
-            connection.exec_query("UPDATE #{self.table_name} SET running = 'f', executions = #{executions}, exception_executions = '#{connection.quote_string(exceptions.to_json.to_s)}', scheduled_at = TO_TIMESTAMP(#{timestamp.to_f}), updated_at = NOW() WHERE id = '#{job['id']}' RETURNING *").first
-          end
-        elsif exceptions.key?('activejob_retry')
-          connection.exec_query("UPDATE #{self.table_name} SET running = 'f', executions = #{job_data['executions']}, exception_executions = '#{connection.quote_string(job_data['exception_executions'].to_json.to_s)}', scheduled_at = TO_TIMESTAMP(#{job_data['scheduled_at'].to_f}), updated_at = NOW() WHERE id = '#{job['id']}' RETURNING *").first
-        elsif job['cron']
-          data['last_cron_run'] = Time.now.utc.to_s
-          next_cron_at = Cron.next_schedule_at(job['cron'])
-          if next_cron_at
-            connection.exec_query("UPDATE #{self.table_name} SET running = 'f', scheduled_at = TO_TIMESTAMP(#{next_cron_at}), executions = 1, exception_executions = NULL, data = '#{connection.quote_string(data.to_json.to_s)}', updated_at = NOW() WHERE id = '#{job['id']}' RETURNING *").first
-          else
-            connection.exec_query("DELETE FROM #{self.table_name} WHERE id = '#{job['id']}' RETURNING *").first
-          end
-        elsif Settings['purge_completion']
-          connection.exec_query("DELETE FROM #{self.table_name} WHERE id = '#{job['id']}' RETURNING *").first
+        Skiplock.on_error.call(ex) if Skiplock.on_error.is_a?(Proc) && (job.executions % 3 == 1)
+      elsif job.exception_executions.key?('activejob_retry')
+        job.save!
+      elsif job.cron
+        job.data['last_cron_run'] = Time.now.utc.to_s
+        next_cron_at = Cron.next_schedule_at(job.cron)
+        if next_cron_at
+          job.executions = 1
+          job.exception_executions = nil
+          job.scheduled_at = Time.at(next_cron_at)
+          job.save!
         else
-          connection.exec_query("UPDATE #{self.table_name} SET running = 'f', executions = #{executions}, exception_executions = NULL, finished_at = NOW(), updated_at = NOW() WHERE id = '#{job['id']}' RETURNING *").first
+          job.delete
         end
+      elsif Settings['purge_completion']
+        job.delete
+      else
+        job.finished_at = Time.now
+        job.exception_executions = nil
+        job.save!
+      end
+      job
+    rescue Exception => ex
+      if performed
+        Errors[job.id] = true
+        File.write('tmp/cache/skiplock', job.id + "\n", mode: 'a')
       else
-        connection.exec_query('END')
-        job ? job['scheduled_at'].to_f : Float::INFINITY
+        Errors[job.id] = false
       end
+      raise ex
     ensure
-      Thread.current[:skiplock_dispatch_data] = nil
+      Thread.current[:skiplock_dispatch_job] = nil
     end
 
-    def self.enqueue_at(job, timestamp)
-      if Thread.current[:skiplock_dispatch_data]
-        job.exception_executions['activejob_retry'] = true
-        Thread.current[:skiplock_dispatch_data]['executions'] = job.executions
-        Thread.current[:skiplock_dispatch_data]['exception_executions'] = job.exception_executions
-        Thread.current[:skiplock_dispatch_data]['scheduled_at'] = Time.at(timestamp)
-        self.new(Thread.current[:skiplock_dispatch_data].slice(*self.column_names).merge(id: job.job_id))
+    def self.enqueue_at(activejob, timestamp)
+      timestamp = Time.at(timestamp) if timestamp
+      if Thread.current[:skiplock_dispatch_job].try(:id) == activejob.job_id
+        Thread.current[:skiplock_dispatch_job].exception_executions = activejob.exception_executions.merge('activejob_retry' => true)
+        Thread.current[:skiplock_dispatch_job].executions = activejob.executions
+        Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
+        Thread.current[:skiplock_dispatch_job]
       else
-        timestamp = Time.at(timestamp) if timestamp
-        Job.create!(id: job.job_id, job_class: job.class.name, queue_name: job.queue_name, locale: job.locale, timezone: job.timezone, priority: job.priority, executions: job.executions, data: { 'arguments' => job.arguments }, scheduled_at: timestamp)
+        Job.create!(id: activejob.job_id, job_class: activejob.class.name, queue_name: activejob.queue_name, locale: activejob.locale, timezone: activejob.timezone, priority: activejob.priority, executions: activejob.executions, data: { 'arguments' => activejob.serialize['arguments'] }, scheduled_at: timestamp)
       end
     end
   end

data/lib/skiplock/manager.rb

@@ -1,8 +1,22 @@
 module Skiplock
   class Manager
-    def self.start(standalone: false)
-      load_settings
-      return unless standalone || (caller.any?{|l| l =~ %r{/rack/}} && (Settings['workers'] == 0 || Rails.env.development?))
+    def self.start(standalone: false, restart: false, workers: nil, max_retries: nil, max_threads: nil, min_threads: nil, logging: nil)
+      unless Settings.frozen?
+        load_settings
+        Settings['logging'] = logging if logging
+        Settings['max_retries'] = max_retries if max_retries
+        Settings['max_threads'] = max_threads if max_threads
+        Settings['min_threads'] = min_threads if min_threads
+        Settings['workers'] = workers if workers
+        Settings['max_retries'] = 20 if Settings['max_retries'] > 20
+        Settings['max_retries'] = 0 if Settings['max_retries'] < 0
+        Settings['max_threads'] = 1 if Settings['max_threads'] < 1
+        Settings['max_threads'] = 20 if Settings['max_threads'] > 20
+        Settings['min_threads'] = 0 if Settings['min_threads'] < 0
+        Settings['workers'] = 0 if Settings['workers'] < 0
+        Settings.freeze
+      end
+      return unless standalone || restart || (caller.any?{|l| l =~ %r{/rack/}} && (Settings['workers'] == 0 || Rails.env.development?))
       if standalone
         self.standalone
       else
@@ -10,44 +24,52 @@ module Skiplock
         @thread = @dispatcher.run
         at_exit { self.shutdown }
       end
+      ActiveJob::Base.logger = nil
     end
     
     def self.shutdown(wait: true)
       if @dispatcher && @thread
         @dispatcher.shutdown(wait: wait)
         @thread.join
+        @dispatcher = nil
+        @thread = nil
       end
     end
 
   private
 
     def self.load_settings
-      return if Settings.frozen?
       config = YAML.load_file('config/skiplock.yml') rescue {}
       Settings.merge!(config)
-      Settings['max_retries'] = 20 if Settings['max_retries'] > 20
-      Settings['max_retries'] = 0 if Settings['max_retries'] < 0
-      Settings['max_threads'] = 1 if Settings['max_threads'] < 1
-      Settings['max_threads'] = 20 if Settings['max_threads'] > 20
-      Settings['min_threads'] = 0 if Settings['min_threads'] < 0
-      Settings['workers'] = 0 if Settings['workers'] < 0
-      Settings.freeze
+      Settings['queues'].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if Settings['queues'].is_a?(Hash)
+      case Settings['notification'].to_s.downcase
+      when 'auto'
+        if defined?(Airbrake)
+          Skiplock.on_error = -> (ex, previous = nil) { Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace) }
+        elsif defined?(Bugsnag)
+          Skiplock.on_error = -> (ex, previous = nil) { Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace) }
+        elsif defined?(ExceptionNotifier)
+          Skiplock.on_error = -> (ex, previous = nil) { ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace) }
+        else
+          puts "Unable to detect any known exception notification gem. Please define custom 'on_error' function and disable notification in 'config/skiplock.yml'"
+          exit
+        end
+      when 'airbrake'
+        raise 'airbrake gem not found' unless defined?(Airbrake)
+        Skiplock.on_error = -> (ex, previous = nil) { Airbrake.notify_sync(ex) unless ex.backtrace == previous.try(:backtrace) }
+      when 'bugsnag'
+        raise 'bugsnag gem not found' unless defined?(Bugsnag)
+        Skiplock.on_error = -> (ex, previous = nil) { Bugsnag.notify(ex) unless ex.backtrace == previous.try(:backtrace) }
+      when 'exception_notification'
+        raise 'exception_notification gem not found' unless defined?(ExceptionNotifier)
+        Skiplock.on_error = -> (ex, previous = nil) { ExceptionNotifier.notify_exception(ex) unless ex.backtrace == previous.try(:backtrace) }
+      end
+    rescue Exception => e
+      STDERR.puts "Invalid configuration 'config/skiplock.yml': #{e.message}"
+      exit
     end
 
     def self.standalone
-      title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
-      puts "-"*(title.length)
-      puts title
-      puts "-"*(title.length)
-      puts "Additional workers: #{Settings['workers']}"
-      puts "  Purge completion: #{Settings['purge_completion']}"
-      puts "       Max retries: #{Settings['max_retries']}"
-      puts "       Min threads: #{Settings['min_threads']}"
-      puts "       Max threads: #{Settings['max_threads']}"
-      puts "       Environment: #{Rails.env}"
-      puts "           Logging: #{Settings['logging']}"
-      puts "               PID: #{Process.pid}"
-      puts "-"*(title.length)
       if Settings['logging']
         log_timestamp = (Settings['logging'].to_s == 'timestamp')
         logfile = File.open('log/skiplock.log', 'a')
@@ -61,14 +83,29 @@ module Skiplock
         Rails.logger.reopen('/dev/null')
         Rails.logger.extend(ActiveSupport::Logger.broadcast(logger))
       end
+      title = "Skiplock version: #{Skiplock::VERSION} (Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
+      puts "-"*(title.length)
+      puts title
+      puts "-"*(title.length)
+      puts "Purge completion: #{Settings['purge_completion']}"
+      puts "    Notification: #{Settings['notification']}"
+      puts "     Max retries: #{Settings['max_retries']}"
+      puts "     Min threads: #{Settings['min_threads']}"
+      puts "     Max threads: #{Settings['max_threads']}"
+      puts "     Environment: #{Rails.env}"
+      puts "         Logging: #{Settings['logging']}"
+      puts "         Workers: #{Settings['workers']}"
+      puts "          Queues: #{Settings['queues'].map {|k,v| k + '(' + v.to_s + ')'}.join(', ')}" if Settings['queues'].is_a?(Hash)
+      puts "             PID: #{Process.pid}"
+      puts "-"*(title.length)
       parent_id = Process.pid
       shutdown = false
       Signal.trap("INT") { shutdown = true }
       Signal.trap("TERM") { shutdown = true }
-      Settings['workers'].times do |w|
-        fork do
-          Process.setproctitle("skiplock-worker[#{w+1}]")
-          dispatcher = Dispatcher.new(master: false)
+      worker_pids = []
+      (Settings['workers']-1).times do |n|
+        worker_pids << fork do
+          dispatcher = Dispatcher.new(master: false, worker_num: n+1)
           thread = dispatcher.run
           loop do
             sleep 0.5
@@ -80,8 +117,7 @@ module Skiplock
         end
       end
       sleep 0.1
-      Process.setproctitle("skiplock-master")
-      dispatcher = Dispatcher.new
+      dispatcher = Dispatcher.new(worker_pids: worker_pids)
       thread = dispatcher.run
       loop do
         sleep 0.5

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.3'
+  VERSION = Version = '1.0.8'
 end
 

data/lib/skiplock/worker.rb

@@ -0,0 +1,5 @@
+module Skiplock
+  class Worker < ActiveRecord::Base
+    self.table_name = 'skiplock.workers'
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.3
+  version: 1.0.8
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-03-17 00:00:00.000000000 Z
+date: 2021-04-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -86,8 +86,8 @@ files:
 - lib/skiplock/dispatcher.rb
 - lib/skiplock/job.rb
 - lib/skiplock/manager.rb
-- lib/skiplock/notification.rb
 - lib/skiplock/version.rb
+- lib/skiplock/worker.rb
 homepage: https://github.com/vtt/skiplock
 licenses:
 - MIT
@@ -107,7 +107,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.1.4
 signing_key: 
 specification_version: 4
 summary: ActiveJob Queue Adapter for PostgreSQL

data/lib/skiplock/notification.rb

@@ -1,5 +0,0 @@
-module Skiplock
-  class Notification
-  end
-end
-