skiplock 1.0.14 → 1.0.18

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 55c98833e2792a216b6389d399e992aae9660f23ea076fcf852fb164cbc32d55
-  data.tar.gz: 43c8c2f90ee55ec423242042d1ab2986c0cb9f70fcc04db56aaf2c566bbb0382
+  metadata.gz: 8112589dcbdf58a743529ec9bf5ddd8371791e83e79c0d455ed42cbca05ee5eb
+  data.tar.gz: 149f826e93ecd8ac6165a8db46a8e7e234636a4481e68bf09d208b04a1dea1ce
 SHA512:
-  metadata.gz: 873492b815e6b5fc653630fed3bc64c5a363339551bbad304ecbad9453ea3738c24f3f0b874f4d6fe47cd4cc0ea5d46fb61b4c0dc322814d08dbc11002c24443
-  data.tar.gz: e84f7081e07ba0aff19834251de09e08df573d45ff159299fe08baf621b76845cd2fdf0461229da77b07cf707f151e7d8fc00d97b19a9bf30f1129376c2dd053
+  metadata.gz: 437f622f2398a0a6c965ea1720f77f11b9241373f277e249e4f3d39afbd3bccc7d8ad2b919abd141ebc5bc6a2543408a7940e53f2968c3cda9fc0519ac387910
+  data.tar.gz: '0473939dc2497842ff66fb02e2f74371f7281d58cc5ac82f1999b669f5b45e526d5d75054a3cc95594e26fc5f4e4aafc42e15d2f7336b88223256d6f7ac46d4e'

data/README.md

@@ -4,7 +4,7 @@
 
 It only uses the `LISTEN/NOTIFY/SKIP LOCKED` features provided natively on PostgreSQL 9.5+ to efficiently and reliably dispatch jobs to worker processes and threads ensuring that each job can be completed successfully **only once**.  No other polling or timer is needed.
 
-The library is quite small compared to other PostgreSQL job queues (eg. *delay_job*, *queue_classic*, *que*, *good_job*) with less than 400 lines of codes; and it still provides similar set of features and more...
+The library is quite small compared to other PostgreSQL job queues (eg. *delay_job*, *queue_classic*, *que*, *good_job*) with less than 500 lines of codes; and it still provides similar set of features and more...
 
 #### Compatibility:
 
@@ -51,9 +51,9 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     # config/skiplock.yml (default settings)
     ---
     min_threads: 1
-    max_threads: 5
+    max_threads: 10
     max_retries: 20
-    logfile: log/skiplock.log
+    logfile: skiplock.log
     loglevel: info
     notification: custom
     extensions: false
@@ -67,11 +67,11 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     - **min_threads** (*integer*): sets minimum number of threads staying idle
     - **max_threads** (*integer*): sets the maximum number of threads allowed to run jobs
     - **max_retries** (*integer*): sets the maximum attempt a job will be retrying before it is marked expired.  See `Retry system` for more details
-    - **logfile** (*string*): path filename for skiplock logs; empty logfile will disable logging
+    - **logfile** (*string*): filename for skiplock logs; empty logfile will disable logging
     - **loglevel** (*string*): sets logging level (`debug, info, warn, error, fatal, unknown`)
     - **notification** (*string*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, custom`); using `auto` will detect library if available.  See `Notification system` for more details
-    - **extensions** (*boolean*): enable or disable the class method extension.  See `ClassMethod extension` for more details
-    - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months)
+    - **extensions** (*multi*): enable or disable the class method extension.  See `ClassMethod extension` for more details
+    - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months); queued jobs can manually override using `purge` option
     - **queues** (*hash*): defines the set of queues with priorities; lower priority takes precedence
     - **workers** (*integer*) sets the maximum number of processes when running in standalone mode using the `skiplock` executable; setting this to **0** will enable **async mode**
 
@@ -84,11 +84,12 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     $ bundle exec skiplock -h
     Usage: skiplock [options]
       -e, --environment STRING         Rails environment
-      -l, --logfile STRING             Full path to logfile
+      -l, --logfile STRING             Log filename
       -s, --graceful-shutdown NUM      Number of seconds to wait for graceful shutdown
       -r, --max-retries NUM            Number of maxixum retries
       -t, --max-threads NUM            Number of maximum threads
       -T, --min-threads NUM            Number of minimum threads
+      -v, --version                    Show version information
       -w, --workers NUM                Number of workers
       -h, --help                       Show this message
     ```
@@ -99,19 +100,27 @@ Inside the Rails application:
     ```ruby
     MyJob.perform_later
     ```
-- Skiplock supports all ActiveJob features
+- Skiplock supports all ActiveJob features and options
     ```ruby
     MyJob.set(queue: 'my_queue', wait: 5.minutes, priority: 10).perform_later(1,2,3)
+    MyJob.set(wait_until: Day.tomorrow.noon).perform_later(1,2,3)
+    ```
+- Skiplock supports custom options which override the global `Skiplock` configuration options for specified jobs
+    - **purge** (*boolean*): whether to remove this job after it has completed successfully
+    - **max_retries** (*integer*): set maximum retry attempt for this job
+    ```ruby
+    MyJob.set(purge: false, max_retries: 5).perform_later(1,2,3)
     ```
 Outside the Rails application:
 - queue the jobs by inserting the job records directly to the database table
     ```sql
     INSERT INTO skiplock.jobs(job_class) VALUES ('MyJob');
     ```
-- with scheduling, priority, queue and arguments
+- with scheduling, priority, queue, arguments and custom options
     ```sql
     INSERT INTO skiplock.jobs(job_class, queue_name, priority, scheduled_at, data)
-      VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min', '{"arguments":[1,2,3]}');
+      VALUES ('MyJob', 'my_queue', 10, NOW() + INTERVAL '5 min',
+      '{"arguments":[1,2,3],"options":{"purge":false,"max_retries":5}}');
     ```
 ## Queue priority vs Job priority
 *Why do queues use priorities when jobs already have priorities?*
@@ -155,9 +164,9 @@ Outside the Rails application:
     # ...
     end
     ```
-If the retry attempt limit configured in ActiveJob has been reached, then the control will be passed back to `skiplock` to be marked as an expired job.
+If the retry attempt limit configured in ActiveJob has been reached, then the control will be passed back to `Skiplock` to be marked as an expired job.
 
-If the `retry_on` block is not defined, then the built-in retry system of `skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
+If the `retry_on` block is not defined, then the built-in retry system of `Skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `Skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
 
 ## Notification system
 `Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  Custom notification can also be called whenever an exception occurs; it can be configured in an initializer like below:
@@ -165,15 +174,25 @@ If the `retry_on` block is not defined, then the built-in retry system of `skipl
   # config/initializers/skiplock.rb
   Skiplock.on_error do |ex, previous|
     if ex.backtrace != previous.try(:backtrace)
-      # sends custom email on new exceptions only
+      # sends text message using Amazon SNS on new exceptions only
       # the same repeated exceptions will only be sent once to avoid SPAM
       # NOTE: exceptions generated from Job executions will not provide 'previous' exceptions
+      sms = Aws::SNS::Client.new(region: 'us-west-2', access_key_id: Rails.application.credentials[:aws][:access_key_id], secret_access_key: Rails.application.credentials[:aws][:secret_access_key])
+      sms.publish({ phone_number: '+122233334444', message: "Exception: #{ex.message}"[0..130] })
     end
   end
   # supports multiple 'on_error' event callbacks
 ```
 ## ClassMethod extension
-`Skiplock` can add extension to allow all class methods to be performed as a background job; it is disabled in the default configuration.  To enable, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`.
+`Skiplock` can add extension to allow class methods to be performed as a background job; it is disabled in the default configuration.  To enable globally for all classes and modules, edit the `config/skiplock.yml` configuration file and change `extensions` to `true`; this can expose remote code execution if the `skiplock.jobs` database table is not secured properly.
+
+To enable extension for specific classes and modules only then set the configuration to an array of names of the classes and modules eg. `['MyClass', 'MyModule']`
+- An example of remote code execution if the extension is enabled globally (ie: configuration is set to `true`) and attacker can insert `skiplock.jobs`
+  ```sql
+  INSERT INTO skiplock.jobs(job_class, data)
+    VALUES ('Skiplock::Extension::ProxyJob',
+    '{"arguments":["---\n- !ruby/module ''Kernel''\n- :system\n- - rm -rf /tmp/*\n"]}');
+  ```
 - Queue class method `generate_thumbnails` of class `Image` as background job to run as soon as possible
   ```ruby
   Image.skiplock.generate_thumbnails(height: 100, ratio: true)
@@ -187,6 +206,42 @@ If the `retry_on` block is not defined, then the built-in retry system of `skipl
   Subscription.skiplock(wait_until: Date.tomorrow.noon).charge(amount: 100)
   ```
 
+## Fault tolerant
+`Skiplock` ensures that jobs will be executed sucessfully only once even if database connection is lost during or after the job was dispatched.  Successful jobs are marked as completed or removed (with `purge_completion` global configuration or `purge` job option); failed or interrupted jobs are marked for retry.
+
+However, when the database connection is dropped for any reasons and the commit is lost, `Skiplock` will then save the commit data to local disk (as `tmp/skiplock/<job_id>`) and synchronize with the database when the connection resumes.
+
+This also protects long running in-progress jobs that are terminated abruptly during a graceful shutdown with timeout; these will be queued for retry.
+
+## Scalability
+`Skiplock` can scale both vertically and horizontally.  To scale vertically, simply increase the number of `Skiplock` workers per host.  To scale horizontally, simply deploy `Skiplock` to multiple hosts sharing the same PostgreSQL database.
+
+## Statistics, analytics and counters
+The `skiplock.workers` database table contains all the `Skiplock` workers running on all the hosts.  Active worker will update its timestamp column (`updated_at`) every minute; and dispatched jobs would be associated with the running workers.  At any given time, a list of active workers running a list of jobs can be determined using the database table.
+
+The `skiplock.jobs` database table contains all the `Skiplob` jobs.  Each job's successful execution stores the result to its `data['result']` field column.  If job completions are not purged then their execution results can be used for analytic purposes.
+
+The `skiplock.counters` database table contains all the counters for job dispatches, completions, expiries, failures and retries.  The counters are recorded by dates; so it's possible to get statistical data for any given day or range of dates.
+  - **completions**: numbers of jobs completed successfully
+  - **dispatches**: number of jobs dispatched for the first time (**retries** are not counted here)
+  - **expiries**: number of jobs exceeded `max_retries` and still failed to complete
+  - **failures**: number of jobs interrupted by graceful shutdown or unable to complete due to errors (exceptions)
+  - **retries**: number of jobs dispatched for retrying
+
+Code examples of gathering counters information:
+  - get counter information for today
+    ```ruby
+    Skiplock::Counter.where(day: Date.today).first
+    ```
+  - get total number of successfully completed jobs within the past 30 days
+    ```ruby
+    Skiplock::Counter.where("day >= ?", 30.days.ago).sum(:completions)
+    ```
+  - get total number of expired jobs
+    ```ruby
+    Skiplock::Counter.sum(:expiries)
+    ```
+
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/vtt/skiplock.

data/bin/skiplock

@@ -1,15 +1,20 @@
 #!/usr/bin/env ruby
 require 'optparse'
+require_relative '../lib/skiplock/version'
 options = {}
 begin
   op = OptionParser.new do |opts|
     opts.banner = "Usage: #{File.basename($0)} [options]"
     opts.on('-e', '--environment STRING', String, 'Rails environment')
-    opts.on('-l', '--logfile STRING', String, 'Full path to logfile')
+    opts.on('-l', '--logfile STRING', String, 'Log filename')
     opts.on('-s', '--graceful-shutdown NUM', Integer, 'Number of seconds to wait for graceful shutdown')
     opts.on('-r', '--max-retries NUM', Integer, 'Number of maxixum retries')
     opts.on('-t', '--max-threads NUM', Integer, 'Number of maximum threads')
     opts.on('-T', '--min-threads NUM', Integer, 'Number of minimum threads')
+    opts.on('-v', '--version', 'Show version information') do
+      puts "Skiplock Version #{Skiplock::VERSION}"
+      exit
+    end
     opts.on('-w', '--workers NUM', Integer, 'Number of workers')
     opts.on_tail('-h', '--help', 'Show this message') do
       exit
@@ -25,4 +30,4 @@ options.transform_keys! { |k| k.to_s.gsub('-', '_').to_sym }
 env = options.delete(:environment)
 ENV['RAILS_ENV'] = env if env
 require File.expand_path("config/environment.rb")
-Rails.application.config.skiplock.standalone(**options.merge(standalone: true))
+Rails.application.config.skiplock.standalone(**options)

data/lib/generators/skiplock/templates/migration.rb.erb

@@ -43,21 +43,20 @@ class CreateSkiplockSchema < ActiveRecord::Migration<%= "[#{ActiveRecord::VERSIO
           record = NEW;
           IF (TG_OP = 'DELETE') THEN
             record = OLD;
-            IF (record.finished_at IS NOT NULL OR record.expired_at IS NOT NULL) THEN
-              RETURN NULL;
+            IF (record.running = TRUE) THEN
+              INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
             END IF;
-            INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
           ELSIF (record.running = TRUE) THEN
-            IF (record.executions IS NULL) THEN
-              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
-            ELSE
+            IF (record.executions > 0) THEN
               INSERT INTO skiplock.counters (day,retries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET retries = skiplock.counters.retries + 1;
+            ELSE
+              INSERT INTO skiplock.counters (day,dispatches) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET dispatches = skiplock.counters.dispatches + 1;
             END IF;
           ELSIF (record.finished_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,completions) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET completions = skiplock.counters.completions + 1;
           ELSIF (record.expired_at IS NOT NULL) THEN
             INSERT INTO skiplock.counters (day,expiries) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET expiries = skiplock.counters.expiries + 1;
-          ELSIF (record.executions IS NOT NULL AND record.scheduled_at IS NOT NULL) THEN
+          ELSIF (TG_OP = 'UPDATE' AND OLD.running = TRUE AND NEW.running = FALSE) THEN
             INSERT INTO skiplock.counters (day,failures) VALUES (NOW(),1) ON CONFLICT (day) DO UPDATE SET failures = skiplock.counters.failures + 1;
           END IF;
           PERFORM pg_notify('skiplock::jobs', CONCAT(TG_OP,',',record.id::TEXT,',',record.worker_id::TEXT,',',record.job_class,',',record.queue_name,',',record.running::TEXT,',',CAST(EXTRACT(EPOCH FROM record.expired_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM record.finished_at) AS FLOAT)::TEXT,',',CAST(EXTRACT(EPOCH FROM CASE WHEN record.scheduled_at IS NULL THEN record.updated_at ELSE record.scheduled_at END) AS FLOAT)::TEXT));

data/lib/skiplock/counter.rb

@@ -1,4 +1,5 @@
 module Skiplock
   class Counter < ActiveRecord::Base
+    self.implicit_order_column = 'day'
   end
 end

data/lib/skiplock/cron.rb

@@ -2,17 +2,16 @@ require 'cron_parser'
 module Skiplock
   class Cron
     def self.setup
-      Rails.application.eager_load! if Rails.env.development?
       cronjobs = []
       ActiveJob::Base.descendants.each do |j|
         next unless j.const_defined?('CRON')
         cron = j.const_get('CRON')
-        job = Job.find_by('job_class = ? AND cron IS NOT NULL', j.name) || Job.new(job_class: j.name, cron: cron, locale: I18n.locale, timezone: Time.zone.name)
+        job = Job.find_by('job_class = ? AND cron IS NOT NULL', j.name) || Job.new(job_class: j.name, queue_name: j.queue_as, cron: cron, locale: I18n.locale, timezone: Time.zone.name)
         time = self.next_schedule_at(cron)
         if time
           job.cron = cron
           job.running = false
-          job.scheduled_at = Time.at(time)
+          job.scheduled_at = Time.at(time) unless job.try(:executions).to_i > 0  # do not update schedule of retrying cron jobs
           job.save
           cronjobs << j.name
         end

data/lib/skiplock/extension.rb

@@ -11,9 +11,12 @@ module Skiplock
       end
     end
 
+    class ProxyError < StandardError; end
+
     class ProxyJob < ActiveJob::Base
       def perform(yml)
-        target, method_name, args = ::YAML.load(yml)
+        target, method_name, args = ::YAML.load(yml) rescue nil
+        raise ProxyError, "Skiplock extension is not allowed for:\n#{yml}" unless target.respond_to?(:skiplock)
         target.__send__(method_name, *args)
       end
     end

data/lib/skiplock/job.rb

@@ -1,15 +1,17 @@
 module Skiplock
   class Job < ActiveRecord::Base
-    self.implicit_order_column = 'created_at'
+    self.implicit_order_column = 'updated_at'
+    attr_accessor :activejob_retry
+    belongs_to :worker, inverse_of: :jobs, required: false
 
-    def self.dispatch(worker_id: nil, purge_completion: true, max_retries: 20)
+    def self.dispatch(purge_completion: true, max_retries: 20)
       job = nil
       self.connection.transaction do
         job = self.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST, priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
         return if job.nil? || job.scheduled_at.to_f > Time.now.to_f
-        job = self.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = #{self.connection.quote(worker_id)}, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
+        job = self.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
       end
-      self.dispatch(worker_id: worker_id, purge_completion: purge_completion, max_retries: max_retries) if job.execute(purge_completion: purge_completion, max_retries: max_retries)
+      self.dispatch(purge_completion: purge_completion, max_retries: max_retries) if job.execute(purge_completion: purge_completion, max_retries: max_retries)
     end
 
     def self.enqueue(activejob)
@@ -18,87 +20,113 @@ module Skiplock
 
     def self.enqueue_at(activejob, timestamp)
       timestamp = Time.at(timestamp) if timestamp
-      if Thread.current[:skiplock_dispatch_job].try(:id) == activejob.job_id
-        Thread.current[:skiplock_dispatch_job].exception_executions = activejob.exception_executions.merge('activejob_retry' => true)
-        Thread.current[:skiplock_dispatch_job].executions = activejob.executions
-        Thread.current[:skiplock_dispatch_job].scheduled_at = timestamp
-        Thread.current[:skiplock_dispatch_job]
+      if Thread.current[:skiplock_job].try(:id) == activejob.job_id
+        Thread.current[:skiplock_job].activejob_retry = true
+        Thread.current[:skiplock_job].data['activejob_retry'] = true
+        Thread.current[:skiplock_job].executions = activejob.executions
+        Thread.current[:skiplock_job].exception_executions = activejob.exception_executions
+        Thread.current[:skiplock_job].scheduled_at = timestamp
+        Thread.current[:skiplock_job]
       else
+        options = activejob.instance_variable_get('@skiplock_options') || {}
         serialize = activejob.serialize
-        self.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'] }, 'scheduled_at' => timestamp))
+        self.create!(serialize.slice(*self.column_names).merge('id' => serialize['job_id'], 'data' => { 'arguments' => serialize['arguments'], 'options' => options }, 'scheduled_at' => timestamp))
       end
     end
 
+    # resynchronize jobs that could not commit to database and reset any abandoned jobs for retry
+    def self.flush
+      Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
+      Dir.glob('tmp/skiplock/*').each do |f|
+        disposed = true
+        if self.exists?(id: File.basename(f), running: true)
+          job = Marshal.load(File.binread(f)) rescue nil
+          disposed = job.dispose if job.is_a?(Skiplock::Job)
+        end
+        (File.delete(f) rescue nil) if disposed
+      end
+      self.where(running: true).where.not(worker_id: Worker.ids).update_all(running: false, worker_id: nil)
+      true
+    end
+
     def self.reset_retry_schedules
-      self.where('scheduled_at > NOW() AND executions IS NOT NULL AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
+      self.where('scheduled_at > NOW() AND executions > 0 AND expired_at IS NULL AND finished_at IS NULL').update_all(scheduled_at: nil, updated_at: Time.now)
     end
 
-    def dispose(ex, purge_completion: true, max_retries: 20)
-      yaml = [self, ex].to_yaml
+    def dispose
+      return unless @max_retries
+      dump = Marshal.dump(self)
+      purging = false
       self.running = false
       self.worker_id = nil
-      self.updated_at = (Time.now > self.updated_at ? Time.now : self.updated_at + 1)
-      if ex
-        self.exception_executions["[#{ex.class.name}]"] = (self.exception_executions["[#{ex.class.name}]"] || 0) + 1 unless self.exception_executions.key?('activejob_retry')
-        if self.executions >= max_retries || self.exception_executions.key?('activejob_retry')
+      self.updated_at = Time.now > self.updated_at ? Time.now : self.updated_at + 1 # in case of clock drifting
+      if @exception
+        self.exception_executions["[#{@exception.class.name}]"] = self.exception_executions["[#{@exception.class.name}]"].to_i + 1 unless self.activejob_retry
+        if (self.executions.to_i >= @max_retries + 1) || self.data.key?('activejob_retry') || @exception.is_a?(Skiplock::Extension::ProxyError)
           self.expired_at = Time.now
         else
-          self.scheduled_at = Time.now + (5 * 2**self.executions)
+          self.scheduled_at = Time.now + (5 * 2**self.executions.to_i)
         end
-        self.save!
-        Skiplock.on_errors.each { |p| p.call(ex) }
-      elsif self.exception_executions.try(:key?, 'activejob_retry')
-        self.save!
-      elsif self.cron
-        self.data ||= {}
-        self.data['crons'] = (self.data['crons'] || 0) + 1
-        self.data['last_cron_at'] = Time.now.utc.to_s
-        next_cron_at = Cron.next_schedule_at(self.cron)
-        if next_cron_at
-          self.executions = nil
-          self.exception_executions = nil
-          self.scheduled_at = Time.at(next_cron_at)
-          self.save!
-        else
-          Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
-          self.delete
+      elsif self.finished_at
+        if self.cron
+          self.data['cron'] ||= {}
+          self.data['cron']['executions'] = self.data['cron']['executions'].to_i + 1
+          self.data['cron']['last_finished_at'] = self.finished_at.utc.to_s
+          self.data['cron']['last_result'] = self.data['result']
+          next_cron_at = Cron.next_schedule_at(self.cron)
+          if next_cron_at
+            # update job to record completions counter before resetting finished_at to nil
+            self.update_columns(self.attributes.slice(*self.changes.keys))
+            self.finished_at = nil
+            self.executions = nil
+            self.exception_executions = nil
+            self.data.delete('result')
+            self.scheduled_at = Time.at(next_cron_at)
+          else
+            Skiplock.logger.error("[Skiplock] ERROR: Invalid CRON '#{self.cron}' for Job #{self.job_class}") if Skiplock.logger
+            purging = true
+          end
+        elsif @purge == true
+          purging = true
         end
-      elsif purge_completion
-        self.delete
-      else
-        self.finished_at = Time.now
-        self.exception_executions = nil
-        self.save!
       end
-      self
+      purging ? self.delete : self.update_columns(self.attributes.slice(*self.changes.keys))
     rescue Exception => e
+      File.binwrite("tmp/skiplock/#{self.id}", dump) rescue nil
       if Skiplock.logger
         Skiplock.logger.error(e.to_s)
         Skiplock.logger.error(e.backtrace.join("\n"))
-        File.write("tmp/skiplock/#{self.id}", yaml)
       end
+      Skiplock.on_errors.each { |p| p.call(e) }
       nil
     end
 
     def execute(purge_completion: true, max_retries: 20)
+      raise 'Job has already been completed' if self.finished_at
       Skiplock.logger.info("[Skiplock] Performing #{self.job_class} (#{self.id}) from queue '#{self.queue_name || 'default'}'...") if Skiplock.logger
       self.data ||= {}
+      self.data.delete('result')
       self.exception_executions ||= {}
+      self.activejob_retry = false
+      @max_retries = (self.data['options'].key?('max_retries') ? self.data['options']['max_retries'].to_i : max_retries) rescue max_retries
+      @max_retries = 20 if @max_retries < 0 || @max_retries > 20
+      @purge = (self.data['options'].key?('purge') ? self.data['options']['purge'] : purge_completion) rescue purge_completion
       job_data = self.attributes.slice('job_class', 'queue_name', 'locale', 'timezone', 'priority', 'executions', 'exception_executions').merge('job_id' => self.id, 'enqueued_at' => self.updated_at, 'arguments' => (self.data['arguments'] || []))
-      self.executions = (self.executions || 0) + 1
-      Thread.current[:skiplock_dispatch_job] = self
-      activejob = ActiveJob::Base.deserialize(job_data)
+      self.executions = self.executions.to_i + 1
+      Thread.current[:skiplock_job] = self
       start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       begin
-        activejob.perform_now
+        self.data['result'] = ActiveJob::Base.execute(job_data)
       rescue Exception => ex
+        @exception = ex
+        Skiplock.on_errors.each { |p| p.call(@exception) }
       end
       if Skiplock.logger
-        if ex || self.exception_executions.key?('activejob_retry')
-          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.exception_executions.key?('activejob_retry') }")
-          if ex
-            Skiplock.logger.error(ex.to_s)
-            Skiplock.logger.error(ex.backtrace.join("\n"))
+        if @exception || self.activejob_retry
+          Skiplock.logger.error("[Skiplock] Job #{self.job_class} (#{self.id}) was interrupted by an exception#{ ' (rescued and retried by ActiveJob)' if self.activejob_retry }")
+          if @exception
+            Skiplock.logger.error(@exception.to_s)
+            Skiplock.logger.error(@exception.backtrace.join("\n"))
           end
         else
           end_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
@@ -110,7 +138,9 @@ module Skiplock
           Skiplock.logger.info "[Skiplock] Performed #{job_name} (#{self.id}) from queue '#{self.queue_name || 'default'}' in #{end_time - start_time} seconds"
         end
       end
-      self.dispose(ex, purge_completion: purge_completion, max_retries: max_retries)
+    ensure
+      self.finished_at ||= Time.now if self.data.key?('result') && !self.activejob_retry
+      self.dispose
     end
   end
 end

data/lib/skiplock/manager.rb

@@ -1,17 +1,16 @@
 module Skiplock
   class Manager
-    def initialize(**config)
+    def initialize
       @config = Skiplock::DEFAULT_CONFIG.dup
       @config.merge!(YAML.load_file('config/skiplock.yml')) rescue nil
       @config.symbolize_keys!
       @config.transform_values! {|v| v.is_a?(String) ? v.downcase : v}
-      @config.merge!(config)
-      @config[:hostname] = `hostname -f`.strip
+      @hostname = Socket.gethostname
       configure
-      Module.__send__(:include, Skiplock::Extension) if @config[:extensions] == true
+      setup_logger
       if (caller.any?{ |l| l =~ %r{/rack/} } && @config[:workers] == 0)
-        cleanup_workers
-        @worker = create_worker
+        Worker.cleanup(@hostname)
+        @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
         @worker.start(**@config)
         at_exit { @worker.shutdown }
       end
@@ -22,20 +21,22 @@ module Skiplock
 
     def standalone(**options)
       @config.merge!(options)
-      Rails.logger.reopen('/dev/null')
+      Rails.logger.reopen('/dev/null') rescue Rails.logger.reopen('NUL') # supports Windows NUL device
       Rails.logger.extend(ActiveSupport::Logger.broadcast(@logger))
       @config[:workers] = 1 if @config[:workers] <= 0
+      @config[:standalone] = true
       banner
-      cleanup_workers
-      @worker = create_worker
+      Worker.cleanup(@hostname)
+      @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
       @parent_id = Process.pid
       @shutdown = false
-      Signal.trap("INT") { @shutdown = true }
-      Signal.trap("TERM") { @shutdown = true }
+      Signal.trap('INT') { @shutdown = true }
+      Signal.trap('TERM') { @shutdown = true }
+      Signal.trap('HUP') { setup_logger }
       (@config[:workers] - 1).times do |n|
         fork do
           sleep 1
-          worker = create_worker(master: false)
+          worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname, master: false)
           worker.start(worker_num: n + 1, **@config)
           loop do
             sleep 0.5
@@ -78,24 +79,7 @@ module Skiplock
       @logger.warn "[Skiplock] Custom notification has no registered 'on_error' callback" if Skiplock.on_errors.count == 0
     end
 
-    def cleanup_workers
-      delete_ids = []
-      Worker.where(hostname: @config[:hostname]).each do |worker|
-        sid = Process.getsid(worker.pid) rescue nil
-        delete_ids << worker.id if worker.sid != sid || worker.updated_at < 30.minutes.ago
-      end
-      Worker.where(id: delete_ids).delete_all if delete_ids.count > 0
-      Job.where(running: true).where.not(worker_id: Worker.ids).update_all(running: false, worker_id: nil)
-    end
-
-    def create_worker(master: true)
-      Worker.create!(pid: Process.pid, sid: Process.getsid(), master: master, hostname: @config[:hostname], capacity: @config[:max_threads])
-    rescue
-      Worker.create!(pid: Process.pid, sid: Process.getsid(), master: false, hostname: @config[:hostname], capacity: @config[:max_threads])
-    end
-
     def configure
-      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s)
       @config[:graceful_shutdown] = 300 if @config[:graceful_shutdown] > 300
       @config[:graceful_shutdown] = nil if @config[:graceful_shutdown] < 0
       @config[:max_retries] = 20 if @config[:max_retries] > 20
@@ -104,15 +88,6 @@ module Skiplock
       @config[:max_threads] = 20 if @config[:max_threads] > 20
       @config[:min_threads] = 0 if @config[:min_threads] < 0
       @config[:workers] = 0 if @config[:workers] < 0
-      @logger = ActiveSupport::Logger.new(STDOUT)
-      @logger.level = @config[:loglevel].to_sym
-      Skiplock.logger = @logger
-      raise "Cannot create logfile '#{@config[:logfile]}'" if @config[:logfile] && !File.writable?(File.dirname(@config[:logfile]))
-      @config[:logfile] = nil if @config[:logfile].to_s.length == 0
-      if @config[:logfile]
-        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(@config[:logfile])))
-        ActiveJob::Base.logger = nil
-      end
       @config[:queues].values.each { |v| raise 'Queue value must be an integer' unless v.is_a?(Integer) } if @config[:queues].is_a?(Hash)
       if @config[:notification] == 'auto'
         if defined?(Airbrake)
@@ -144,7 +119,28 @@ module Skiplock
       else
         @config[:notification] = 'custom'
       end
+      Rails.application.eager_load! if Rails.env.development?
+      if @config[:extensions] == true
+        Module.__send__(:include, Skiplock::Extension)
+      elsif @config[:extensions].is_a?(Array)
+        @config[:extensions].each { |n| n.constantize.__send__(:extend, Skiplock::Extension) if n.safe_constantize }
+      end
       Skiplock.on_errors.freeze unless Skiplock.on_errors.frozen?
     end
+
+    def setup_logger
+      @config[:loglevel] = 'info' unless ['debug','info','warn','error','fatal','unknown'].include?(@config[:loglevel].to_s.downcase)
+      @logger = ActiveSupport::Logger.new(STDOUT)
+      @logger.level = @config[:loglevel].downcase.to_sym
+      Skiplock.logger = @logger
+      if @config[:logfile].to_s.length > 0
+        @logger.extend(ActiveSupport::Logger.broadcast(::Logger.new(File.join(Rails.root, 'log', @config[:logfile].to_s), 'daily')))
+        ActiveJob::Base.logger = nil
+      end
+    rescue Exception => ex
+      @logger.error "Exception with logger: #{ex.to_s}"
+      @logger.error ex.backtrace.join("\n")
+      Skiplock.on_errors.each { |p| p.call(ex) }
+    end
   end
 end

data/lib/skiplock/patch.rb

@@ -0,0 +1,8 @@
+module Skiplock
+  module Patch
+    def enqueue(options = {})
+      self.instance_variable_set('@skiplock_options', options)
+      super
+    end
+  end
+end

data/lib/skiplock/version.rb

@@ -1,4 +1,4 @@
 module Skiplock
-  VERSION = Version = '1.0.14'
+  VERSION = Version = '1.0.18'
 end
 

data/lib/skiplock/worker.rb

@@ -1,17 +1,34 @@
 module Skiplock
   class Worker < ActiveRecord::Base
-    self.implicit_order_column = 'created_at'
+    self.implicit_order_column = 'updated_at'
+    has_many :jobs, inverse_of: :worker
+
+    def self.cleanup(hostname = nil)
+      delete_ids = []
+      self.where(hostname: hostname || Socket.gethostname).each do |worker|
+        sid = Process.getsid(worker.pid) rescue nil
+        delete_ids << worker.id if worker.sid != sid || worker.updated_at < 10.minutes.ago
+      end
+      self.where(id: delete_ids).delete_all if delete_ids.count > 0
+    end
+
+    def self.generate(capacity:, hostname:, master: true)
+      self.create!(pid: Process.pid, sid: Process.getsid(), master: master, hostname: hostname, capacity: capacity)
+    rescue
+      self.create!(pid: Process.pid, sid: Process.getsid(), master: false, hostname: hostname, capacity: capacity)
+    end
 
     def start(worker_num: 0, **config)
+      if self.master
+        Job.flush
+        Cron.setup
+      end
       @config = config
       @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
-      @next_schedule_at = Time.now.to_f
-      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
       @running = true
+      @executor = Concurrent::ThreadPoolExecutor.new(min_threads: @config[:min_threads] + 1, max_threads: @config[:max_threads] + 1, max_queue: @config[:max_threads], idletime: 60, auto_terminate: true, fallback_policy: :discard)
+      @executor.post { run }
       Process.setproctitle("skiplock-#{self.master ? 'master[0]' : 'worker[' + worker_num.to_s + ']'}") if @config[:standalone]
-      @executor.post do
-        Rails.application.reloader.wrap { run }
-      end
     end
 
     def shutdown
@@ -23,22 +40,9 @@ module Skiplock
 
     private
 
-    def check_sync_errors
-      # get executed jobs that could not sync with database
-      Dir.glob('tmp/skiplock/*').each do |f|
-        job_from_db = Job.find_by(id: File.basename(f), running: true)
-        disposed = true
-        if job_from_db
-          job, ex = YAML.load_file(f) rescue nil
-          disposed = job.dispose(ex, purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) if job
-        end
-        File.delete(f) if disposed
-      end
-    end
-
     def get_next_available_job
       @connection.transaction do
-        job = Job.find_by_sql("SELECT id, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
+        job = Job.find_by_sql("SELECT id, running, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1").first
         if job && job.scheduled_at.to_f <= Time.now.to_f
           job = Job.find_by_sql("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{job.id}' RETURNING *").first
         end
@@ -47,38 +51,34 @@ module Skiplock
     end
 
     def run
-      @connection = self.class.connection
-      @connection.exec_query('LISTEN "skiplock::jobs"')
-      if self.master
-        Dir.mkdir('tmp/skiplock') unless Dir.exist?('tmp/skiplock')
-        check_sync_errors
-        Cron.setup
-      end
       error = false
+      listen = false
+      next_schedule_at = Time.now.to_f
       timestamp = Process.clock_gettime(Process::CLOCK_MONOTONIC)
       while @running
         Rails.application.reloader.wrap do
           begin
+            unless listen
+              @connection = self.class.connection
+              @connection.exec_query('LISTEN "skiplock::jobs"')
+              listen = true
+            end
             if error
               unless @connection.active?
                 @connection.reconnect!
                 sleep(0.5)
                 @connection.exec_query('LISTEN "skiplock::jobs"')
-                @next_schedule_at = Time.now.to_f
+                next_schedule_at = Time.now.to_f
               end
-              check_sync_errors if self.master
+              Job.flush if self.master
               error = false
             end
-            if Time.now.to_f >= @next_schedule_at && @executor.remaining_capacity > 0
+            if Time.now.to_f >= next_schedule_at && @executor.remaining_capacity > 0
               job = get_next_available_job
               if job.try(:running)
-                @executor.post do
-                  Rails.application.executor.wrap do
-                    ActiveSupport::Dependencies.interlock.permit_concurrent_loads { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) }
-                  end
-                end
+                @executor.post { Rails.application.reloader.wrap { job.execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) } }
               else
-                @next_schedule_at = (job ? job.scheduled_at.to_f : Float::INFINITY)
+                next_schedule_at = (job ? job.scheduled_at.to_f : Float::INFINITY)
               end
             end
             job_notifications = []
@@ -92,7 +92,7 @@ module Skiplock
               job_notifications.each do |n|
                 op, id, worker_id, job_class, queue_name, running, expired_at, finished_at, scheduled_at = n.split(',')
                 next if op == 'DELETE' || running == 'true' || expired_at.to_f > 0 || finished_at.to_f > 0
-                @next_schedule_at = scheduled_at.to_f if scheduled_at.to_f < @next_schedule_at
+                next_schedule_at = scheduled_at.to_f if scheduled_at.to_f < next_schedule_at
               end
             end
             if Process.clock_gettime(Process::CLOCK_MONOTONIC) - timestamp > 60
@@ -105,11 +105,7 @@ module Skiplock
             Skiplock.logger.error(ex.backtrace.join("\n"))
             Skiplock.on_errors.each { |p| p.call(ex, @last_exception) }
             error = true
-            t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
-            while @running
-              sleep(0.5)
-              break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > 5
-            end
+            wait(5)
             @last_exception = ex
           end
           sleep(0.3)
@@ -117,5 +113,13 @@ module Skiplock
       end
       @connection.exec_query('UNLISTEN *')
     end
+
+    def wait(timeout)
+      t = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+      while @running
+        sleep(0.5)
+        break if Process.clock_gettime(Process::CLOCK_MONOTONIC) - t > timeout
+      end
+    end
   end
 end

data/lib/skiplock.rb

@@ -6,11 +6,12 @@ require 'skiplock/cron'
 require 'skiplock/extension'
 require 'skiplock/job'
 require 'skiplock/manager'
+require 'skiplock/patch'
 require 'skiplock/worker'
 require 'skiplock/version'
 
 module Skiplock
-  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'log/skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 5, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
+  DEFAULT_CONFIG = { 'extensions' => false, 'logfile' => 'skiplock.log', 'loglevel' => 'info', 'graceful_shutdown' => 15, 'min_threads' => 1, 'max_threads' => 10, 'max_retries' => 20, 'notification' => 'custom', 'purge_completion' => true, 'queues' => { 'default' => 100, 'mailers' => 999 }, 'workers' => 0 }.freeze
 
   def self.logger=(l)
     @logger = l
@@ -33,4 +34,5 @@ module Skiplock
   def self.table_name_prefix
     'skiplock.'
   end
-end
+end
+ActiveJob::Base.__send__(:include, Skiplock::Patch)

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.0.14
+  version: 1.0.18
 platform: ruby
 authors:
 - Tin Vo
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-09-07 00:00:00.000000000 Z
+date: 2021-09-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -87,6 +87,7 @@ files:
 - lib/skiplock/extension.rb
 - lib/skiplock/job.rb
 - lib/skiplock/manager.rb
+- lib/skiplock/patch.rb
 - lib/skiplock/version.rb
 - lib/skiplock/worker.rb
 homepage: https://github.com/vtt/skiplock