skiplock 1.1.8 → 1.1.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2ff56b97e28d0f5b162dfb538e33461569af6aa19ba062a85d7efb4ad0d129f2
-  data.tar.gz: 32472f59ee0ed8422fadfa52bb4cb59a15518c0211d58f562006748e30a1d72c
+  metadata.gz: 21372493169f916f319d425482e65f015777befaa5e4a65a9c29895a65f87719
+  data.tar.gz: c4ddc48122b091efb1bfc378e23c41e53f748251c914cc42b34f2bc6c0a51a97
 SHA512:
-  metadata.gz: 3b1a4e1314d300ae26e93f77256c6d543d46a425a44ab12d20b07e6c14e5d8d070db8cab532e6fd0fcb0b636fdde7cbee78aff65651e2ee9694b2eab03bfc276
-  data.tar.gz: e549e1db28766922325f8ef7b3b59e205df5bc6a6a5b3ca5b40899861c6baf732d973144aa0de2a09bc4d0fa42dc2fe3ec45958696fbbcb93fd60a8bc8b866cc
+  metadata.gz: 8d4bb1df0948887cc5ad7a70e6e586bb021040817848cf649030918102cfacb09ca4a70e051d259a17348133ed88a287ba20cbe8e68e3cff817c56b3bae068da
+  data.tar.gz: 2ea888f4e274f58e0617548857bcdc3936793e6e1270d70e0b51fcdfa3c06f1793fbf37f41cbe9deb9e83eed5bc8d5861924659cab95205433d48cb7fa42c955

data/README.md

@@ -76,7 +76,7 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     - **log_level** (*string*): sets logging level (`debug, info, warn, error, fatal, unknown`)
     - **log_count** (*integer*): number of log files to keep (ie: log rotation)
     - **log_size** (*integer*): maximum size per log file (in bytes)
-    - **namespace** (*string*): sets namespace for jobs (workers will only process jobs of specified namespace)
+    - **namespace** (*string*): routes a category of jobs to workers configured with the same namespace; an empty value selects the default, unnamespaced category. Cron jobs always use the default category.
     - **notification** (*string*): sets the library to be used for notifying errors and exceptions (`auto, airbrake, bugsnag, exception_notification, custom`); using `auto` will detect library if available.  See `Notification system` for more details
     - **extensions** (*multi*): enable or disable the class method extension.  See `ClassMethod extension` for more details
     - **purge_completion** (*boolean*): when set to **true** will delete jobs after they were completed successfully; if set to **false** then the completed jobs should be purged periodically to maximize performance (eg. clean up old jobs after 3 months); queued jobs can manually override using `purge` option
@@ -86,8 +86,12 @@ The library is quite small compared to other PostgreSQL job queues (eg. *delay_j
     #### **Async mode**
     When **workers** is set to **0** then the jobs will be performed in the web server process using separate threads.  If using multi-worker cluster mode web server like Puma, then all the Puma workers will also be able to perform `Skiplock` jobs.
 
+    In async mode the application server owns process signals. Skiplock does not install `INT` or `TERM` handlers; it shuts down its thread pool from a process-local, idempotent exit callback. This allows Puma workers to finish jobs for up to `graceful_shutdown` seconds without Skiplock interfering with Puma's signal handling. Puma's own worker shutdown timeout must allow at least the same amount of time. Async workers must be initialized after Puma forks; if `preload_app!` is enabled, use a Puma worker-boot hook or run Skiplock in standalone mode because worker threads are not inherited across a fork.
+
     #### **Standalone mode**
     `Skiplock` standalone mode can be launched by using the `skiplock` executable; command line options can be provided to override the `Skiplock` configuration file.  
+
+    In standalone multi-process mode the Skiplock parent owns its child processes. On `INT` or `TERM`, it forwards `TERM` to every child, shuts down all worker thread pools against one shared `graceful_shutdown` deadline, and terminates any child that remains after the deadline.
     ```
     $ bundle exec skiplock -h
     Usage: skiplock [options]
@@ -140,6 +144,8 @@ Outside the Rails application:
 
 ## Cron system
 `Skiplock` provides the capability to setup cron jobs for running tasks periodically.  It fully supports the cron syntax to specify the frequency of the jobs.  To setup a cron job, simply assign a valid cron schedule to the constant `CRON` for the Job Class.
+
+Cron schedules are global and are intentionally stored in the default, empty Skiplock namespace. Ruby module namespaces remain part of the complete job class name, so `Reports::CleanupJob` and `Mail::CleanupJob` are distinct schedules. At least one unnamespaced worker must be running to execute cron jobs; workers configured with a nonempty Skiplock namespace do not process them. Both async and standalone master workers reconcile cron definitions at startup.
 - setup `MyJob` to run as cron job every hour at 30 minutes past
 
     ```ruby
@@ -176,7 +182,7 @@ Outside the Rails application:
     ```
 If the retry attempt limit configured in ActiveJob has been reached, then the control will be passed back to `Skiplock` to be marked as an expired job.
 
-If the `retry_on` block is not defined, then the built-in retry system of `Skiplock` will kick in automatically.  The retrying schedule is using an exponential formula (5 + 2**attempt).  The `Skiplock` configuration `max_retries` determines the the limit of attempts before the failing job is marked as expired.  The maximum retry limit can be set as high as 20; this allows up to 12 days of retrying before the job is marked as expired.
+If the `retry_on` block is not defined, then the built-in retry system of `Skiplock` will kick in automatically. The delay before each retry uses the exponential formula `5 + 2**attempt` seconds. The first retry is delayed 7 seconds; at the maximum supported retry count of 20, the final individual delay is approximately 12.1 days. The cumulative delay across all 20 retries is approximately 24.3 days. The `Skiplock` configuration `max_retries` determines the limit of attempts before the failing job is marked as expired.
 
 ## Notification system
 `Skiplock` can use existing exception notification library to notify errors and exceptions.  It supports `airbrake`, `bugsnag`, and `exception_notification`.  Custom notification can also be called whenever an exception occurs; it can be configured in an initializer like below:
@@ -213,12 +219,16 @@ To enable extension for specific classes and modules only then set the configura
   ```
 
 ## Fault tolerant
-`Skiplock` ensures that jobs will be executed sucessfully only once even if database connection is lost during or after the job was dispatched.  Successful jobs are marked as completed or removed (with `purge_completion` global configuration or `purge` job option); failed or interrupted jobs are marked for retry.
+`Skiplock` ensures that jobs will be executed successfully only once even if database connection is lost during or after the job was dispatched.  Successful jobs are marked as completed or removed (with `purge_completion` global configuration or `purge` job option); failed or interrupted jobs are marked for retry.
+
+"Successfully only once" means that a job record can be recorded as successfully completed only once. Failed or interrupted attempts may execute again and do not count as a successful completion. Jobs that produce external, non-transactional side effects such as SMTP delivery or HTTP requests should therefore be idempotent, because a process can be interrupted after the external system accepts an operation but before Skiplock records the successful completion.
 
 However, when the database connection is dropped for any reasons and the commit is lost, `Skiplock` will then save the commit data to local disk (as `tmp/skiplock/<job_id>`) and synchronize with the database when the connection resumes.
 
 This also protects long running in-progress jobs that are terminated abruptly during a graceful shutdown with timeout; these will be queued for retry.
 
+Worker ownership is host-affine by design. If a job is assigned to a worker on a particular host and that host becomes unavailable, another host does not take ownership of it. The job waits until the original host returns, removes its obsolete worker records, and releases the job for retry. Hosts running Skiplock should therefore be supervised and stale worker heartbeats should be monitored.
+
 ## Scalability
 `Skiplock` can scale both vertically and horizontally.  To scale vertically, simply increase the number of `Skiplock` workers per host.  To scale horizontally, simply deploy `Skiplock` to multiple hosts sharing the same PostgreSQL database.
 

data/lib/skiplock/job.rb

@@ -69,7 +69,7 @@ module Skiplock
         if (self.executions.to_i >= self.max_retries + 1) || self.exception_executions.key?('activejob_error') || self.exception.is_a?(Skiplock::Extension::ProxyError)
           self.expired_at = Time.now
         else
-          self.scheduled_at = Time.now + (5 * 2**self.executions.to_i)
+          self.scheduled_at = Time.now + (5 + 2**self.executions.to_i)
         end
       elsif self.finished_at
         if self.cron

data/lib/skiplock/manager.rb

@@ -12,14 +12,28 @@ module Skiplock
       configure
       Worker.cleanup(@hostname)
       @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
+      @worker_pid = Process.pid
       Cron.setup if @worker.master
       @worker.start(**@config)
-      at_exit { @worker.shutdown }
+      at_exit { shutdown }
     rescue Exception => ex
       @logger.error(ex.to_s)
       @logger.error(ex.backtrace.join("\n"))
     end
 
+    # Puma owns process signals in async mode. Its worker exit invokes this through
+    # at_exit, so shutdown must be safe when callbacks run more than once and when
+    # an application was preloaded before Puma forked its workers.
+    def shutdown
+      return unless @worker && @worker_pid == Process.pid
+      @shutdown_mutex ||= Mutex.new
+      @shutdown_mutex.synchronize do
+        return if @shutdown_complete
+        @worker.shutdown
+        @shutdown_complete = true
+      end
+    end
+
     def standalone(**options)
       @config.merge!(options)
       @config[:standalone] = true
@@ -34,9 +48,12 @@ module Skiplock
       Signal.trap('HUP') { setup_logger }
       Worker.cleanup(@hostname)
       @worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname)
+      @worker_pid = Process.pid
+      Cron.setup if @worker.master
       ActiveRecord::Base.connection.disconnect! if @config[:workers] > 1
+      @child_pids = []
       (@config[:workers] - 1).times do |n|
-        fork do
+        @child_pids << fork do
           sleep(0.25*n + 1)
           ActiveRecord::Base.establish_connection
           worker = Worker.generate(capacity: @config[:max_threads], hostname: @hostname, master: false)
@@ -55,8 +72,13 @@ module Skiplock
         break if @shutdown
       end
       @logger.info "[Skiplock] Terminating signal... Waiting for jobs to finish (up to #{@config[:graceful_shutdown]} seconds)..." if @config[:graceful_shutdown]
-      Process.waitall
-      @worker.shutdown
+      deadline = Process.clock_gettime(Process::CLOCK_MONOTONIC) + @config[:graceful_shutdown] if @config[:graceful_shutdown]
+      signal_children(@child_pids, 'TERM')
+      begin
+        shutdown
+      ensure
+        reap_children(@child_pids, deadline)
+      end
     rescue Exception => ex
       @logger.error(ex.to_s)
       @logger.error(ex.backtrace.join("\n"))
@@ -64,6 +86,32 @@ module Skiplock
 
     private
 
+    def signal_children(child_pids, signal)
+      child_pids.each do |pid|
+        Process.kill(signal, pid)
+      rescue Errno::ESRCH
+      end
+    end
+
+    def reap_children(child_pids, deadline)
+      remaining = child_pids.dup
+      while remaining.any?
+        remaining.delete_if do |pid|
+          Process.waitpid(pid, Process::WNOHANG)
+        rescue Errno::ECHILD
+          true
+        end
+        break if remaining.empty?
+        break if deadline && Process.clock_gettime(Process::CLOCK_MONOTONIC) >= deadline
+        sleep 0.05
+      end
+      signal_children(remaining, 'KILL')
+      remaining.each do |pid|
+        Process.waitpid(pid)
+      rescue Errno::ECHILD
+      end
+    end
+
     def banner
       title = "Skiplock #{Skiplock::VERSION} (Rails #{Rails::VERSION::STRING} | Ruby #{RUBY_VERSION}-p#{RUBY_PATCHLEVEL})"
       @logger.info "-"*(title.length)

data/lib/skiplock/version.rb

@@ -1,3 +1,3 @@
 module Skiplock
-  VERSION = Version = '1.1.8'
+  VERSION = Version = '1.1.9'
 end

data/lib/skiplock/worker.rb

@@ -19,10 +19,25 @@ module Skiplock
     end
 
     def shutdown
-      @running = false
-      @executor.shutdown
-      @executor.kill unless @executor.wait_for_termination(@config[:graceful_shutdown])
-      Skiplock.logger.info "[Skiplock] Shutdown of #{self.master ? 'master' : 'cluster'} worker#{(' ' + @num.to_s) if @num > 0 && @config[:workers] > 2} (PID: #{self.pid}) was completed."
+      @shutdown_mutex ||= Mutex.new
+      @shutdown_mutex.synchronize do
+        return if @shutdown_complete
+        @running = false
+        @dispatch_mutex ||= Mutex.new
+        if @executor
+          @dispatch_mutex.synchronize { @executor.shutdown }
+          @executor.kill unless @executor.wait_for_termination((@config || {})[:graceful_shutdown])
+        end
+        @shutdown_complete = true
+        begin
+          self.delete
+        rescue StandardError => ex
+          Skiplock.logger.error("[Skiplock] Unable to delete worker #{self.id} during shutdown: #{ex}") if Skiplock.logger
+        end
+        worker_num = @num.to_i
+        worker_count = (@config || {})[:workers].to_i
+        Skiplock.logger.info "[Skiplock] Shutdown of #{self.master ? 'master' : 'cluster'} worker#{(' ' + worker_num.to_s) if worker_num > 0 && worker_count > 2} (PID: #{self.pid}) was completed." if Skiplock.logger
+      end
     end
 
     def start(worker_num: 0, **config)
@@ -32,6 +47,7 @@ module Skiplock
       @namespace_query = Skiplock.namespace.nil? ? "namespace IS NULL" : "namespace = '#{Skiplock.namespace}'"
       @queues_order_query = @config[:queues].map { |q,v| "WHEN queue_name = '#{q}' THEN #{v}" }.join(' ') if @config[:queues].is_a?(Hash) && @config[:queues].count > 0
       @running = true
+      @dispatch_mutex = Mutex.new
       @map = ::PG::TypeMapByOid.new
       @map.add_coder(::PG::TextDecoder::Boolean.new(oid: 16, name: 'bool'))
       @map.add_coder(::PG::TextDecoder::Integer.new(oid: 20, name: 'int8'))
@@ -73,18 +89,22 @@ module Skiplock
               next_schedule_at = Time.now.to_f
             end
             if Time.now.to_f >= next_schedule_at && @executor.remaining_capacity > 1  # reserves 1 slot in queue for Job.flush in case of pg_connection error
-              result = nil
-              @connection.transaction do |conn|
-                conn.exec("SELECT id, running, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL AND #{@namespace_query} ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1") do |r|
-                  result = r.first
-                  conn.exec("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{result['id']}' RETURNING *") { |r| result = r.first } if result && result['scheduled_at'].to_f <= Time.now.to_f
+              @dispatch_mutex.synchronize do
+                if @running
+                  result = nil
+                  @connection.transaction do |conn|
+                    conn.exec("SELECT id, running, scheduled_at FROM skiplock.jobs WHERE running = FALSE AND expired_at IS NULL AND finished_at IS NULL AND #{@namespace_query} ORDER BY scheduled_at ASC NULLS FIRST,#{@queues_order_query ? ' CASE ' + @queues_order_query + ' ELSE NULL END ASC NULLS LAST,' : ''} priority ASC NULLS LAST, created_at ASC FOR UPDATE SKIP LOCKED LIMIT 1") do |r|
+                      result = r.first
+                      conn.exec("UPDATE skiplock.jobs SET running = TRUE, worker_id = '#{self.id}', updated_at = NOW() WHERE id = '#{result['id']}' RETURNING *") { |r| result = r.first } if result && result['scheduled_at'].to_f <= Time.now.to_f
+                    end
+                  end
+                  if result && result['running']
+                    @executor.post { Rails.application.executor.wrap { Job.instantiate(result).execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) } }
+                  else
+                    next_schedule_at = (result ? result['scheduled_at'].to_f : Float::INFINITY)
+                  end
                 end
               end
-              if result && result['running']
-                @executor.post { Rails.application.executor.wrap { Job.instantiate(result).execute(purge_completion: @config[:purge_completion], max_retries: @config[:max_retries]) } }
-              else
-                next_schedule_at = (result ? result['scheduled_at'].to_f : Float::INFINITY)
-              end
             end
             notifications = { 'skiplock::jobs' => [], 'skiplock::workers' => [] }
             @connection.wait_for_notify(0.2) do |channel, pid, payload|

metadata

@@ -1,14 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: skiplock
 version: !ruby/object:Gem::Version
-  version: 1.1.8
+  version: 1.1.9
 platform: ruby
 authors:
 - Tin Vo
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-12-08 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -66,6 +65,48 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.0'
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '12.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '12.0'
 description: High performance ActiveJob Queue Adapter for PostgreSQL that provides
   maximum reliability and ACID compliance
 email:
@@ -94,7 +135,6 @@ homepage: https://github.com/vtt/skiplock
 licenses:
 - MIT
 metadata: {}
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -109,8 +149,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.16
-signing_key:
+rubygems_version: 4.0.19
 specification_version: 4
 summary: ActiveJob Queue Adapter for PostgreSQL
 test_files: []