workhorse 0.6.9 → 1.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a328b92854a461c8b0e3a67f740b21581c9652b428286fa917c7942208d5ab83
-  data.tar.gz: 802d62becfdef759b6e1edad34eec6c5c606bf1de5a19ee75347149ccf6118f9
+  metadata.gz: 6b381576f799a0b69cf90e0df551cae0f967b1462bb744383933ca3e82be785d
+  data.tar.gz: c298d93548baee24c30a739b8ff3ea60ce61e91fcaf4c40cead630b29bb024d9
 SHA512:
-  metadata.gz: ac93faa479a9f4b228c75b41d7001eadf60a395986160d86342e7423a7e9712c9abeb7e4e69463eafa7d9b36647868ad123ca9efeb776ece3accf5d28d0ec889
-  data.tar.gz: c00ef0fc1c007f7a11ff208664c6c68653cd3cdf00f5750178f56484fb95721f0aaec095b0a4e8e091ecab7700bbbddaeeadbd1b9457c1fdd18882aa224737c7
+  metadata.gz: cc45be8c2c2ee319867798427b6994d5eba4ad6623419f54c13ed500f2a722b90acc8c5dffbd5d0e92ec6a82b4de9484559eed6798e845159210eaaf00fb324a
+  data.tar.gz: 8d7926b6c830b94aad5e1e77f4951ebd68c6d0cfd4baccd4d004d63fa9aa4701673114b97371f72f05806a2616bc5fcf633e747441e860aff13fd4b90bf4d158

data/.travis.yml

@@ -1,7 +1,7 @@
 language: ruby
 rvm:
- - 2.3.0
  - 2.5.0
+ - 2.6.0
 services:
  - mysql
 before_install:

data/CHANGELOG.md

@@ -1,5 +1,61 @@
 # Workhorse Changelog
 
+## 1.0.1 - 2020-12-15
+
+* Fix handling of empty pid files
+
+## 1.0.0 - 2020-09-21
+
+* Stable release, identical to 1.0.0.beta2 but now extensively battle-tested
+
+## 1.0.0.beta2 - 2020-08-27
+
+* Add option `config.silence_poller_exceptions` (default `false`)
+
+* Add option `config.silence_watcher` (default `false`)
+
+## 1.0.0.beta1 - 2020-08-20
+
+This is a stability release that is still experimental and has to be tested in
+battle before it can be considered stable.
+
+* Stop passing ActiveRecord job objects between polling and worker threads to
+  avoid AR race conditions. Now only IDs are passed between threads.
+
+## 1.0.0.beta0 - 2020-08-19
+
+This is a stability release that is still experimental and has to be tested in
+battle before it can be considered stable.
+
+* Simplify locking during polling. Other than locking individual jobs, pollers
+  now acquire a global lock. While this can lead to many pollers waiting for
+  each others locks, performing a poll is usually done very quickly and the
+  performance drawback is to be considered neglegible. This change should work
+  around some deadlock issues as well as an issue where a job was obtained by
+  more than one poller.
+
+* Shut down worker if polling encountered any kind of error (running jobs will
+  be completed whenever possible). This leads to potential watcher jobs being
+  able to restore the failed process.
+
+* Make unit test database connection configurable using environment variables
+  `DB_NAME`, `DB_USERNAME`, `DB_PASSWORD` and `DB_HOST`. This is only relevant
+  if you are working on workhorse and need to run the unit tests.
+
+* Fix misbehaviour where queueless jobs were not picked up by workers as long as
+  a named queue was in a locked state.
+
+* Add built-in job `Workhorse::Jobs::DetectStaleJobsJob` which you can schedule.
+  It picks up jobs that remained `locked` or `started` (running) for more than a
+  certain amount of time. If any of these jobs are found, an exception is thrown
+  (which may cause a notification if you configured `on_exception` accordingly).
+  See the job's API documentation for more information.
+
+**If using oracle:** Make sure to grant execute permission to the package
+`DBMS_LOCK` for your oracle database schema:
+
+```GRANT execute ON DBMS_LOCK TO <schema-name>;```
+
 ## 0.6.9 - 2020-04-22
 
 * Fix error where processes may have mistakenly been detected as running (add a

data/FAQ.md

@@ -74,4 +74,8 @@ production mode.
 
 ## Why does workhorse not support timeouts?
 
-Generic timeout implementations are [a dangerous thing](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/) in Ruby. This is why we decided against providing this feature in Workhorse and recommend to implement the timeouts inside of your jobs - i.e. via network timeouts.
+Generic timeout implementations are [a dangerous
+thing](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/)
+in Ruby. This is why we decided against providing this feature in Workhorse and
+recommend to implement the timeouts inside of your jobs - i.e. via network
+timeouts.

data/README.md

@@ -66,6 +66,15 @@ What it does not do:
 
    Please customize the initializer and worker script to your liking.
 
+### Oracle
+
+When using Oracle databases, make sure your schema has access to the package
+`DBMS_LOCK`:
+
+```
+GRANT execute ON DBMS_LOCK TO <schema-name>;
+```
+
 ## Queuing jobs
 
 ### Basic jobs
@@ -281,6 +290,10 @@ Workhorse.setup do |config|
 end
 ```
 
+Using the settings `config.silence_poller_exceptions` and
+`config.silence_watcher`, you can silence certain exceptions / error outputs
+(both are disabled by default).
+
 ## Handling database jobs
 
 Jobs stored in the database can be accessed via the ActiveRecord model
@@ -320,7 +333,6 @@ DbJob.started
 DbJob.succeeded
 DbJob.failed
 ```
-
 ### Resetting jobs
 
 Jobs in a state other than `waiting` are either being processed or else already
@@ -352,7 +364,6 @@ Performing a reset will reset the job state to `waiting` and it will be
 processed again. All meta fields will be reset as well. See inline documentation
 of `Workhorse::DbJob#reset!` for more details.
 
-
 ## Using workhorse with Rails / ActiveJob
 
 While workhorse can be used though its custom interface as documented above, it
@@ -373,6 +384,37 @@ jobs database on a regular interval. Workhorse provides the job
 `Workhose::Jobs::CleanupSucceededJobs` for this purpose that cleans up all
 succeeded jobs. You can run this using your scheduler in a specific interval.
 
+## Caveats
+
+### Errors during polling / crashed workers
+
+Each worker process includes one thread that polls the database for jobs and
+dispatches them to individual worker threads. In case of an error in the poller
+(usually due to a database connection drop), the poller aborts and gracefully
+shuts down the entire worker. Jobs still being processed by this worker are
+attempted to be completed during this shutdown (which only works if the database
+connection is still active).
+
+This means that you should always have an external *watcher* (usually a
+cronjob), that calls the `workhorse watch` command regularly. This would
+automatically restart crashed worker processes.
+
+### Stuck queues
+
+Jobs in named queues (non-null queues) are always run sequentially. This means
+that if a job in such a queue is stuck in states `locked` or `started` (i.e. due
+to a database connection failure), no more jobs of this queue will be run as the
+entire queue is considered locked to ensure that no jobs of the same queue run
+in parallel.
+
+For this purpose, Workhorse provides the built-in job
+`Workhorse::Jobs::DetectStaleJobsJob` which you are advised schedule on a
+regular basis. It picks up jobs that remained `locked` or `started` (running)
+for more than a certain amount of time. If any of these jobs are found, an
+exception is thrown (which may cause a notification if you configured
+`on_exception` accordingly). See the job's API documentation for more
+information.
+
 ## Frequently asked questions
 
 Please consult the [FAQ](FAQ.md).

data/Rakefile

@@ -19,6 +19,7 @@ task :gemspec do
     spec.add_development_dependency 'colorize'
     spec.add_development_dependency 'benchmark-ips'
     spec.add_development_dependency 'activejob'
+    spec.add_development_dependency 'pry'
     spec.add_dependency 'activesupport'
     spec.add_dependency 'activerecord'
     spec.add_dependency 'schemacop', '~> 2.0'

data/VERSION

@@ -1 +1 @@
-0.6.9
+1.0.1

data/lib/workhorse.rb

@@ -1,7 +1,8 @@
-require 'socket'
-require 'active_support/all'
 require 'active_record'
+require 'active_support/all'
 require 'concurrent'
+require 'socket'
+require 'uri'
 
 require 'workhorse/enqueuer'
 require 'workhorse/scoped_env'
@@ -30,6 +31,17 @@ module Workhorse
     # ExceptionNotifier.notify_exception(exception)
   end
 
+  # If set to `true`, the defined `on_exception` will not be called when the
+  # poller encounters an exception and the worker has to be shut down. The
+  # exception will still be logged.
+  mattr_accessor :silence_poller_exceptions
+  self.silence_poller_exceptions = false
+
+  # If set to `true`, the `watch` command won't produce any output. This does
+  # not include warnings such as the "development mode" warning.
+  mattr_accessor :silence_watcher
+  self.silence_watcher = false
+
   mattr_accessor :perform_jobs_in_tx
   self.perform_jobs_in_tx = true
 
@@ -46,6 +58,7 @@ require 'workhorse/worker'
 require 'workhorse/jobs/run_rails_op'
 require 'workhorse/jobs/run_active_job'
 require 'workhorse/jobs/cleanup_succeeded_jobs'
+require 'workhorse/jobs/detect_stale_jobs_job'
 
 # Daemon functionality is not available on java platforms
 if RUBY_PLATFORM != 'java'

data/lib/workhorse/daemon.rb

@@ -38,21 +38,21 @@ module Workhorse
       @workers << Worker.new(@workers.size + 1, name, &block)
     end
 
-    def start
+    def start(quiet: false)
       code = 0
 
       for_each_worker do |worker|
         pid_file, pid = read_pid(worker)
 
         if pid_file && pid
-          warn "Worker ##{worker.id} (#{worker.name}): Already started (PID #{pid})"
+          warn "Worker ##{worker.id} (#{worker.name}): Already started (PID #{pid})" unless quiet
           code = 1
         elsif pid_file
           File.delete pid_file
-          puts "Worker ##{worker.id} (#{worker.name}): Starting (stale pid file)"
+          puts "Worker ##{worker.id} (#{worker.name}): Starting (stale pid file)" unless quiet
           start_worker worker
         else
-          warn "Worker ##{worker.id} (#{worker.name}): Starting"
+          warn "Worker ##{worker.id} (#{worker.name}): Starting" unless quiet
           start_worker worker
         end
       end
@@ -109,7 +109,7 @@ module Workhorse
       end
 
       if should_be_running && status(quiet: true) != 0
-        return start
+        return start(quiet: Workhorse.silence_watcher)
       else
         return 0
       end
@@ -177,7 +177,9 @@ module Workhorse
       file = pid_file_for(worker)
 
       if File.exist?(file)
-        pid = IO.read(file).to_i
+        raw_pid = IO.read(file)
+        return nil, nil if raw_pid.blank?
+        pid = Integer(raw_pid)
         return file, process?(pid) ? pid : nil
       else
         return nil, nil

data/lib/workhorse/db_job.rb

@@ -69,7 +69,14 @@ module Workhorse
         fail "Dirty jobs can't be locked."
       end
 
+      # TODO: Remove this debug output
+      # if Workhorse::DbJob.lock.find(id).locked_at
+      #   puts "Already locked (with FOR UPDATE)"
+      # end
+
       if locked_at
+        # TODO: Remove this debug output
+        # puts "Already locked. Job: #{self.id} Worker: #{worker_id}"
         fail "Job #{id} is already locked by #{locked_by.inspect}."
       end
 

data/lib/workhorse/jobs/cleanup_succeeded_jobs.rb

@@ -1,6 +1,6 @@
 module Workhorse::Jobs
   class CleanupSucceededJobs
-    # Instantiates a new cleanup job
+    # Instantiates a new job.
     #
     # @param max_age [Integer] The maximal age of jobs to retain, in days. Will
     #   be evaluated at perform time.

data/lib/workhorse/jobs/detect_stale_jobs_job.rb

@@ -0,0 +1,48 @@
+module Workhorse::Jobs
+  class DetectStaleJobsJob
+    # Instantiates a new stale detection job.
+    #
+    # @param locked_to_started_threshold [Integer] The maximum number of seconds
+    #   a job is allowed to stay 'locked' before this job throws an exception.
+    #   Set this to 0 to skip this check.
+    # @param run_time_threshold [Integer] The maximum number of seconds
+    #   a job is allowed to run before this job throws an exception. Set this to
+    #   0 to skip this check.
+    def initialize(locked_to_started_threshold: 3 * 60, run_time_threshold: 12 * 60)
+      @locked_to_started_threshold = locked_to_started_threshold
+      @run_time_threshold = run_time_threshold
+    end
+
+    def perform
+      messages = []
+
+      # Detect jobs that are locked for too long #
+      if @locked_to_started_threshold != 0
+        rel = Workhorse::DbJob.locked
+        rel = rel.where('locked_at < ?', @locked_to_started_threshold.seconds.ago)
+        ids = rel.pluck(:id)
+
+        unless ids.empty?
+          messages << "Detected #{ids.size} jobs that were locked more than "\
+                      "#{@locked_to_started_threshold}s ago and might be stale: #{ids.inspect}."
+        end
+      end
+
+      # Detect jobs that are running for too long #
+      if @run_time_threshold != 0
+        rel = Workhorse::DbJob.started
+        rel = rel.where('started_at < ?', @run_time_threshold.seconds.ago)
+        ids = rel.pluck(:id)
+
+        unless ids.empty?
+          messages << "Detected #{ids.size} jobs that are running for longer than "\
+                      "#{@run_time_threshold}s ago and might be stale: #{ids.inspect}."
+        end
+      end
+
+      if messages.any?
+        fail messages.join(' ')
+      end
+    end
+  end
+end

data/lib/workhorse/performer.rb

@@ -2,8 +2,8 @@ module Workhorse
   class Performer
     attr_reader :worker
 
-    def initialize(db_job, worker)
-      @db_job = db_job
+    def initialize(db_job_id, worker)
+      @db_job = Workhorse::DbJob.find(db_job_id)
       @worker = worker
       @started = false
     end

data/lib/workhorse/poller.rb

@@ -1,5 +1,11 @@
 module Workhorse
   class Poller
+    MIN_LOCK_TIMEOUT = 0.1 # In seconds
+    MAX_LOCK_TIMEOUT = 1.0 # In seconds
+
+    ORACLE_LOCK_MODE   = 6           # X_MODE (exclusive)
+    ORACLE_LOCK_HANDLE = 478_564_848 # Randomly chosen number
+
     attr_reader :worker
     attr_reader :table
 
@@ -20,15 +26,20 @@ module Workhorse
       @running = true
 
       @thread = Thread.new do
-        begin
-          loop do
-            break unless running?
+        loop do
+          break unless running?
+
+          begin
             poll
             sleep
+          rescue Exception => e
+            worker.log %(Poll encountered exception:\n#{e.message}\n#{e.backtrace.join("\n")})
+            worker.log 'Worker shutting down...'
+            Workhorse.on_exception.call(e) unless Workhorse.silence_poller_exceptions
+            @running = false
+            worker.instance_variable_get(:@pool).shutdown
+            break
           end
-        rescue Exception => e
-          worker.log %(Poller stopped with exception:\n#{e.message}\n#{e.backtrace.join("\n")})
-          Workhorse.on_exception.call(e)
         end
       end
     end
@@ -61,41 +72,66 @@ module Workhorse
       end
     end
 
+    def with_global_lock(name: :workhorse, timeout: 2, &_block)
+      if @is_oracle
+        result = Workhorse::DbJob.connection.select_all(
+          "SELECT DBMS_LOCK.REQUEST(#{ORACLE_LOCK_HANDLE}, #{ORACLE_LOCK_MODE}, #{timeout}) FROM DUAL"
+        ).first.values.last
+
+        success = result == 0
+      else
+        result = Workhorse::DbJob.connection.select_all(
+          "SELECT GET_LOCK(CONCAT(DATABASE(), '_#{name}'), #{timeout})"
+        ).first.values.last
+        success = result == 1
+      end
+
+      return unless success
+
+      yield
+    ensure
+      if success
+        if @is_oracle
+          Workhorse::DbJob.connection.execute("SELECT DBMS_LOCK.RELEASE(#{ORACLE_LOCK_HANDLE}) FROM DUAL")
+        else
+          Workhorse::DbJob.connection.execute("SELECT RELEASE_LOCK(CONCAT(DATABASE(), '_#{name}'))")
+        end
+      end
+    end
+
     def poll
       @instant_repoll.make_false
 
-      Workhorse.tx_callback.call do
-        # As we are the only thread posting into the worker pool, it is safe to
-        # get the number of idle threads without mutex synchronization. The
-        # actual number of idle workers at time of posting can only be larger
-        # than or equal to the number we get here.
-        idle = worker.idle
-
-        worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
-
-        unless idle.zero?
-          jobs = queued_db_jobs(idle)
-          jobs.each do |job|
-            worker.log "Marking job #{job.id} as locked", :debug
-            job.mark_locked!(worker.id)
-            worker.perform job
+      timeout = [MIN_LOCK_TIMEOUT, [MAX_LOCK_TIMEOUT, worker.polling_interval].min].max
+
+      with_global_lock timeout: timeout do
+        job_ids = []
+
+        Workhorse.tx_callback.call do
+          # As we are the only thread posting into the worker pool, it is safe to
+          # get the number of idle threads without mutex synchronization. The
+          # actual number of idle workers at time of posting can only be larger
+          # than or equal to the number we get here.
+          idle = worker.idle
+
+          worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
+
+          unless idle.zero?
+            jobs = queued_db_jobs(idle)
+            jobs.each do |job|
+              worker.log "Marking job #{job.id} as locked", :debug
+              job.mark_locked!(worker.id)
+              job_ids << job.id
+            end
           end
         end
+
+        job_ids.each { |job_id| worker.perform(job_id) }
       end
     end
 
     # Returns an Array of #{Workhorse::DbJob}s that can be started
     def queued_db_jobs(limit)
-      # ---------------------------------------------------------------
-      # Lock all queued jobs that are waiting
-      # ---------------------------------------------------------------
-      Workhorse::DbJob.connection.execute(
-        Workhorse::DbJob.select('null').where(
-          table[:queue].not_eq(nil)
-          .and(table[:state].eq(:waiting))
-        ).lock.to_sql
-      )
-
       # ---------------------------------------------------------------
       # Select jobs to execute
       # ---------------------------------------------------------------
@@ -147,20 +183,6 @@ module Workhorse
       # Limit number of records
       select = agnostic_limit(select, limit)
 
-      # Wrap the entire query in an other subselect to enable locking under
-      # Oracle SQL. As MySQL is able to lock the records without this additional
-      # complication, only do this when using the Oracle backend.
-      if @is_oracle
-        if AREL_GTE_7
-          select = Arel::SelectManager.new(Arel.sql('(' + select.to_sql + ')'))
-        else
-          select = Arel::SelectManager.new(ActiveRecord::Base, Arel.sql('(' + select.to_sql + ')'))
-        end
-        select = table.project(Arel.star).where(table[:id].in(select.project(:id)))
-      end
-
-      select = select.lock
-
       return Workhorse::DbJob.find_by_sql(select.to_sql).to_a
     end
 
@@ -214,7 +236,7 @@ module Workhorse
                                .where(table[:state].in(bad_states))
       # .distinct is not chainable in older Arel versions
       bad_queues_select.distinct
-      select = select.where(table[:queue].not_in(bad_queues_select))
+      select = select.where(table[:queue].not_in(bad_queues_select).or(table[:queue].eq(nil)))
 
       # Restrict queues to valid ones as indicated by the options given to the
       # worker

data/lib/workhorse/worker.rb

@@ -127,14 +127,14 @@ module Workhorse
       @pool.idle
     end
 
-    def perform(db_job)
+    def perform(db_job_id)
       mutex.synchronize do
         assert_state! :running
-        log "Posting job #{db_job.id} to thread pool"
+        log "Posting job #{db_job_id} to thread pool"
 
         @pool.post do
           begin
-            Workhorse::Performer.new(db_job, self).perform
+            Workhorse::Performer.new(db_job_id, self).perform
           rescue Exception => e
             log %(#{e.message}\n#{e.backtrace.join("\n")}), :error
           end

data/test/lib/test_helper.rb

@@ -1,6 +1,8 @@
 require 'minitest/autorun'
 require 'active_record'
 require 'active_job'
+require 'pry'
+require 'colorize'
 require 'mysql2'
 require 'benchmark'
 require 'jobs'
@@ -40,7 +42,14 @@ class WorkhorseTest < ActiveSupport::TestCase
   end
 end
 
-ActiveRecord::Base.establish_connection adapter: 'mysql2', database: 'workhorse', username: 'travis', password: '', pool: 10, host: :localhost
+ActiveRecord::Base.establish_connection(
+  adapter:  'mysql2',
+  database: ENV['DB_NAME'] || 'workhorse',
+  username: ENV['DB_USERNAME'] || 'root',
+  password: ENV['DB_PASSWORD'] || '',
+  host:     ENV['DB_HOST'] || '127.0.0.1',
+  pool:     10
+)
 
 require 'db_schema'
 require 'workhorse'

data/test/workhorse/poller_test.rb

@@ -48,6 +48,24 @@ class Workhorse::PollerTest < WorkhorseTest
     assert_equal %w[q1 q2], w.poller.send(:valid_queues)
   end
 
+  def test_valid_queues_2
+    w = Workhorse::Worker.new(polling_interval: 60)
+
+    assert_equal [], w.poller.send(:valid_queues)
+
+    Workhorse.enqueue BasicJob.new(sleep_time: 2), queue: nil
+
+    assert_equal [nil], w.poller.send(:valid_queues)
+
+    a_job = Workhorse.enqueue BasicJob.new(sleep_time: 2), queue: :a
+
+    assert_equal [nil, 'a'], w.poller.send(:valid_queues)
+
+    a_job.update_attribute :state, :locked
+
+    assert_equal [nil], w.poller.send(:valid_queues)
+  end
+
   def test_no_queues
     w = Workhorse::Worker.new(polling_interval: 60)
     assert_equal [], w.poller.send(:valid_queues)
@@ -96,6 +114,69 @@ class Workhorse::PollerTest < WorkhorseTest
     assert_equal 1, Workhorse::DbJob.where(state: :succeeded).count
   end
 
+  def test_already_locked_issue
+    # Create 100 jobs
+    100.times do |i|
+      Workhorse.enqueue BasicJob.new(some_param: i, sleep_time: 0)
+    end
+
+    # Create 25 worker processes that work for 10s each
+    25.times do
+      Process.fork do
+        work 10, pool_size: 1, polling_interval: 0.1
+      end
+    end
+
+    # Create additional 100 jobs that are scheduled while the workers are
+    # already polling (to make sure those are picked up as well)
+    100.times do
+      sleep 0.05
+      Workhorse.enqueue BasicJob.new(sleep_time: 0)
+    end
+
+    # Wait for all forked processes to finish (should take ~10s)
+    Process.waitall
+
+    total = Workhorse::DbJob.count
+    succeeded = Workhorse::DbJob.succeeded.count
+    used_workers = Workhorse::DbJob.lock.pluck(:locked_by).uniq.size
+
+    # Make sure there are 200 jobs, all jobs have succeeded and that all of the
+    # workers have had their turn.
+    assert_equal 200, total
+    assert_equal 200, succeeded
+    assert_equal 25,  used_workers
+  end
+
+  # rubocop: disable Style/GlobalVars
+  def test_connection_loss
+    $thread_conn = nil
+
+    Workhorse.enqueue BasicJob.new(sleep_time: 3)
+
+    t = Thread.new do
+      w = Workhorse::Worker.new(pool_size: 5, polling_interval: 0.1)
+      w.start
+
+      sleep 0.5
+
+      w.poller.define_singleton_method :poll do
+        fail ActiveRecord::StatementInvalid, 'Mysql2::Error: Connection was killed'
+      end
+
+      w.wait
+    end
+
+    assert_nothing_raised do
+      Timeout.timeout(6) do
+        t.join
+      end
+    end
+
+    assert_equal 1, Workhorse::DbJob.succeeded.count
+  end
+  # rubocop: enable Style/GlobalVars
+
   private
 
   def setup

data/workhorse.gemspec

@@ -1,15 +1,15 @@
 # -*- encoding: utf-8 -*-
-# stub: workhorse 0.6.9 ruby lib
+# stub: workhorse 1.0.1 ruby lib
 
 Gem::Specification.new do |s|
   s.name = "workhorse".freeze
-  s.version = "0.6.9"
+  s.version = "1.0.1"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version=
   s.require_paths = ["lib".freeze]
   s.authors = ["Sitrox".freeze]
-  s.date = "2020-04-22"
-  s.files = [".gitignore".freeze, ".releaser_config".freeze, ".rubocop.yml".freeze, ".travis.yml".freeze, "CHANGELOG.md".freeze, "FAQ.md".freeze, "Gemfile".freeze, "LICENSE".freeze, "README.md".freeze, "RUBY_VERSION".freeze, "Rakefile".freeze, "VERSION".freeze, "bin/rubocop".freeze, "lib/active_job/queue_adapters/workhorse_adapter.rb".freeze, "lib/generators/workhorse/install_generator.rb".freeze, "lib/generators/workhorse/templates/bin/workhorse.rb".freeze, "lib/generators/workhorse/templates/config/initializers/workhorse.rb".freeze, "lib/generators/workhorse/templates/create_table_jobs.rb".freeze, "lib/workhorse.rb".freeze, "lib/workhorse/daemon.rb".freeze, "lib/workhorse/daemon/shell_handler.rb".freeze, "lib/workhorse/db_job.rb".freeze, "lib/workhorse/enqueuer.rb".freeze, "lib/workhorse/jobs/cleanup_succeeded_jobs.rb".freeze, "lib/workhorse/jobs/run_active_job.rb".freeze, "lib/workhorse/jobs/run_rails_op.rb".freeze, "lib/workhorse/performer.rb".freeze, "lib/workhorse/poller.rb".freeze, "lib/workhorse/pool.rb".freeze, "lib/workhorse/scoped_env.rb".freeze, "lib/workhorse/worker.rb".freeze, "test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze, "workhorse.gemspec".freeze]
+  s.date = "2020-12-15"
+  s.files = [".gitignore".freeze, ".releaser_config".freeze, ".rubocop.yml".freeze, ".travis.yml".freeze, "CHANGELOG.md".freeze, "FAQ.md".freeze, "Gemfile".freeze, "LICENSE".freeze, "README.md".freeze, "RUBY_VERSION".freeze, "Rakefile".freeze, "VERSION".freeze, "bin/rubocop".freeze, "lib/active_job/queue_adapters/workhorse_adapter.rb".freeze, "lib/generators/workhorse/install_generator.rb".freeze, "lib/generators/workhorse/templates/bin/workhorse.rb".freeze, "lib/generators/workhorse/templates/config/initializers/workhorse.rb".freeze, "lib/generators/workhorse/templates/create_table_jobs.rb".freeze, "lib/workhorse.rb".freeze, "lib/workhorse/daemon.rb".freeze, "lib/workhorse/daemon/shell_handler.rb".freeze, "lib/workhorse/db_job.rb".freeze, "lib/workhorse/enqueuer.rb".freeze, "lib/workhorse/jobs/cleanup_succeeded_jobs.rb".freeze, "lib/workhorse/jobs/detect_stale_jobs_job.rb".freeze, "lib/workhorse/jobs/run_active_job.rb".freeze, "lib/workhorse/jobs/run_rails_op.rb".freeze, "lib/workhorse/performer.rb".freeze, "lib/workhorse/poller.rb".freeze, "lib/workhorse/pool.rb".freeze, "lib/workhorse/scoped_env.rb".freeze, "lib/workhorse/worker.rb".freeze, "test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze, "workhorse.gemspec".freeze]
   s.rubygems_version = "3.0.3".freeze
   s.summary = "Multi-threaded job backend with database queuing for ruby.".freeze
   s.test_files = ["test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze]
@@ -26,6 +26,7 @@ Gem::Specification.new do |s|
       s.add_development_dependency(%q<colorize>.freeze, [">= 0"])
       s.add_development_dependency(%q<benchmark-ips>.freeze, [">= 0"])
       s.add_development_dependency(%q<activejob>.freeze, [">= 0"])
+      s.add_development_dependency(%q<pry>.freeze, [">= 0"])
       s.add_runtime_dependency(%q<activesupport>.freeze, [">= 0"])
       s.add_runtime_dependency(%q<activerecord>.freeze, [">= 0"])
       s.add_runtime_dependency(%q<schemacop>.freeze, ["~> 2.0"])
@@ -39,6 +40,7 @@ Gem::Specification.new do |s|
       s.add_dependency(%q<colorize>.freeze, [">= 0"])
       s.add_dependency(%q<benchmark-ips>.freeze, [">= 0"])
       s.add_dependency(%q<activejob>.freeze, [">= 0"])
+      s.add_dependency(%q<pry>.freeze, [">= 0"])
       s.add_dependency(%q<activesupport>.freeze, [">= 0"])
       s.add_dependency(%q<activerecord>.freeze, [">= 0"])
       s.add_dependency(%q<schemacop>.freeze, ["~> 2.0"])
@@ -53,6 +55,7 @@ Gem::Specification.new do |s|
     s.add_dependency(%q<colorize>.freeze, [">= 0"])
     s.add_dependency(%q<benchmark-ips>.freeze, [">= 0"])
     s.add_dependency(%q<activejob>.freeze, [">= 0"])
+    s.add_dependency(%q<pry>.freeze, [">= 0"])
     s.add_dependency(%q<activesupport>.freeze, [">= 0"])
     s.add_dependency(%q<activerecord>.freeze, [">= 0"])
     s.add_dependency(%q<schemacop>.freeze, ["~> 2.0"])

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: workhorse
 version: !ruby/object:Gem::Version
-  version: 0.6.9
+  version: 1.0.1
 platform: ruby
 authors:
 - Sitrox
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-04-22 00:00:00.000000000 Z
+date: 2020-12-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -122,6 +122,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: activesupport
   requirement: !ruby/object:Gem::Requirement
@@ -178,8 +192,8 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: 
-email: 
+description:
+email:
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -208,6 +222,7 @@ files:
 - lib/workhorse/db_job.rb
 - lib/workhorse/enqueuer.rb
 - lib/workhorse/jobs/cleanup_succeeded_jobs.rb
+- lib/workhorse/jobs/detect_stale_jobs_job.rb
 - lib/workhorse/jobs/run_active_job.rb
 - lib/workhorse/jobs/run_rails_op.rb
 - lib/workhorse/performer.rb
@@ -226,10 +241,10 @@ files:
 - test/workhorse/pool_test.rb
 - test/workhorse/worker_test.rb
 - workhorse.gemspec
-homepage: 
+homepage:
 licenses: []
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -244,8 +259,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.2
-signing_key: 
+rubygems_version: 3.1.4
+signing_key:
 specification_version: 4
 summary: Multi-threaded job backend with database queuing for ruby.
 test_files: