workhorse 0.6.9 → 1.0.0.beta0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a328b92854a461c8b0e3a67f740b21581c9652b428286fa917c7942208d5ab83
-  data.tar.gz: 802d62becfdef759b6e1edad34eec6c5c606bf1de5a19ee75347149ccf6118f9
+  metadata.gz: 5dd13b11d5f7488e58812f57ad2bb70da07b95f18b89653687bf9ec579bbcaa5
+  data.tar.gz: a0e7c53cf4de56b6109830decf16f6f70a8aead13ca703a795ce7709234c1b05
 SHA512:
-  metadata.gz: ac93faa479a9f4b228c75b41d7001eadf60a395986160d86342e7423a7e9712c9abeb7e4e69463eafa7d9b36647868ad123ca9efeb776ece3accf5d28d0ec889
-  data.tar.gz: c00ef0fc1c007f7a11ff208664c6c68653cd3cdf00f5750178f56484fb95721f0aaec095b0a4e8e091ecab7700bbbddaeeadbd1b9457c1fdd18882aa224737c7
+  metadata.gz: 2c6e9162d5b2735cd1de678b56e4c4e4960e25776a64cc45dfe5fbfee4fe0e50b1b3a5d15e81ca65ebfe496cce6ac4d81cfe1ba47a65b591a54916807bbe5c68
+  data.tar.gz: a24094b51e7751010920e39ad6ac3924dbccdfb84f1728efec73ed1cc0e9376a873cdb2ff4a562f49d78b17ae08e5659fd596684b299fc35b8bdcd228be15f0e

data/CHANGELOG.md

@@ -1,5 +1,39 @@
 # Workhorse Changelog
 
+## 1.0.0.beta0 - 2020-08-19
+
+This is a stability release that is still experimental and has to be tested in
+battle before it can be considered stable.
+
+* Simplify locking during polling. Other than locking individual jobs, pollers
+  now acquire a global lock. While this can lead to many pollers waiting for
+  each others locks, performing a poll is usually done very quickly and the
+  performance drawback is to be considered neglegible. This change should work
+  around some deadlock issues as well as an issue where a job was obtained by
+  more than one poller.
+
+* Shut down worker if polling encountered any kind of error (running jobs will
+  be completed whenever possible). This leads to potential watcher jobs being
+  able to restore the failed process.
+
+* Make unit test database connection configurable using environment variables
+  `DB_NAME`, `DB_USERNAME`, `DB_PASSWORD` and `DB_HOST`. This is only relevant
+  if you are working on workhorse and need to run the unit tests.
+
+* Fix misbehaviour where queueless jobs were not picked up by workers as long as
+  a named queue was in a locked state.
+
+* Add built-in job `Workhorse::Jobs::DetectStaleJobsJob` which you can schedule.
+  It picks up jobs that remained `locked` or `started` (running) for more than a
+  certain amount of time. If any of these jobs are found, an exception is thrown
+  (which may cause a notification if you configured `on_exception` accordingly).
+  See the job's API documentation for more information.
+
+**If using oracle:** Make sure to grant execute permission to the package
+`DBMS_LOCK` for your oracle database schema:
+
+```GRANT execute ON DBMS_LOCK TO <schema-name>;```
+
 ## 0.6.9 - 2020-04-22
 
 * Fix error where processes may have mistakenly been detected as running (add a

data/FAQ.md

@@ -74,4 +74,8 @@ production mode.
 
 ## Why does workhorse not support timeouts?
 
-Generic timeout implementations are [a dangerous thing](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/) in Ruby. This is why we decided against providing this feature in Workhorse and recommend to implement the timeouts inside of your jobs - i.e. via network timeouts.
+Generic timeout implementations are [a dangerous
+thing](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api/)
+in Ruby. This is why we decided against providing this feature in Workhorse and
+recommend to implement the timeouts inside of your jobs - i.e. via network
+timeouts.

data/README.md

@@ -66,6 +66,15 @@ What it does not do:
 
    Please customize the initializer and worker script to your liking.
 
+### Oracle
+
+When using Oracle databases, make sure your schema has access to the package
+`DBMS_LOCK`:
+
+```
+GRANT execute ON DBMS_LOCK TO <schema-name>;
+```
+
 ## Queuing jobs
 
 ### Basic jobs
@@ -320,7 +329,6 @@ DbJob.started
 DbJob.succeeded
 DbJob.failed
 ```
-
 ### Resetting jobs
 
 Jobs in a state other than `waiting` are either being processed or else already
@@ -352,7 +360,6 @@ Performing a reset will reset the job state to `waiting` and it will be
 processed again. All meta fields will be reset as well. See inline documentation
 of `Workhorse::DbJob#reset!` for more details.
 
-
 ## Using workhorse with Rails / ActiveJob
 
 While workhorse can be used though its custom interface as documented above, it
@@ -373,6 +380,37 @@ jobs database on a regular interval. Workhorse provides the job
 `Workhose::Jobs::CleanupSucceededJobs` for this purpose that cleans up all
 succeeded jobs. You can run this using your scheduler in a specific interval.
 
+## Caveats
+
+### Errors during polling / crashed workers
+
+Each worker process includes one thread that polls the database for jobs and
+dispatches them to individual worker threads. In case of an error in the poller
+(usually due to a database connection drop), the poller aborts and gracefully
+shuts down the entire worker. Jobs still being processed by this worker are
+attempted to be completed during this shutdown (which only works if the database
+connection is still active).
+
+This means that you should always have an external *watcher* (usually a
+cronjob), that calls the `workhorse watch` command regularly. This would
+automatically restart crashed worker processes.
+
+### Stuck queues
+
+Jobs in named queues (non-null queues) are always run sequentially. This means
+that if a job in such a queue is stuck in states `locked` or `started` (i.e. due
+to a database connection failure), no more jobs of this queue will be run as the
+entire queue is considered locked to ensure that no jobs of the same queue run
+in parallel.
+
+For this purpose, Workhorse provides the built-in job
+`Workhorse::Jobs::DetectStaleJobsJob` which you are advised schedule on a
+regular basis. It picks up jobs that remained `locked` or `started` (running)
+for more than a certain amount of time. If any of these jobs are found, an
+exception is thrown (which may cause a notification if you configured
+`on_exception` accordingly). See the job's API documentation for more
+information.
+
 ## Frequently asked questions
 
 Please consult the [FAQ](FAQ.md).

data/Rakefile

@@ -19,6 +19,7 @@ task :gemspec do
     spec.add_development_dependency 'colorize'
     spec.add_development_dependency 'benchmark-ips'
     spec.add_development_dependency 'activejob'
+    spec.add_development_dependency 'pry'
     spec.add_dependency 'activesupport'
     spec.add_dependency 'activerecord'
     spec.add_dependency 'schemacop', '~> 2.0'

data/VERSION

@@ -1 +1 @@
-0.6.9
+1.0.0.beta0

data/lib/workhorse.rb

@@ -46,6 +46,7 @@ require 'workhorse/worker'
 require 'workhorse/jobs/run_rails_op'
 require 'workhorse/jobs/run_active_job'
 require 'workhorse/jobs/cleanup_succeeded_jobs'
+require 'workhorse/jobs/detect_stale_jobs_job'
 
 # Daemon functionality is not available on java platforms
 if RUBY_PLATFORM != 'java'

data/lib/workhorse/db_job.rb

@@ -69,7 +69,14 @@ module Workhorse
         fail "Dirty jobs can't be locked."
       end
 
+      # TODO: Remove this debug output
+      # if Workhorse::DbJob.lock.find(id).locked_at
+      #   puts "Already locked (with FOR UPDATE)"
+      # end
+
       if locked_at
+        # TODO: Remove this debug output
+        # puts "Already locked. Job: #{self.id} Worker: #{worker_id}"
         fail "Job #{id} is already locked by #{locked_by.inspect}."
       end
 

data/lib/workhorse/jobs/cleanup_succeeded_jobs.rb

@@ -1,6 +1,6 @@
 module Workhorse::Jobs
   class CleanupSucceededJobs
-    # Instantiates a new cleanup job
+    # Instantiates a new job.
     #
     # @param max_age [Integer] The maximal age of jobs to retain, in days. Will
     #   be evaluated at perform time.

data/lib/workhorse/jobs/detect_stale_jobs_job.rb

@@ -0,0 +1,48 @@
+module Workhorse::Jobs
+  class DetectStaleJobsJob
+    # Instantiates a new stale detection job.
+    #
+    # @param locked_to_started_threshold [Integer] The maximum number of seconds
+    #   a job is allowed to stay 'locked' before this job throws an exception.
+    #   Set this to 0 to skip this check.
+    # @param run_time_threshold [Integer] The maximum number of seconds
+    #   a job is allowed to run before this job throws an exception. Set this to
+    #   0 to skip this check.
+    def initialize(locked_to_started_threshold: 3 * 60, run_time_threshold: 12 * 60)
+      @locked_to_started_threshold = locked_to_started_threshold
+      @run_time_threshold = run_time_threshold
+    end
+
+    def perform
+      messages = []
+
+      # Detect jobs that are locked for too long #
+      if @locked_to_started_threshold != 0
+        rel = Workhorse::DbJob.locked
+        rel = rel.where('locked_at < ?', @locked_to_started_threshold.seconds.ago)
+        ids = rel.pluck(:id)
+
+        if ids.size > 0
+          messages << "Detected #{ids.size} jobs that were locked more than "\
+                      "#{@locked_to_started_threshold}s ago and might be stale: #{ids.inspect}."
+        end
+      end
+
+      # Detect jobs that are running for too long #
+      if @run_time_threshold != 0
+        rel = Workhorse::DbJob.started
+        rel = rel.where('started_at < ?', @run_time_threshold.seconds.ago)
+        ids = rel.pluck(:id)
+
+        if ids.size > 0
+          messages << "Detected #{ids.size} jobs that are running for longer than "\
+                      "#{@run_time_threshold}s ago and might be stale: #{ids.inspect}."
+        end
+      end
+
+      if messages.any?
+        fail messages.join(' ')
+      end
+    end
+  end
+end

data/lib/workhorse/poller.rb

@@ -1,5 +1,11 @@
 module Workhorse
   class Poller
+    MIN_LOCK_TIMEOUT = 0.1 # In seconds
+    MAX_LOCK_TIMEOUT = 1.0 # In seconds
+
+    ORACLE_LOCK_MODE = 6           # X_MODE (exclusive)
+    ORACLE_LOCK_HANDLE = 478564848 # Randomly chosen number
+
     attr_reader :worker
     attr_reader :table
 
@@ -20,15 +26,20 @@ module Workhorse
       @running = true
 
       @thread = Thread.new do
-        begin
-          loop do
-            break unless running?
+        loop do
+          break unless running?
+
+          begin
             poll
             sleep
+          rescue Exception => e
+            worker.log %(Poll encountered exception:\n#{e.message}\n#{e.backtrace.join("\n")})
+            worker.log 'Worker shutting down...'
+            Workhorse.on_exception.call(e)
+            @running = false
+            worker.instance_variable_get(:@pool).shutdown
+            break
           end
-        rescue Exception => e
-          worker.log %(Poller stopped with exception:\n#{e.message}\n#{e.backtrace.join("\n")})
-          Workhorse.on_exception.call(e)
         end
       end
     end
@@ -61,24 +72,55 @@ module Workhorse
       end
     end
 
+    def with_global_lock(name: :workhorse, timeout: 2, &block)
+      if @is_oracle
+        result = Workhorse::DbJob.connection.select_all(
+          "SELECT DBMS_LOCK.REQUEST(#{ORACLE_LOCK_HANDLE}, #{ORACLE_LOCK_MODE}, #{timeout}) FROM DUAL"
+        ).first.values.last
+
+        success = result == 0
+      else
+        result = Workhorse::DbJob.connection.select_all(
+          "SELECT GET_LOCK(CONCAT(DATABASE(), '_#{name}'), #{timeout})"
+        ).first.values.last
+        success = result == 1
+      end
+
+      return unless success
+
+      yield
+    ensure
+      if success
+        if @is_oracle
+          Workhorse::DbJob.connection.execute("SELECT DBMS_LOCK.RELEASE(#{ORACLE_LOCK_HANDLE}) FROM DUAL")
+        else
+          Workhorse::DbJob.connection.execute("SELECT RELEASE_LOCK(CONCAT(DATABASE(), '_#{name}'))")
+        end
+      end
+    end
+
     def poll
       @instant_repoll.make_false
 
-      Workhorse.tx_callback.call do
-        # As we are the only thread posting into the worker pool, it is safe to
-        # get the number of idle threads without mutex synchronization. The
-        # actual number of idle workers at time of posting can only be larger
-        # than or equal to the number we get here.
-        idle = worker.idle
-
-        worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
-
-        unless idle.zero?
-          jobs = queued_db_jobs(idle)
-          jobs.each do |job|
-            worker.log "Marking job #{job.id} as locked", :debug
-            job.mark_locked!(worker.id)
-            worker.perform job
+      timeout = [MIN_LOCK_TIMEOUT, [MAX_LOCK_TIMEOUT, worker.polling_interval].min].max
+
+      with_global_lock timeout: timeout do
+        Workhorse.tx_callback.call do
+          # As we are the only thread posting into the worker pool, it is safe to
+          # get the number of idle threads without mutex synchronization. The
+          # actual number of idle workers at time of posting can only be larger
+          # than or equal to the number we get here.
+          idle = worker.idle
+
+          worker.log "Polling DB for jobs (#{idle} available threads)...", :debug
+
+          unless idle.zero?
+            jobs = queued_db_jobs(idle)
+            jobs.each do |job|
+              worker.log "Marking job #{job.id} as locked", :debug
+              job.mark_locked!(worker.id)
+              worker.perform job
+            end
           end
         end
       end
@@ -86,16 +128,6 @@ module Workhorse
 
     # Returns an Array of #{Workhorse::DbJob}s that can be started
     def queued_db_jobs(limit)
-      # ---------------------------------------------------------------
-      # Lock all queued jobs that are waiting
-      # ---------------------------------------------------------------
-      Workhorse::DbJob.connection.execute(
-        Workhorse::DbJob.select('null').where(
-          table[:queue].not_eq(nil)
-          .and(table[:state].eq(:waiting))
-        ).lock.to_sql
-      )
-
       # ---------------------------------------------------------------
       # Select jobs to execute
       # ---------------------------------------------------------------
@@ -147,20 +179,6 @@ module Workhorse
       # Limit number of records
       select = agnostic_limit(select, limit)
 
-      # Wrap the entire query in an other subselect to enable locking under
-      # Oracle SQL. As MySQL is able to lock the records without this additional
-      # complication, only do this when using the Oracle backend.
-      if @is_oracle
-        if AREL_GTE_7
-          select = Arel::SelectManager.new(Arel.sql('(' + select.to_sql + ')'))
-        else
-          select = Arel::SelectManager.new(ActiveRecord::Base, Arel.sql('(' + select.to_sql + ')'))
-        end
-        select = table.project(Arel.star).where(table[:id].in(select.project(:id)))
-      end
-
-      select = select.lock
-
       return Workhorse::DbJob.find_by_sql(select.to_sql).to_a
     end
 
@@ -214,7 +232,7 @@ module Workhorse
                                .where(table[:state].in(bad_states))
       # .distinct is not chainable in older Arel versions
       bad_queues_select.distinct
-      select = select.where(table[:queue].not_in(bad_queues_select))
+      select = select.where(table[:queue].not_in(bad_queues_select).or(table[:queue].eq(nil)))
 
       # Restrict queues to valid ones as indicated by the options given to the
       # worker

data/test/lib/test_helper.rb

@@ -1,6 +1,8 @@
 require 'minitest/autorun'
 require 'active_record'
 require 'active_job'
+require 'pry'
+require 'colorize'
 require 'mysql2'
 require 'benchmark'
 require 'jobs'
@@ -40,7 +42,14 @@ class WorkhorseTest < ActiveSupport::TestCase
   end
 end
 
-ActiveRecord::Base.establish_connection adapter: 'mysql2', database: 'workhorse', username: 'travis', password: '', pool: 10, host: :localhost
+ActiveRecord::Base.establish_connection(
+  adapter:  'mysql2',
+  database: ENV['DB_NAME'] || 'workhorse',
+  username: ENV['DB_USERNAME'] || 'root',
+  password: ENV['DB_PASSWORD'] || '',
+  host:     ENV['DB_HOST'] || '127.0.0.1',
+  pool:     10
+)
 
 require 'db_schema'
 require 'workhorse'

data/test/workhorse/poller_test.rb

@@ -48,6 +48,24 @@ class Workhorse::PollerTest < WorkhorseTest
     assert_equal %w[q1 q2], w.poller.send(:valid_queues)
   end
 
+  def test_valid_queues
+    w = Workhorse::Worker.new(polling_interval: 60)
+
+    assert_equal [], w.poller.send(:valid_queues)
+
+    Workhorse.enqueue BasicJob.new(sleep_time: 2), queue: nil
+
+    assert_equal [nil], w.poller.send(:valid_queues)
+
+    a_job = Workhorse.enqueue BasicJob.new(sleep_time: 2), queue: :a
+
+    assert_equal [nil, 'a'], w.poller.send(:valid_queues)
+
+    a_job.update_attribute :state, :locked
+
+    assert_equal [nil], w.poller.send(:valid_queues)
+  end
+
   def test_no_queues
     w = Workhorse::Worker.new(polling_interval: 60)
     assert_equal [], w.poller.send(:valid_queues)
@@ -96,6 +114,67 @@ class Workhorse::PollerTest < WorkhorseTest
     assert_equal 1, Workhorse::DbJob.where(state: :succeeded).count
   end
 
+  def test_already_locked_issue
+    # Create 100 jobs
+    100.times do |i|
+      Workhorse.enqueue BasicJob.new(some_param: i, sleep_time: 0)
+    end
+
+    # Create 25 worker processes that work for 10s each
+    25.times do
+      Process.fork do
+        work 10, pool_size: 1, polling_interval: 0.1
+      end
+    end
+
+    # Create additional 100 jobs that are scheduled while the workers are
+    # already polling (to make sure those are picked up as well)
+    100.times do
+      sleep 0.05
+      Workhorse.enqueue BasicJob.new(sleep_time: 0)
+    end
+
+    # Wait for all forked processes to finish (should take ~10s)
+    Process.waitall
+
+    total = Workhorse::DbJob.count
+    succeeded = Workhorse::DbJob.succeeded.count
+    used_workers = Workhorse::DbJob.lock.pluck(:locked_by).uniq.size
+
+    # Make sure there are 200 jobs, all jobs have succeeded and that all of the
+    # workers have had their turn.
+    assert_equal 200, total
+    assert_equal 200, succeeded
+    assert_equal 25,  used_workers
+  end
+
+  def test_connection_loss
+    $thread_conn = nil
+
+    Workhorse.enqueue BasicJob.new(sleep_time: 3)
+
+    t = Thread.new do
+      w = Workhorse::Worker.new(pool_size: 5, polling_interval: 0.1)
+      w.start
+
+      sleep 0.5
+
+      w.poller.define_singleton_method :poll do
+        fail ActiveRecord::StatementInvalid, 'Mysql2::Error: Connection was killed'
+      end
+
+      w.wait
+    end
+
+    assert_nothing_raised do
+      Timeout.timeout(6) do
+        t.join
+      end
+    end
+
+    assert_equal 1, Workhorse::DbJob.succeeded.count
+  end
+
   private
 
   def setup

data/workhorse.gemspec

@@ -1,15 +1,15 @@
 # -*- encoding: utf-8 -*-
-# stub: workhorse 0.6.9 ruby lib
+# stub: workhorse 1.0.0.beta0 ruby lib
 
 Gem::Specification.new do |s|
   s.name = "workhorse".freeze
-  s.version = "0.6.9"
+  s.version = "1.0.0.beta0"
 
-  s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version=
+  s.required_rubygems_version = Gem::Requirement.new("> 1.3.1".freeze) if s.respond_to? :required_rubygems_version=
   s.require_paths = ["lib".freeze]
   s.authors = ["Sitrox".freeze]
-  s.date = "2020-04-22"
-  s.files = [".gitignore".freeze, ".releaser_config".freeze, ".rubocop.yml".freeze, ".travis.yml".freeze, "CHANGELOG.md".freeze, "FAQ.md".freeze, "Gemfile".freeze, "LICENSE".freeze, "README.md".freeze, "RUBY_VERSION".freeze, "Rakefile".freeze, "VERSION".freeze, "bin/rubocop".freeze, "lib/active_job/queue_adapters/workhorse_adapter.rb".freeze, "lib/generators/workhorse/install_generator.rb".freeze, "lib/generators/workhorse/templates/bin/workhorse.rb".freeze, "lib/generators/workhorse/templates/config/initializers/workhorse.rb".freeze, "lib/generators/workhorse/templates/create_table_jobs.rb".freeze, "lib/workhorse.rb".freeze, "lib/workhorse/daemon.rb".freeze, "lib/workhorse/daemon/shell_handler.rb".freeze, "lib/workhorse/db_job.rb".freeze, "lib/workhorse/enqueuer.rb".freeze, "lib/workhorse/jobs/cleanup_succeeded_jobs.rb".freeze, "lib/workhorse/jobs/run_active_job.rb".freeze, "lib/workhorse/jobs/run_rails_op.rb".freeze, "lib/workhorse/performer.rb".freeze, "lib/workhorse/poller.rb".freeze, "lib/workhorse/pool.rb".freeze, "lib/workhorse/scoped_env.rb".freeze, "lib/workhorse/worker.rb".freeze, "test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze, "workhorse.gemspec".freeze]
+  s.date = "2020-08-19"
+  s.files = [".gitignore".freeze, ".releaser_config".freeze, ".rubocop.yml".freeze, ".travis.yml".freeze, "CHANGELOG.md".freeze, "FAQ.md".freeze, "Gemfile".freeze, "LICENSE".freeze, "README.md".freeze, "RUBY_VERSION".freeze, "Rakefile".freeze, "VERSION".freeze, "bin/rubocop".freeze, "lib/active_job/queue_adapters/workhorse_adapter.rb".freeze, "lib/generators/workhorse/install_generator.rb".freeze, "lib/generators/workhorse/templates/bin/workhorse.rb".freeze, "lib/generators/workhorse/templates/config/initializers/workhorse.rb".freeze, "lib/generators/workhorse/templates/create_table_jobs.rb".freeze, "lib/workhorse.rb".freeze, "lib/workhorse/daemon.rb".freeze, "lib/workhorse/daemon/shell_handler.rb".freeze, "lib/workhorse/db_job.rb".freeze, "lib/workhorse/enqueuer.rb".freeze, "lib/workhorse/jobs/cleanup_succeeded_jobs.rb".freeze, "lib/workhorse/jobs/detect_stale_jobs_job.rb".freeze, "lib/workhorse/jobs/run_active_job.rb".freeze, "lib/workhorse/jobs/run_rails_op.rb".freeze, "lib/workhorse/performer.rb".freeze, "lib/workhorse/poller.rb".freeze, "lib/workhorse/pool.rb".freeze, "lib/workhorse/scoped_env.rb".freeze, "lib/workhorse/worker.rb".freeze, "test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze, "workhorse.gemspec".freeze]
   s.rubygems_version = "3.0.3".freeze
   s.summary = "Multi-threaded job backend with database queuing for ruby.".freeze
   s.test_files = ["test/active_job/queue_adapters/workhorse_adapter_test.rb".freeze, "test/lib/db_schema.rb".freeze, "test/lib/jobs.rb".freeze, "test/lib/test_helper.rb".freeze, "test/workhorse/db_job_test.rb".freeze, "test/workhorse/enqueuer_test.rb".freeze, "test/workhorse/performer_test.rb".freeze, "test/workhorse/poller_test.rb".freeze, "test/workhorse/pool_test.rb".freeze, "test/workhorse/worker_test.rb".freeze]
@@ -26,6 +26,7 @@ Gem::Specification.new do |s|
       s.add_development_dependency(%q<colorize>.freeze, [">= 0"])
       s.add_development_dependency(%q<benchmark-ips>.freeze, [">= 0"])
       s.add_development_dependency(%q<activejob>.freeze, [">= 0"])
+      s.add_development_dependency(%q<pry>.freeze, [">= 0"])
       s.add_runtime_dependency(%q<activesupport>.freeze, [">= 0"])
       s.add_runtime_dependency(%q<activerecord>.freeze, [">= 0"])
       s.add_runtime_dependency(%q<schemacop>.freeze, ["~> 2.0"])
@@ -39,6 +40,7 @@ Gem::Specification.new do |s|
       s.add_dependency(%q<colorize>.freeze, [">= 0"])
       s.add_dependency(%q<benchmark-ips>.freeze, [">= 0"])
       s.add_dependency(%q<activejob>.freeze, [">= 0"])
+      s.add_dependency(%q<pry>.freeze, [">= 0"])
       s.add_dependency(%q<activesupport>.freeze, [">= 0"])
       s.add_dependency(%q<activerecord>.freeze, [">= 0"])
       s.add_dependency(%q<schemacop>.freeze, ["~> 2.0"])
@@ -53,6 +55,7 @@ Gem::Specification.new do |s|
     s.add_dependency(%q<colorize>.freeze, [">= 0"])
     s.add_dependency(%q<benchmark-ips>.freeze, [">= 0"])
     s.add_dependency(%q<activejob>.freeze, [">= 0"])
+    s.add_dependency(%q<pry>.freeze, [">= 0"])
     s.add_dependency(%q<activesupport>.freeze, [">= 0"])
     s.add_dependency(%q<activerecord>.freeze, [">= 0"])
     s.add_dependency(%q<schemacop>.freeze, ["~> 2.0"])

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: workhorse
 version: !ruby/object:Gem::Version
-  version: 0.6.9
+  version: 1.0.0.beta0
 platform: ruby
 authors:
 - Sitrox
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-04-22 00:00:00.000000000 Z
+date: 2020-08-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -122,6 +122,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: activesupport
   requirement: !ruby/object:Gem::Requirement
@@ -208,6 +222,7 @@ files:
 - lib/workhorse/db_job.rb
 - lib/workhorse/enqueuer.rb
 - lib/workhorse/jobs/cleanup_succeeded_jobs.rb
+- lib/workhorse/jobs/detect_stale_jobs_job.rb
 - lib/workhorse/jobs/run_active_job.rb
 - lib/workhorse/jobs/run_rails_op.rb
 - lib/workhorse/performer.rb
@@ -240,9 +255,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubygems_version: 3.1.2
 signing_key: