backburner 1.8.0 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 50e2f0b2644659644c0789a09a93c786e1bf7ab914bdadc40e80be8cdf941fb7
-  data.tar.gz: e86d98085d8fbeb91079d9d52095d2ba62f9c4ec114e28053c8d457398003b35
+  metadata.gz: ad41e83594bda50d1b504a4e0e5777c477e5586dc88faddbbad476a81afb34d4
+  data.tar.gz: f53ac6873638f40f627bbcd1326f50fbd642e5454bce2c083b809cdc6904863b
 SHA512:
-  metadata.gz: f707e632bc937197959a4e919b8f12b010a201b7e18add316c3cb66bf1598219a1089c480f7012f9d888debc63fc1d0c4eb0be85b4a943ac3220dce2453c22e6
-  data.tar.gz: c79112e436fd4bff2b68cc417de9fe23b782b433882697bae60b937f8c36ae76177e056a3a86cff1ab714da48a59d5a548e8113469382a387967295dc697aee1
+  metadata.gz: '0384db95f53944823490c2105948153c8aac2d0fb5e8268c8c6f82895ae722bf293bb93f7a0536967a9ad91a138d37f02d92f4f2b0e6860a749cb830065dd668'
+  data.tar.gz: 1e6cbde19cd41a7540346d5d66af0537af1fc53baec32466790217b0643b610cbd7d17f8ea74b1ed6aedd55a26c152d798df86f4ef6b21ac11b5bc848db9d175

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # CHANGELOG
 
+## Version 1.9.0 (Nov 1 2026)
+
+* The gem now ships only `lib/`, `bin/`, `README.md`, `HOOKS.md`, `CHANGELOG.md` and `LICENSE` (no tests, examples or CI files), and its rubygems.org page links the source, the changelog and the issue tracker. **Behaviour change for the gem's owners:** pushing a release to rubygems.org now requires MFA (`rubygems_mfa_required`); the release workflow's trusted publishing is unaffected. **What to do:** owners who push by hand need MFA enabled on their rubygems.org account
+* Docs: the README's Simple Async Jobs example says a `Backburner::Performable` class's queue defaults to the primary queue (`backburner-jobs`), not one named after the class, and its Persistence examples show the JSON jobs are stored as (`null`, double quotes) on the class's queue
+* Docs: the README says Backburner uses only the first of several `beanstalk_url`s (no client-side federation), and that the Threading worker's `queue:threads` CLI format needs `config.default_worker = Backburner::Workers::Threading`; `examples/god.rb` sets `QUEUE` (which the rake tasks read) instead of `QUEUES`; `examples/custom.rb` enqueues on the queue it works; CONTRIBUTING says the tests need beanstalkd on 127.0.0.1:11300; `backburner --help` shows `-q, --queues QUEUES`; dead links point to beanstalkd.github.io, rubygems.org and rubydoc.info; the 1.5.0 and 1.6.0 changelog entries are filled in from their commits, with their release dates
+* Docs: the README has an Active Job section: setting the adapter, how queues map to tubes, starting workers with `-q` for the Active Job queues (a worker started without `-q` doesn't pick up a queue first used after it started), the TTR, how `retry_on` and `max_job_retries` add up, and using Active Job callbacks rather than Backburner's hooks
+* Active Job: a job's TTR is now its class's `queue_respond_timeout` when it defines one (`def self.queue_respond_timeout; 3600; end`; subclasses inherit it), as for a `Backburner::Queue` job; before, every Active Job job got `config.respond_timeout` (120 seconds by default), so a job that ran longer was stopped with `Backburner::Job::JobTimeout` and retried or buried. Jobs of classes without one still get `config.respond_timeout`. The TTR is set when a job is enqueued, so jobs already queued, or enqueued by an app still on an older backburner, keep the TTR they had. **What to do:** give long-running Active Job classes a `queue_respond_timeout` longer than they take
+* Rails apps: a worker now runs each job inside the app's reloader (`Rails.application.reloader.wrap`), as Rails does for its own Active Job jobs and as Sidekiq does. **Behaviour change:** `ActiveSupport::CurrentAttributes` (a `Current.user` or tenant a job sets) no longer leak into the next job the worker runs; the database connections a job uses go back to the pool when it finishes; Active Record's query cache is on while a job runs, as it is for requests and Active Job jobs; and in development a job runs the app's latest code after a file changes (a reload waits for the jobs running in other threads to finish). On Rails 7.0 and newer, an error a job raises is now also reported to `Rails.error`, as an Active Job job's error already was (including a job whose class this worker doesn't have, each time it's released), but not a `Backburner::Job::RetryJob`, nor the `SystemExit` of a worker stopped by `TERM` or `INT` (or of a job that calls `exit`): an error tracker subscribed to `Rails.error` that `config.on_error` also calls may get it twice. An Active Job job runs in the worker's wrap, so Active Job's own wrap inside it does nothing: it's still wrapped once, but on Rails 7.1 and newer its errors, like other jobs', are now reported with the source `"job.backburner"` instead of `"application.active_support"` (a `Rails.error` subscriber that filters on the source sees the new one). `config.on_error` and the `on_retry` and `on_bury` hooks run after the job's wrap ends, so they no longer see the `CurrentAttributes` the job set, and a database connection they use is taken outside the executor (the `on_failure` hooks still run inside it). Outside Rails nothing changes. **What to do:** nothing, unless a job relies on state an earlier job left in `CurrentAttributes` (pass it as an argument instead), a job waits for a change another process makes to the database (read it inside `ActiveRecord::Base.uncached { ... }`, as the query cache would return the first result again), or an `on_error` handler or retry or bury hook reads the job's `CurrentAttributes` (pass what it needs in the job's arguments, or read it in an `on_failure` hook)
+* Fix "Worker exiting..." often not being logged when a worker stops on `TERM` or `INT`: the process could exit before the thread that logs it ran. The worker now waits up to a second for that line to be written before it exits. **What to do:** nothing
+* `Backburner::Performable` jobs (enqueued with `async`, `handle_asynchronously` or `handle_static_asynchronously`) now call only methods the app defines on the class, its superclasses (such as `ActiveRecord::Base`) or the modules they include, private ones too. **Behaviour change:** a job naming a method the class only gets from Ruby itself (`Object`, `Kernel` and `BasicObject`, or for a class method `Class` and `Module`, including what libraries add to them, such as `to_s`, `frozen?`, `name` or `try`), one of Backburner's own (`queue`, `perform`), or `method_missing` now fails with `NoMethodError`, and is retried and then buried like any other failure. A class that overrides such a method (`def to_s`) still has its own called. **What to do:** if your app enqueues such a method (`User.async.name`), define a method on the class that calls it and enqueue that instead. Jobs already queued keep the method name they were enqueued with, so they fail the same way once the workers are upgraded: to run them, define that same method on the class (`def self.name; super; end`) and kick them from the buried list, or delete them and enqueue the new method
+* Fix the Threading worker taking `shutdown_timeout` (10 seconds by default) to exit on `TERM` or `INT` when it had no jobs to work: its threads waited in `reserve` until the shutdown gave up on them and killed them. Each thread now waits for a job at most a second at a time (or `config.reserve_timeout`, if shorter; a negative one, which made beanstalkd wait forever, counts as a second), so an idle worker exits within about a second. A busy worker is unchanged: its in-flight jobs get up to `shutdown_timeout` to finish before their threads are killed. A job a thread reserves as the shutdown begins is worked like any in-flight job; one still running when its thread is killed is not lost: beanstalkd puts it back in the queue when the worker exits, and it runs again. **What to do:** nothing; an idle Threading worker now sends beanstalkd a `reserve-with-timeout` once a second per thread
+
 ## Version 1.8.0 (Oct 31 2026)
 
 * Fix ThreadsOnFork and Forking children crashing on Ruby 3.3 with `[BUG] lazy sweeping underway when freeing object space` when they stopped on a `TERM` or `INT`, on an error, or when a job called `exit`, if a thread in the parent was running finalizers when the child was forked (an interpreter bug fixed in Ruby 3.4). On Ruby 3.3 a forked child now disables the GC as it exits, so its `at_exit` handlers (the app's, and those it inherited from the parent) run with the GC off, and a child running several threads stops its job threads before those handlers run (a job they were working on goes back to the queue and runs again). Other Rubies are unchanged. **What to do:** nothing; on Ruby 3.3 an `at_exit` handler that allocates a lot in a child now grows the child's memory until it exits
@@ -37,13 +49,21 @@
 
 * Change `File.exists?`` to `File.exist?` as latest ruby versions have removed support (@ShrutiSaagar).
 
-## Version 1.6.0 (December 30 2021)
+## Version 1.6.0 (Dec 30 2021)
 
-* TBD (please help backfill)
+* Allow per-job `queue_retry_delay`, `queue_retry_delay_proc` and `queue_max_job_retries` (@boncey in #162)
+* Fix the beanstalkd links in the README (@a112121788 in #171)
 
-## Version 1.5.0 (September 10 2018)
+## Version 1.5.0 (Sep 19 2018)
 
-* TBD
+* Add `Backburner::Job::RetryJob`: raise it from a job to retry it without logging an error (@eltone in #159)
+* Threading worker: shut down gracefully on `TERM` or `INT`, letting running jobs finish; a second signal stops it at once (@eltone in #157)
+* Make the logger's job timings thread-safe (@eltone in #157)
+* Log, retry and bury jobs whose class is unknown instead of raising while handling their failure (@eltone in #154)
+* Fix Ruby warnings about a shadowed and an unused variable (@utilum in #153)
+* Allow concurrent-ruby 1.x after 1.0.1 (`~> 1.0`, `>= 1.0.1`)
+* Docs: fix the link to Adam Wiggins' beanstalkd comparison (@dplummer in #152); add a CI badge (@szTheory in #151)
+* CI: test against Ruby 2.x (@nicolasleger in #150)
 
 ## Version 1.4.1 (June 10 2017)
 

data/README.md

@@ -57,7 +57,7 @@ Beanstalkd supports the following features out of the box:
 | **Fast**            | Processes thousands of jobs per second without breaking a sweat. |
 | **Priorities**      | Specify priority so important jobs can be processed quickly. |
 | **Persistence**     | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
-| **Federation**      | Horizontal scalability provided through federation by the client. |
+| **Federation**      | Horizontal scalability provided through federation by the client (Backburner connects to a single beanstalkd: given several `beanstalk_url`s, it uses the first). |
 | **Error Handling**  | Bury any job which causes an error for later debugging and inspection.|
 
 Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.
@@ -70,6 +70,9 @@ Depending on your platform, this should be as simple as (for Ubuntu):
 
     $ sudo apt-get install beanstalkd
 
+beanstalkd has no authentication: anyone who can connect to it can add jobs, so make it reachable only
+from your app's own hosts (bind it to `127.0.0.1` or a private network, or firewall its port).
+
 Add this line to your application's Gemfile:
 
     gem 'backburner'
@@ -238,7 +241,7 @@ includes `Backburner::Performable`. Async enqueuing works for both instance and
 ```ruby
 class User
   include Backburner::Performable
-  queue "user-jobs"  # defaults to 'user'
+  queue "user-jobs"  # defaults to the primary queue, 'backburner-jobs'
   queue_priority 500 # most urgent priority is 0
   queue_respond_timeout 300 # number of seconds before job times out (minimum 1; 0 does not disable it)
 
@@ -263,6 +266,10 @@ This automatically enqueues a job for that user record that will run `activate`
 Note that you can set the queue name and queue priority at the class level and
 you are also able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`.
 
+The worker calls only methods your app defines on the class, its superclasses or the modules they include (private
+ones too). A job naming a method every object or class has (`frozen?`, `instance_variable_get`, `name`, ...), or one
+of Backburner's own (`queue`, `perform`), fails with `NoMethodError`.
+
 The queue name used by default is `{namespace}.backburner-jobs` if not otherwise
 specified.
 
@@ -372,12 +379,12 @@ example from above. We'll run the following code to create a job:
 User.async.reset_password(@user.id)
 ```
 
-The following JSON will be put on the `{namespace}.backburner-jobs` queue:
+The following JSON will be put on the `{namespace}.user-jobs` queue (the class's queue; `{namespace}.backburner-jobs` for a class without one):
 
 ``` javascript
 {
-    'class': 'User',
-    'args': [nil, 'reset_password', 123]
+    "class": "User",
+    "args": [null, "reset_password", 123]
 }
 ```
 
@@ -393,8 +400,8 @@ would be stored as:
 
 ``` javascript
 {
-    'class': 'User',
-    'args': [246, 'activate', 987]
+    "class": "User",
+    "args": [246, "activate", 987]
 }
 ```
 
@@ -474,7 +481,7 @@ For more information on the threads_on_fork worker, check out the
 
 #### More info: Threading Worker (thread-pool-based)
 
-Configuration options for the Threading worker are similar to the threads_on_fork worker, sans the garbage option. When running via the `backburner` CLI, it's simplest to provide the queue names and maximum number of threads in the format "{queue name}:{max threads in pool}[,{name}:{threads}]":
+Configuration options for the Threading worker are similar to the threads_on_fork worker, sans the garbage option. The `backburner` CLI runs the default worker, so set `config.default_worker = Backburner::Workers::Threading` in the app it loads; then it's simplest to provide the queue names and maximum number of threads in the format "{queue name}:{max threads in pool}[,{name}:{threads}]" (the Simple and Forking workers ignore the thread counts):
 
 ```
 $ bundle exec backburner -q queue1:4,queue2:4  # and then other options, like environment, pidfile, app root, etc. See docs for the CLI
@@ -573,9 +580,58 @@ Often using hooks is much easier then trying to monkey patch the externals.
 
 Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md) for a detailed overview on using hooks.
 
+### Active Job
+
+Backburner is an [Active Job](https://guides.rubyonrails.org/active_job_basics.html) adapter. Set it in
+`config/application.rb` (or an environment file):
+
+```ruby
+config.active_job.queue_adapter = :backburner
+```
+
+An Active Job queue is a tube named like any other queue: the `tube_namespace`, the separator and the queue name,
+dasherized. With the namespace `myapp.production`, `queue_as :low_priority` uses the tube `myapp.production.low-priority`;
+a `queue_name_prefix` is part of the queue name (`production_mailers` uses `myapp.production.production-mailers`).
+A job's priority (`queue_with_priority`, `set(priority: 10)`) is its beanstalkd priority (lower runs first), and
+`set(wait: ...)` or `set(wait_until: ...)` delays it (rounded up to whole seconds).
+
+Start workers with `-q` listing the Active Job queues, named as Active Job names them (with any prefix):
+
+```
+bundle exec backburner -q default,mailers,low_priority
+```
+
+A worker started without `-q` only watches the tubes that are on the server when it starts (and those of the
+`Backburner::Queue` classes it has loaded, and the primary queue), so the jobs of a queue first used after it started
+aren't worked until it restarts.
+
+A job's time to run (TTR) is its class's `queue_respond_timeout` when it defines one, otherwise `config.respond_timeout`
+(120 seconds by default). A job still running when its TTR is up is stopped with `Backburner::Job::JobTimeout`:
+
+```ruby
+class ReportJob < ApplicationJob
+  def self.queue_respond_timeout
+    3600 # seconds; subclasses inherit it
+  end
+end
+```
+
+Active Job's `retry_on` and Backburner's `max_job_retries` add up: once `retry_on` gives up and the error reaches
+Backburner, it retries the job `max_job_retries` more times before burying it, calling `on_error` each time. Keep
+`max_job_retries` at 0 (the default) for Active Job jobs and retry with `retry_on`, or expect both.
+
+Backburner's [hooks](#hooks) don't run for Active Job jobs (they are all enqueued as the adapter's wrapper class):
+use Active Job's callbacks (`before_perform`, `around_perform`, `after_enqueue` and so on) instead. `config.on_error`
+is still called when a job raises.
+
+In a Rails app, a worker runs each job (Active Job or not) in the app's reloader, as Rails does: `CurrentAttributes` are
+reset between jobs, a job's database connections go back to the pool when it's done, Active Record's query cache is on
+while a job runs (a job that waits for a change another process makes reads it in `ActiveRecord::Base.uncached { ... }`),
+and in development jobs run the latest code (a reload waits for the jobs running in other threads).
+
 ### Workers in Production
 
-Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html)
+Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](https://beanstalkd.github.io/download.html)
 is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile:
 
 ```ruby
@@ -674,8 +730,8 @@ Thanks to these projects for inspiration and certain design and implementation d
 
 ## Links
 
- * Code: `git clone git://github.com/nesquena/backburner.git`
- * Home: <http://github.com/nesquena/backburner>
- * Docs: <http://rdoc.info/github/nesquena/backburner/master/frames>
- * Bugs: <http://github.com/nesquena/backburner/issues>
- * Gems: <http://gemcutter.org/gems/backburner>
+ * Code: `git clone https://github.com/nesquena/backburner.git`
+ * Home: <https://github.com/nesquena/backburner>
+ * Docs: <https://www.rubydoc.info/gems/backburner>
+ * Bugs: <https://github.com/nesquena/backburner/issues>
+ * Gems: <https://rubygems.org/gems/backburner>

data/lib/active_job/queue_adapters/backburner_adapter.rb

@@ -15,16 +15,23 @@ module ActiveJob
     # To use Backburner set the queue_adapter config to +:backburner+.
     #
     #   Rails.application.config.active_job.queue_adapter = :backburner
+    #
+    # A job's TTR is its class's +queue_respond_timeout+ when it has one, as for a Backburner::Queue job
+    # (+ttr: job.class+ resolves it), else +config.respond_timeout+:
+    #
+    #   class ReportJob < ApplicationJob
+    #     def self.queue_respond_timeout; 3600; end
+    #   end
     class BackburnerAdapter < (rails_7_2_or_newer ? ::ActiveJob::QueueAdapters::AbstractAdapter : Object)
       def enqueue(job)
-        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority)
+        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority, ttr: job.class)
         job.provider_job_id = response[:id] if response.is_a?(Hash)
         response
       end
 
       def enqueue_at(job, timestamp)
         delay = timestamp - Time.current.to_f
-        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority, delay: delay)
+        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority, delay: delay, ttr: job.class)
         job.provider_job_id = response[:id] if response.is_a?(Hash)
         response
       end

data/lib/backburner/cli.rb

@@ -10,7 +10,7 @@ module Backburner
         opts.on("-r", "--require PATH", String, "The path to load as the environment.") do |req|
           options[:require] = req
         end
-        opts.on("-q", "--queues PATH", String, "The specific queues to work.") do |queues|
+        opts.on("-q", "--queues QUEUES", String, "The specific queues to work (comma-separated).") do |queues|
           options[:queues] = queues
         end
         opts.on("-e", "--environment ENVIRONMENT", String, "The environment to run Backburner within") do |environment|

data/lib/backburner/enqueue_pool.rb

@@ -9,6 +9,8 @@ module Backburner
   class EnqueuePool
     MAX_IDLE_CONNECTIONS = 5
 
+    @current = nil
+
     # The pool of this process: a forked child gets a new one and leaves its parent's
     # sockets (and a lock another thread may have held at the fork) alone.
     # ceiling: no lock, so threads that race on the first enqueue (or the first after a fork) may

data/lib/backburner/performable.rb

@@ -28,14 +28,16 @@ module Backburner
         Backburner::AsyncProxy.new(self, nil, opts)
       end
 
-      # Defines perform method for job processing
+      # Defines perform method for job processing. It calls a method the app defines on the class,
+      # its superclasses or the modules they include, and raises NoMethodError for any other.
       # @example
       #   perform(55, :do_something, "foo", "bar")
       def perform(id, method, *args)
         if id # instance
-          find(id).send(method, *args)
+          record = find(id)
+          Backburner::Performable.perform_job_method(record, record.class, Object, method, args)
         else # class method
-          send(method, *args)
+          Backburner::Performable.perform_job_method(self, singleton_class, Object.singleton_class, method, args)
         end
       end # perform
 
@@ -90,6 +92,29 @@ module Backburner
     end
     private_class_method :_handle_asynchronously
 
+    # Calls a job's method on `receiver`. It raises NoMethodError when the method isn't one the app
+    # defined: one that `methods_of` (the receiver's class, or the singleton class for a class method)
+    # gets from `core` (Object, or Object's singleton class for a class method) or the modules mixed
+    # into it, from Backburner's own modules, or Ruby's method_missing hook. A method with the same name
+    # as one of core's must be the app's own definition: one that method_missing would handle, or one
+    # from Delegator's copy of Kernel (the module delegate.rb aliases __raise__ in), is refused too.
+    def self.perform_job_method(receiver, methods_of, core, method, args)
+      owner = begin
+        methods_of.instance_method(method).owner
+      rescue NameError
+        nil
+      end
+      refused = if owner && (core.ancestors.include?(owner) || [InstanceMethods, ClassMethods, Backburner::Queue::ClassMethods].include?(owner))
+        true
+      elsif core.method_defined?(method) || core.private_method_defined?(method)
+        owner.nil? || method.to_sym == :method_missing || (defined?(::Delegator) && owner.equal?(::Delegator.instance_method(:__raise__).owner))
+      end
+      if refused
+        kind = methods_of.singleton_class? ? "class method" : "method"
+        raise NoMethodError.new("undefined job #{kind} '#{method}' for #{methods_of.singleton_class? ? receiver : methods_of}", method.to_sym, receiver: receiver)
+      end
+      receiver.__send__(method, *args)
+    end
 
   end # Performable
 end # Backburner

data/lib/backburner/version.rb

@@ -1,3 +1,3 @@
 module Backburner
-  VERSION = "1.8.0"
+  VERSION = "1.9.0"
 end

data/lib/backburner/worker.rb

@@ -73,6 +73,7 @@ module Backburner
     # @example
     #   Worker.new(['test.job'])
     def initialize(tube_names=nil)
+      @existing_tube_names = nil # set by all_existing_queues, read by expand_tube_name
       @connection = new_connection
       @tube_names = self.process_tube_names(tube_names)
       register_signal_handlers!
@@ -104,9 +105,12 @@ module Backburner
 
     # Triggers this worker to shutdown
     def shutdown
-      Thread.new do
-        log_info 'Worker exiting...'
-      end
+      # Logs from another thread, since a trap handler can't take a logger's lock, and waits for it (a
+      # second at most): the process exits right after, and an unjoined thread never got to run
+      # ceiling: if the signal lands while the main thread holds the logger's lock, the line is lost after a
+      # second; handling the signal outside the trap (the self-pipe Threading uses) would lift it
+      Thread.new { log_info 'Worker exiting...' }.join(1)
+    ensure
       Kernel.exit
     end
 
@@ -136,7 +140,7 @@ module Backburner
       end
 
       self.log_job_begin(job.name, job.args)
-      job.process
+      wrap_in_rails_reloader { job.process }
       self.log_job_end(job.name)
 
     rescue Backburner::Job::JobFormatInvalid => e
@@ -203,6 +207,36 @@ module Backburner
       Connection.new(Backburner.configuration.beanstalk_url) { |conn| Backburner::Hooks.invoke_hook_events(self, :on_reconnect, conn) }
     end
 
+    # Runs the block (a job's processing) in the Rails app's reloader when there is one, as Rails does for Active Job
+    # jobs (whose own wrap, inside this one, then does nothing): each job starts with fresh CurrentAttributes, gives
+    # back its database connections when it's done and, in development, runs the app's latest code. On Rails 7.0+ the
+    # wrap reports the errors that leave it to Rails.error (with the source "job.backburner" on 7.1+); a RetryJob, a
+    # request to retry rather than a failure, and a SystemExit (a TERM or INT the worker stops on, or a job calling
+    # exit), which Rails 7.2+ would report too, are raised again outside it instead.
+    # ceiling: config.on_error and the on_retry and on_bury hooks run after the wrap ends, so they don't see the job's
+    # CurrentAttributes, and a database connection they use is outside the executor; running work_one_job's failure
+    # handling inside the wrap would lift it. In development, a Forking or ThreadsOnFork parent never runs the wrap, so
+    # after a file change each child reloads the app once.
+    def wrap_in_rails_reloader
+      app = ::Rails.application if defined?(::Rails) && ::Rails.respond_to?(:application)
+      return yield unless app.respond_to?(:reloader)
+      deferred = nil
+      work = proc do
+        begin
+          yield
+        rescue Backburner::Job::RetryJob, SystemExit => e
+          deferred = e
+        end
+      end
+      # Active Support 7.1+ take the source the executor reports errors with; older ones take no options
+      if ActiveSupport.gem_version >= Gem::Version.new("7.1")
+        app.reloader.wrap(source: "job.backburner", &work)
+      else
+        app.reloader.wrap(&work)
+      end
+      raise deferred if deferred
+    end
+
     # Reserve a job from the watched queues
     def reserve_job(conn, reserve_timeout = Backburner.configuration.reserve_timeout)
       Backburner::Job.new(conn.tubes.reserve(reserve_timeout))

data/lib/backburner/workers/threading.rb

@@ -7,6 +7,9 @@ module Backburner
 
       @shutdown_timeout = 10
 
+      # The longest a pool thread waits in reserve at a time (beanstalkd takes whole seconds)
+      SHUTDOWN_CHECK_INTERVAL = 1
+
       class << self
         attr_accessor :threads_number
         attr_accessor :shutdown_timeout
@@ -15,6 +18,7 @@ module Backburner
       # Custom initializer just to set @tubes_data
       def initialize(*args)
         @tubes_data = {}
+        @in_shutdown = false
         super
         self.process_tube_options
         @exit_on_shutdown = true
@@ -152,6 +156,18 @@ module Backburner
           end
         end
       end
+
+      protected
+
+      # Reserves for at most SHUTDOWN_CHECK_INTERVAL seconds, so an idle thread sees a shutdown
+      # and stops instead of waiting in reserve until shutdown_timeout kills it
+      # ceiling: each idle thread sends beanstalkd a reserve once a second; waking the waiting
+      # reserves at shutdown instead would avoid that polling
+      def reserve_job(conn, reserve_timeout = Backburner.configuration.reserve_timeout)
+        timeout = reserve_timeout.to_s.to_i if reserve_timeout # as beanstalkd reads what beaneater sends: "5", 2.5, 5.seconds
+        timeout = SHUTDOWN_CHECK_INTERVAL if timeout.nil? || timeout < 0 || timeout > SHUTDOWN_CHECK_INTERVAL
+        super(conn, timeout)
+      end
     end # Threading
   end # Workers
 end # Backburner

data/lib/backburner/workers/threads_on_fork.rb

@@ -69,6 +69,7 @@ module Backburner
       # Custom initializer just to set @tubes_data
       def initialize(*args)
         @tubes_data = {}
+        @watching_tube = nil
         super
         self.process_tube_options
       end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: backburner
 version: !ruby/object:Gem::Version
-  version: 1.8.0
+  version: 1.9.0
 platform: ruby
 authors:
 - Nathan Esquenazi
@@ -164,27 +164,11 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/workflows/ci.yml"
-- ".github/workflows/github-release.yml"
-- ".github/workflows/release.yml"
-- ".gitignore"
 - CHANGELOG.md
-- CONTRIBUTING.md
-- Gemfile
 - HOOKS.md
 - LICENSE
 - README.md
-- Rakefile
-- TODO
-- backburner.gemspec
 - bin/backburner
-- examples/custom.rb
-- examples/demo.rb
-- examples/god.rb
-- examples/hooked.rb
-- examples/retried.rb
-- examples/simple.rb
-- examples/stress.rb
 - lib/active_job/queue_adapters/backburner_adapter.rb
 - lib/backburner.rb
 - lib/backburner/async_proxy.rb
@@ -205,41 +189,14 @@ files:
 - lib/backburner/workers/simple.rb
 - lib/backburner/workers/threading.rb
 - lib/backburner/workers/threads_on_fork.rb
-- test/active_job_adapter_test.rb
-- test/async_proxy_test.rb
-- test/back_burner_test.rb
-- test/connection_test.rb
-- test/fixtures/active_job_app.rb
-- test/fixtures/active_jobs.rb
-- test/fixtures/autoload/test_broken_name_job.rb
-- test/fixtures/autoload/test_broken_no_method_job.rb
-- test/fixtures/autoload/test_broken_require_job.rb
-- test/fixtures/autoload/test_broken_runtime_job.rb
-- test/fixtures/autoload/test_broken_syntax_job.rb
-- test/fixtures/autoload/test_broken_timeout_job.rb
-- test/fixtures/autoload/test_same_name_job.rb
-- test/fixtures/hooked.rb
-- test/fixtures/test_fork_jobs.rb
-- test/fixtures/test_forking_jobs.rb
-- test/fixtures/test_jobs.rb
-- test/fixtures/test_queue_settings.rb
-- test/helpers/templogger.rb
-- test/helpers_test.rb
-- test/hooks_test.rb
-- test/job_test.rb
-- test/logger_test.rb
-- test/performable_test.rb
-- test/queue_test.rb
-- test/test_helper.rb
-- test/worker_test.rb
-- test/workers/forking_worker_test.rb
-- test/workers/simple_worker_test.rb
-- test/workers/threading_worker_test.rb
-- test/workers/threads_on_fork_worker_test.rb
-homepage: http://github.com/nesquena/backburner
+homepage: https://github.com/nesquena/backburner
 licenses:
 - MIT
-metadata: {}
+metadata:
+  source_code_uri: https://github.com/nesquena/backburner
+  changelog_uri: https://github.com/nesquena/backburner/blob/master/CHANGELOG.md
+  bug_tracker_uri: https://github.com/nesquena/backburner/issues
+  rubygems_mfa_required: 'true'
 post_install_message:
 rdoc_options: []
 require_paths:
@@ -259,35 +216,4 @@ rubygems_version: 3.5.22
 signing_key:
 specification_version: 4
 summary: Reliable beanstalk background job processing made easy for Ruby and Sinatra
-test_files:
-- test/active_job_adapter_test.rb
-- test/async_proxy_test.rb
-- test/back_burner_test.rb
-- test/connection_test.rb
-- test/fixtures/active_job_app.rb
-- test/fixtures/active_jobs.rb
-- test/fixtures/autoload/test_broken_name_job.rb
-- test/fixtures/autoload/test_broken_no_method_job.rb
-- test/fixtures/autoload/test_broken_require_job.rb
-- test/fixtures/autoload/test_broken_runtime_job.rb
-- test/fixtures/autoload/test_broken_syntax_job.rb
-- test/fixtures/autoload/test_broken_timeout_job.rb
-- test/fixtures/autoload/test_same_name_job.rb
-- test/fixtures/hooked.rb
-- test/fixtures/test_fork_jobs.rb
-- test/fixtures/test_forking_jobs.rb
-- test/fixtures/test_jobs.rb
-- test/fixtures/test_queue_settings.rb
-- test/helpers/templogger.rb
-- test/helpers_test.rb
-- test/hooks_test.rb
-- test/job_test.rb
-- test/logger_test.rb
-- test/performable_test.rb
-- test/queue_test.rb
-- test/test_helper.rb
-- test/worker_test.rb
-- test/workers/forking_worker_test.rb
-- test/workers/simple_worker_test.rb
-- test/workers/threading_worker_test.rb
-- test/workers/threads_on_fork_worker_test.rb
+test_files: []

data/.github/workflows/ci.yml

@@ -1,45 +0,0 @@
-name: CI
-
-on:
-  push:
-    branches: [ "master" ]
-  pull_request:
-    branches: [ "master" ]
-
-permissions:
-  contents: read
-
-jobs:
-  test:
-    name: test (${{ matrix.ruby-version }}${{ matrix.activejob-version && format(', activejob {0}', matrix.activejob-version) || '' }})
-    runs-on: ubuntu-latest
-    strategy:
-      fail-fast: false
-      matrix:
-        ruby-version: ['2.7', '3.0', '3.1', '3.2', '3.3', '3.4', '4.0']
-        activejob-version: [''] # the newest the Ruby version allows
-        include:
-          # Rails before 7.2 ship their own Active Job adapter and have no AbstractAdapter
-          - ruby-version: '3.1'
-            activejob-version: '6.1'
-    env:
-      ACTIVEJOB_VERSION: ${{ matrix.activejob-version }} # read by the Gemfile
-
-    steps:
-    - uses: actions/checkout@v4
-    - name: Download & Extract beanstalkd
-      run: curl -fsSL https://github.com/beanstalkd/beanstalkd/archive/refs/tags/v1.13.tar.gz | tar xz
-    - name: Make beanstalkd
-      run: make
-      working-directory: beanstalkd-1.13
-    - name: Set up Ruby
-      uses: ruby/setup-ruby@v1
-      with:
-        ruby-version: ${{ matrix.ruby-version }}
-        bundler-cache: true # runs 'bundle install' and caches installed gems automatically
-    - name: Daemonize beanstalkd
-      run: ./beanstalkd-1.13/beanstalkd -l 127.0.0.1 -p 11300 &
-    - name: Run tests
-      run: bundle exec rake test
-      env:
-        BEANSTALKD_BIN: ${{ github.workspace }}/beanstalkd-1.13/beanstalkd # for the outage test, which stops and restarts its own server