backburner 1.7.0 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 83834669d5d4ccf208c32935bccba313a04b097611c10f5ebe071f81aae32b6d
-  data.tar.gz: 97d85b5bf0a1e9c623f5c9fd35d2d130d27599b53b9f554a56d36dd0eeab8ea1
+  metadata.gz: ad41e83594bda50d1b504a4e0e5777c477e5586dc88faddbbad476a81afb34d4
+  data.tar.gz: f53ac6873638f40f627bbcd1326f50fbd642e5454bce2c083b809cdc6904863b
 SHA512:
-  metadata.gz: b6eafd91259c9a156077451cbc5ff243317a70b92af17f45887566bfd50a358eff365aa407554760e1781482623fe96882fbc6473cc0541ed058eb0f5aa643e2
-  data.tar.gz: 9d90985cdbc5040961f183cfb560223b2bce8c547a10eeedf460c0f5c3db56726849888903668e8d201b43c5d7b2498f9cf1f1ffbcedd6501316c4222ad4e78f
+  metadata.gz: '0384db95f53944823490c2105948153c8aac2d0fb5e8268c8c6f82895ae722bf293bb93f7a0536967a9ad91a138d37f02d92f4f2b0e6860a749cb830065dd668'
+  data.tar.gz: 1e6cbde19cd41a7540346d5d66af0537af1fc53baec32466790217b0643b610cbd7d17f8ea74b1ed6aedd55a26c152d798df86f4ef6b21ac11b5bc848db9d175

data/CHANGELOG.md

@@ -1,5 +1,46 @@
 # CHANGELOG
 
+## Version 1.9.0 (Nov 1 2026)
+
+* The gem now ships only `lib/`, `bin/`, `README.md`, `HOOKS.md`, `CHANGELOG.md` and `LICENSE` (no tests, examples or CI files), and its rubygems.org page links the source, the changelog and the issue tracker. **Behaviour change for the gem's owners:** pushing a release to rubygems.org now requires MFA (`rubygems_mfa_required`); the release workflow's trusted publishing is unaffected. **What to do:** owners who push by hand need MFA enabled on their rubygems.org account
+* Docs: the README's Simple Async Jobs example says a `Backburner::Performable` class's queue defaults to the primary queue (`backburner-jobs`), not one named after the class, and its Persistence examples show the JSON jobs are stored as (`null`, double quotes) on the class's queue
+* Docs: the README says Backburner uses only the first of several `beanstalk_url`s (no client-side federation), and that the Threading worker's `queue:threads` CLI format needs `config.default_worker = Backburner::Workers::Threading`; `examples/god.rb` sets `QUEUE` (which the rake tasks read) instead of `QUEUES`; `examples/custom.rb` enqueues on the queue it works; CONTRIBUTING says the tests need beanstalkd on 127.0.0.1:11300; `backburner --help` shows `-q, --queues QUEUES`; dead links point to beanstalkd.github.io, rubygems.org and rubydoc.info; the 1.5.0 and 1.6.0 changelog entries are filled in from their commits, with their release dates
+* Docs: the README has an Active Job section: setting the adapter, how queues map to tubes, starting workers with `-q` for the Active Job queues (a worker started without `-q` doesn't pick up a queue first used after it started), the TTR, how `retry_on` and `max_job_retries` add up, and using Active Job callbacks rather than Backburner's hooks
+* Active Job: a job's TTR is now its class's `queue_respond_timeout` when it defines one (`def self.queue_respond_timeout; 3600; end`; subclasses inherit it), as for a `Backburner::Queue` job; before, every Active Job job got `config.respond_timeout` (120 seconds by default), so a job that ran longer was stopped with `Backburner::Job::JobTimeout` and retried or buried. Jobs of classes without one still get `config.respond_timeout`. The TTR is set when a job is enqueued, so jobs already queued, or enqueued by an app still on an older backburner, keep the TTR they had. **What to do:** give long-running Active Job classes a `queue_respond_timeout` longer than they take
+* Rails apps: a worker now runs each job inside the app's reloader (`Rails.application.reloader.wrap`), as Rails does for its own Active Job jobs and as Sidekiq does. **Behaviour change:** `ActiveSupport::CurrentAttributes` (a `Current.user` or tenant a job sets) no longer leak into the next job the worker runs; the database connections a job uses go back to the pool when it finishes; Active Record's query cache is on while a job runs, as it is for requests and Active Job jobs; and in development a job runs the app's latest code after a file changes (a reload waits for the jobs running in other threads to finish). On Rails 7.0 and newer, an error a job raises is now also reported to `Rails.error`, as an Active Job job's error already was (including a job whose class this worker doesn't have, each time it's released), but not a `Backburner::Job::RetryJob`, nor the `SystemExit` of a worker stopped by `TERM` or `INT` (or of a job that calls `exit`): an error tracker subscribed to `Rails.error` that `config.on_error` also calls may get it twice. An Active Job job runs in the worker's wrap, so Active Job's own wrap inside it does nothing: it's still wrapped once, but on Rails 7.1 and newer its errors, like other jobs', are now reported with the source `"job.backburner"` instead of `"application.active_support"` (a `Rails.error` subscriber that filters on the source sees the new one). `config.on_error` and the `on_retry` and `on_bury` hooks run after the job's wrap ends, so they no longer see the `CurrentAttributes` the job set, and a database connection they use is taken outside the executor (the `on_failure` hooks still run inside it). Outside Rails nothing changes. **What to do:** nothing, unless a job relies on state an earlier job left in `CurrentAttributes` (pass it as an argument instead), a job waits for a change another process makes to the database (read it inside `ActiveRecord::Base.uncached { ... }`, as the query cache would return the first result again), or an `on_error` handler or retry or bury hook reads the job's `CurrentAttributes` (pass what it needs in the job's arguments, or read it in an `on_failure` hook)
+* Fix "Worker exiting..." often not being logged when a worker stops on `TERM` or `INT`: the process could exit before the thread that logs it ran. The worker now waits up to a second for that line to be written before it exits. **What to do:** nothing
+* `Backburner::Performable` jobs (enqueued with `async`, `handle_asynchronously` or `handle_static_asynchronously`) now call only methods the app defines on the class, its superclasses (such as `ActiveRecord::Base`) or the modules they include, private ones too. **Behaviour change:** a job naming a method the class only gets from Ruby itself (`Object`, `Kernel` and `BasicObject`, or for a class method `Class` and `Module`, including what libraries add to them, such as `to_s`, `frozen?`, `name` or `try`), one of Backburner's own (`queue`, `perform`), or `method_missing` now fails with `NoMethodError`, and is retried and then buried like any other failure. A class that overrides such a method (`def to_s`) still has its own called. **What to do:** if your app enqueues such a method (`User.async.name`), define a method on the class that calls it and enqueue that instead. Jobs already queued keep the method name they were enqueued with, so they fail the same way once the workers are upgraded: to run them, define that same method on the class (`def self.name; super; end`) and kick them from the buried list, or delete them and enqueue the new method
+* Fix the Threading worker taking `shutdown_timeout` (10 seconds by default) to exit on `TERM` or `INT` when it had no jobs to work: its threads waited in `reserve` until the shutdown gave up on them and killed them. Each thread now waits for a job at most a second at a time (or `config.reserve_timeout`, if shorter; a negative one, which made beanstalkd wait forever, counts as a second), so an idle worker exits within about a second. A busy worker is unchanged: its in-flight jobs get up to `shutdown_timeout` to finish before their threads are killed. A job a thread reserves as the shutdown begins is worked like any in-flight job; one still running when its thread is killed is not lost: beanstalkd puts it back in the queue when the worker exits, and it runs again. **What to do:** nothing; an idle Threading worker now sends beanstalkd a `reserve-with-timeout` once a second per thread
+
+## Version 1.8.0 (Oct 31 2026)
+
+* Fix ThreadsOnFork and Forking children crashing on Ruby 3.3 with `[BUG] lazy sweeping underway when freeing object space` when they stopped on a `TERM` or `INT`, on an error, or when a job called `exit`, if a thread in the parent was running finalizers when the child was forked (an interpreter bug fixed in Ruby 3.4). On Ruby 3.3 a forked child now disables the GC as it exits, so its `at_exit` handlers (the app's, and those it inherited from the parent) run with the GC off, and a child running several threads stops its job threads before those handlers run (a job they were working on goes back to the queue and runs again). Other Rubies are unchanged. **What to do:** nothing; on Ruby 3.3 an `at_exit` handler that allocates a lot in a child now grows the child's memory until it exits
+* Fix a delay of a fraction of a second being cut short: `delay` is now rounded up to whole seconds instead of down, so a job never becomes ready before its delay is up. Before, `delay: 0.5` enqueued a job that was ready at once, and an Active Job `set(wait: 1)` or `set(wait_until: ...)`, whose delay is always a little under the time asked for once it reaches the adapter, ran up to a second early (`wait: 1` ran at once). Whole-second delays are unchanged; a fractional one now waits up to a second longer than asked, never less. A fractional retry delay (a `retry_delay_proc` that returns a Float, or a Float `config.retry_delay`) is rounded up the same way; before, beanstalkd refused it (`Beaneater::BadFormatError`), which stopped the worker (the simple worker exited, and a ThreadsOnFork or Forking child died) instead of retrying the job. A fractional TTR (`:ttr` when enqueueing, a class's `queue_respond_timeout` or `config.respond_timeout`) is rounded up to whole seconds too; a Float `config.respond_timeout` used to fail every enqueue (`Beaneater::BadFormatError`). **Behaviour change:** a Float `:ttr`, `queue_respond_timeout` or `queue_retry_delay` (such as `queue_retry_delay 1.5`) used to be ignored in favour of the global default (`config.respond_timeout`, `config.retry_delay`); it is now used, rounded up. The same goes for a fractional `ActiveSupport::Duration` or Rational (`queue_respond_timeout 1.5.minutes` now gives a TTR of 90 instead of the default). If you set one by mistake, remove it to keep the default. `on_retry` hooks now get the rounded delay, and a `retry_delay_proc` that returns `nil`, a String that isn't a number, or a negative number now retries at once (delay 0) instead of stopping the worker with `Beaneater::BadFormatError`; a `:ttr`, `queue_respond_timeout` or `queue_retry_delay` that isn't finite (`Float::INFINITY`, `NaN`) is still ignored in favour of the default
+* Fix a ThreadsOnFork child running more than one thread (`threads_number` or a tube's threads limit above 1) with `garbage_after` set stopping after its first `garbage_after` jobs on Ruby 3.2 and newer: once its threads had worked them, it waited for every thread in the process to finish, including the one the `timeout` gem keeps running since Ruby 3.2 (`timeout` 0.3), so it never exited to be forked again and the tube's jobs piled up. It now waits only for its own threads, so a thread a job leaves running no longer holds it up either. Ruby 3.1 and older with their bundled `timeout` weren't affected. **What to do:** nothing; a worker stuck this way recovers once upgraded and restarted
+* Fix the usage comment in `bin/backburner`: queues are given with `-q` (`backburner -q foo,bar`); a bare `backburner foo,bar` ignores them and works every known queue
+* Fix the ThreadsOnFork worker logging "Catastrophic failure: tube ... exited with code 0" for every child on a normal shutdown: while the worker shuts down, a child that exits with code 0 is now logged as "Tube ... exited normally.", and one that exits with another code or is killed (by the shutdown's `SIGKILL`) as "Tube ... stopped.", without being forked again. A child that exits with code 0 at another time (a signal sent to the child alone, or a job that called `exit`) is logged as a warning, "Tube ... exited with code 0. Forking it again in Ns."
+* Fix an error raised by `config.on_error` (an error reporter that can't reach its service, say) stopping the worker: it escaped `work_one_job`, which ended the simple worker and killed a ThreadsOnFork or Forking child; it is now logged ("The on_error handler failed for ..."). The failed job is still retried or buried as before
+* Fix the ThreadsOnFork worker forking children in a busy loop when they fail at once (hundreds a second while beanstalkd is down, each logging a "Catastrophic failure"): after a child fails, the parent now waits 1 second before forking the next one, doubling up to 30 seconds while children keep failing, and starts over once a child is recycled or runs for 30 seconds; recycling a child after `garbage_after` jobs is still immediate. A child running several threads now exits with code 1 (instead of 99) when one of its threads died of an error, so it backs off too
+* Fix a job whose class can't be found or fails to load stopping the worker: the `Backburner::Job::JobNotFound` raised while handling its failure escaped `work_one_job` (so the simple worker exited and a ThreadsOnFork child died and was forked again in a busy loop), and so did any error other than a `NameError` raised while its class's file loaded (a `LoadError`, `KeyError`, `SyntaxError` and so on). When the class is missing (renamed or removed, or enqueued by code deployed before the workers: there is no such constant, or its autoload file isn't there), the worker now logs a warning ("... class not found; released for Ns") and releases the job with a delay of its TTR (at most 60 seconds), so a worker that has the class picks it up; it is never buried for this and doesn't use up its retries (beanstalkd still counts the releases, so a later failure gets fewer retries), and `on_error` and the failure hooks aren't called for it, as before. **What to do:** jobs of a class you removed on purpose now keep coming back, with a warning each time, instead of stopping the worker: delete them (with a beanstalkd client, say) or keep a stub class whose `perform` does nothing until they are drained. When the class's file fails to load (it raises a `NameError` for another constant, `NoMethodError`, `ArgumentError` or any other error, has a syntax error, or requires a missing gem), or the job names something that can't be a class (`"not a class"`), the job fails like any other: it is retried and then buried, `on_error` gets the `JobNotFound`, and its message names the cause on one line (`JobNotFound: SomeJob (LoadError: cannot load such file -- some_gem)`). **What to do:** fix the file and deploy, then kick the buried jobs
+* Fix `handle_asynchronously` and `handle_static_asynchronously` wiping the class's queue settings: its jobs went to the primary queue (`backburner-jobs`) with the default priority and TTR instead of the class's `queue`, `queue_priority` and `queue_respond_timeout`. Including `Backburner::Performable` in a class that already includes `Backburner::Queue` no longer resets the settings it set either. **Behaviour change:** jobs of a class that calls `handle_asynchronously`/`handle_static_asynchronously` (including its `async` calls) now use the class's `queue`, `queue_priority` and `queue_respond_timeout` (before, the settings declared ahead of the call were lost). Workers started with `-q` or `default_queues` must also watch the class's tube (a worker started with no queues already does). Jobs enqueued before the upgrade stay on the primary queue, so keep a worker on it until they are drained
+* Fix apps on Rails 6.1, 7.0 and 7.1 failing to boot with backburner 1.7.0 (`uninitialized constant ActiveJob::QueueAdapters::AbstractAdapter` from `require 'backburner'`, or from `config.active_job.queue_adapter = :backburner`): the gem's Active Job adapter subclasses `AbstractAdapter` only on Rails 7.2+, which added it, and `require 'backburner'` loads it only there; older Rails autoload the adapter as before. Where the gem's adapter loads in place of Rails' own (whenever the gem's lib comes first on the load path, as Bundler puts it), it also sets `provider_job_id` on Rails 6.1 and 7.0. The adapter now requires backburner itself, as Rails' own adapter did, so apps with `gem 'backburner', require: false` can enqueue again (since 1.7.0 the first enqueue raised `uninitialized constant Backburner::Worker`). CI now also runs the suite with Active Job 6.1
+* Fix a worker started without queues (`backburner` with no `-q`) never working the tubes it found on the server when the namespace has capitals (`MyApp`), an underscore or regexp characters (`jobs(eu)`, `app+`), when the separator isn't `.`, or for queue names such as ActiveJob's `production_mailers` under the namespace `production` (tube `production.-mailers`) and `report_2026`: it expanded those tube names again and watched tubes nobody enqueues on (`MyApp.my-app.default` instead of `MyApp.default`). It now watches the server's tubes this configuration could have produced (they start with the namespace and separator, and the rest has no capitals or underscores), under their own names, and a found tube with `--` in its name (queue `a__b`) no longer stops it at start with `NoMethodError`. The tube names jobs are enqueued on don't change
+* Fix a `tube_namespace` that isn't a valid regexp (such as `app(`) failing every enqueue and stopping workers with `RegexpError`
+* Fix `namespace_separator = ''` failing every enqueue with `RegexpError`: the namespace and the queue name are now joined directly (`appdefault`, or `app.default` for the namespace `app.`)
+* Fix a failing job stopping the worker when it runs in a directory whose name has regexp characters (such as `jobs[`): logging the error raised `RegexpError`
+* Fix ThreadsOnFork with one thread per tube: each forked child now opens its own beanstalkd connection instead of sharing the parent's socket, which mixed up replies between children (`undefined method 'ttr' for nil`, lost jobs) (reported by @Haerezis in #169 and @supremoUL in #175)
+* Fix workers never recovering after beanstalkd is down for more than a few seconds: a failed reconnect no longer leaves the connection unusable (`undefined method 'tubes'` in a busy loop), and the simple worker watches its tubes again after reconnecting; a job that fails while the connection is down no longer stops the simple worker either. A job that finishes while beanstalkd can't be reached (past beaneater's reconnect attempts) can't be deleted: `on_error` gets the `Beaneater::NotConnected`, and beanstalkd hands the job out again if it still has it (reported by @activestylus in #137)
+* Fix `Connection#retryable` raising `NameError` instead of `Beaneater::NotConnected` once its retries are used up
+* Fix a job that outlives its TTR stopping the worker: when beanstalkd has already taken the job back, the `NOT_FOUND` from the retry or bury is now logged instead of escaping `work_one_job` (reported by @boncey in #82)
+* Behaviour change: a `before_perform` hook returning false now deletes the job, as HOOKS.md describes; before, the job stayed reserved and beanstalkd handed it out again after every TTR (reported by @inoce1e in #145)
+* Fix docs: a TTR of 0 does not disable the timeout, beanstalkd stores it as 1 (reported by @pnabutovsky in #126); the HOOKS.md failure-hook examples no longer call `self.bury`, which raised `NoMethodError` and hid the original error (reported by @contentfree in #172); the README no longer points to the removed `retry_connection!` (reported by @att14 in #62)
+* Fix queue settings not being inherited: a subclass of a `Backburner::Queue` job class now uses its parent's `queue`, `queue_priority`, `queue_respond_timeout`, `queue_max_job_retries`, `queue_retry_delay`, `queue_retry_delay_proc`, `queue_jobs_limit`, `queue_garbage_limit` and `queue_retry_limit` unless it sets its own, and is registered in `Backburner::Worker.known_queue_classes` (anonymous classes, such as `Class.new(SomeJob)`, are left out when workers list the known queues). For the Threading and ThreadsOnFork workers, a tube's `queue_jobs_limit`, `queue_garbage_limit` and `queue_retry_limit` come from the class that declares the tube; subclasses that share it only fill in the limits it leaves unset. **Behaviour change:** a subclass that didn't set `queue` used to enqueue on the primary queue (`backburner-jobs`); it now enqueues on its parent's queue (tube). Jobs it enqueued before the upgrade stay on the primary queue, so keep a worker on that tube until they are drained. A subclass can't unset an inherited setting: the setters ignore `nil` and `false` (reported by @contentfree in #141)
+* Fix every enqueue opening a new TCP connection to beanstalkd (closed sockets piled up in `TIME_WAIT` until enqueueing failed with `Beaneater::NotConnected`): enqueueing reuses pooled connections (up to 5 idle per process, each reused for up to `config.enqueue_connection_max_idle` seconds, default 60), a forked child opens its own, and a connection that an error interrupted is closed rather than reused. While beanstalkd is down, each enqueue that picks up an idle pooled connection (up to 5) now takes about 2 seconds of retries before raising `Beaneater::NotConnected`, instead of failing at once; later ones fail at once (reported by @xinuc in #125, confirmed by @funkyboy)
+* Allow `config.tube_namespace = nil` (or `false`) for tubes without a prefix: jobs go to `backburner-jobs`, `newsletter-sender` and so on. `nil` and `false` used to raise `NoMethodError`; `''` still gives tube names that start with the separator (`.backburner-jobs`), as before. With no namespace, a worker started without tubes watches every tube on the server without capitals or underscores (including `default`), so give your tubes distinct names on a shared beanstalkd (@forresty in #94)
+* Fix `config.default_priority` set to a priority label (such as `:high`) failing every enqueue with `Beaneater::BadFormatError`: the label now resolves like a job's priority label (reported by @rewritten in #52, tracked in #54)
+* Fix `require 'backburner'` on Ruby 4.0 in a bundle without ActiveSupport (`cannot load such file -- logger`): `logger` is no longer a default gem, so the gemspec now depends on it
+* Require Ruby 2.7 or newer (the gemspec now says so; CI covers 2.7 to 4.0)
+
 ## Version 1.7.0 (Jan 2 2026)
 
 * Changes to ActiveJob Rails adapter to remove default adapter and associated tests
@@ -8,13 +49,21 @@
 
 * Change `File.exists?`` to `File.exist?` as latest ruby versions have removed support (@ShrutiSaagar).
 
-## Version 1.6.0 (December 30 2021)
+## Version 1.6.0 (Dec 30 2021)
 
-* TBD (please help backfill)
+* Allow per-job `queue_retry_delay`, `queue_retry_delay_proc` and `queue_max_job_retries` (@boncey in #162)
+* Fix the beanstalkd links in the README (@a112121788 in #171)
 
-## Version 1.5.0 (September 10 2018)
+## Version 1.5.0 (Sep 19 2018)
 
-* TBD
+* Add `Backburner::Job::RetryJob`: raise it from a job to retry it without logging an error (@eltone in #159)
+* Threading worker: shut down gracefully on `TERM` or `INT`, letting running jobs finish; a second signal stops it at once (@eltone in #157)
+* Make the logger's job timings thread-safe (@eltone in #157)
+* Log, retry and bury jobs whose class is unknown instead of raising while handling their failure (@eltone in #154)
+* Fix Ruby warnings about a shadowed and an unused variable (@utilum in #153)
+* Allow concurrent-ruby 1.x after 1.0.1 (`~> 1.0`, `>= 1.0.1`)
+* Docs: fix the link to Adam Wiggins' beanstalkd comparison (@dplummer in #152); add a CI badge (@szTheory in #151)
+* CI: test against Ruby 2.x (@nicolasleger in #150)
 
 ## Version 1.4.1 (June 10 2017)
 

data/HOOKS.md

@@ -17,7 +17,7 @@ There are a variety of hooks available that are triggered during the lifecycle o
   Any exception raised propagates up to the code which queued the job.
 
 * `before_perform`: Called with the job args before perform. If a hook returns false,
-  the job is aborted. Other exceptions are treated like regular job exceptions.
+  the job is aborted and deleted from the queue. Other exceptions are treated like regular job exceptions.
 
 * `after_perform`: Called with the job args after it performs. Uncaught
   exceptions will be treated like regular job exceptions.
@@ -41,9 +41,8 @@ class SomeJob
     logger.info "About to perform #{self} with #{args.inspect}"
   end
 
-  def self.on_failure_bury(e, *args)
+  def self.on_failure_log_job(e, *args)
     logger.info "Performing #{self} caused an exception (#{e})"
-    self.bury
   end
 
   def self.perform(*args)
@@ -56,25 +55,27 @@ class SomeJob
 end
 ```
 
+A failure hook doesn't need to retry or bury the job: the worker retries a failed job up to `max_job_retries`
+times and then buries it. Use an `on_retry` or `on_bury` hook to act on those.
+
 You can also setup modules to create compose-able and reusable hooks for your jobs. For example:
 
 ```ruby
 module LoggedJob
   def before_perform_log_job(*args)
-    Logger.info "About to perform #{self} with #{args.inspect}"
+    puts "About to perform #{self} with #{args.inspect}"
   end
 end
 
-module BuriedJob
-  def on_failure_bury(e, *args)
-    Logger.info "Performing #{self} caused an exception (#{e}). Retrying..."
-    self.bury
+module NotifiedJob
+  def on_failure_notify(e, *args)
+    puts "Performing #{self} caused an exception (#{e})"
   end
 end
 
 class MyJob
   extend LoggedJob
-  extend BuriedJob
+  extend NotifiedJob
 
   def self.perform(*args)
     # ...
@@ -86,14 +87,22 @@ end
 
 Currently, there is just one hook:
 
-* `on_reconnect`: Called on the worker whose connection has been reset. The connection
-  is given as the argument
+* `on_reconnect`: Called on a Simple or ThreadsOnFork worker after Backburner re-establishes
+  a lost connection, with the new connection as the argument. A short outage that beanstalk's
+  client library rides out on its own reconnects without it (and keeps the watched tubes). The
+  Threading worker never calls it (each thread's connection re-watches its own tube), and
+  Forking workers' children reconnect through the client library, so it doesn't fire there either.
+  Every public method whose name starts with `on_reconnect` is called, so add your own as
+  `on_reconnect_<something>`. The Simple, Forking and ThreadsOnFork
+  workers define `on_reconnect` itself to watch their tubes again; if you override
+  `on_reconnect` in a subclass of one of them, call `super`.
 
 An example:
 
 ```ruby
-class MyWorker < Backburner::Worker
-  def on_reconnect(conn)
-    prepare
+class MyWorker < Backburner::Workers::Simple
+  def on_reconnect_log(conn)
+    log_info "reconnected to beanstalkd"
   end
 end
+```

data/README.md

@@ -1,4 +1,4 @@
-# Backburner [![Build Status](https://travis-ci.org/nesquena/backburner.svg?branch=master)](https://travis-ci.org/nesquena/backburner)
+# Backburner [![CI](https://github.com/nesquena/backburner/actions/workflows/ci.yml/badge.svg)](https://github.com/nesquena/backburner/actions/workflows/ci.yml)
 
 Backburner is a [beanstalkd](https://beanstalkd.github.io/)-powered job queue that can handle a very high volume of jobs.
 You create background jobs and place them on multiple work queues to be processed later.
@@ -57,7 +57,7 @@ Beanstalkd supports the following features out of the box:
 | **Fast**            | Processes thousands of jobs per second without breaking a sweat. |
 | **Priorities**      | Specify priority so important jobs can be processed quickly. |
 | **Persistence**     | Jobs are stored in memory for speed, but logged to disk for safe keeping. |
-| **Federation**      | Horizontal scalability provided through federation by the client. |
+| **Federation**      | Horizontal scalability provided through federation by the client (Backburner connects to a single beanstalkd: given several `beanstalk_url`s, it uses the first). |
 | **Error Handling**  | Bury any job which causes an error for later debugging and inspection.|
 
 Keep in mind that these features are supported out of the box with beanstalk and require no special code within this gem to support.
@@ -70,6 +70,9 @@ Depending on your platform, this should be as simple as (for Ubuntu):
 
     $ sudo apt-get install beanstalkd
 
+beanstalkd has no authentication: anyone who can connect to it can add jobs, so make it reachable only
+from your app's own hosts (bind it to `127.0.0.1` or a private network, or firewall its port).
+
 Add this line to your application's Gemfile:
 
     gem 'backburner'
@@ -113,14 +116,14 @@ The key options available are:
 | Option                | Description                                                          |
 | -----------------     | -------------------------------                                      |
 | `beanstalk_url`       | Address for beanstalkd connection i.e 'beanstalk://127.0.0.1'        |
-| `tube_namespace`      | Prefix used for all tubes related to this backburner queue.          |
+| `tube_namespace`      | Prefix used for all tubes related to this backburner queue (`nil` or `false` for no prefix; a worker given no tubes then watches every tube on the server without capitals or underscores, so give your tubes distinct names on a shared beanstalkd; `''` gives tube names that start with the separator, such as `.backburner-jobs`, as before). |
 | `namespace_separator` | Separator used for namespace and queue name                          |
 | `on_error`            | Lambda invoked with the error whenever any job in the system fails.  |
 | `max_job_retries`     | Integer defines how many times to retry a job before burying.        |
-| `retry_delay`         | Integer defines the base time to wait (in secs) between job retries. |
+| `retry_delay`         | Number defines the base time to wait (in secs) between job retries.  |
 | `retry_delay_proc`    | Lambda calculates the delay used, allowing for exponential back-off. |
-| `default_priority`    | Integer The default priority of jobs                                 |
-| `respond_timeout`     | Integer defines how long a job has to complete its task              |
+| `default_priority`    | Integer or priority label: the default priority of jobs              |
+| `respond_timeout`     | Number defines how long a job has to complete its task (in secs)     |
 | `default_worker`      | Worker class that will be used if no other worker is specified.      |
 | `logger`              | Logger recorded to when backburner wants to report info or errors.   |
 | `primary_queue`       | Primary queue used for a job when an alternate queue is not given.   |
@@ -128,6 +131,17 @@ The key options available are:
 | `reserve_timeout`     | Duration to wait for work from a single server, or nil for forever.  |
 | `job_serializer_proc` | Lambda serializes a job body to a string to write to the task        |
 | `job_parser_proc`     | Lambda parses a task body string to a hash                           |
+| `enqueue_connection_max_idle` | Seconds an idle enqueue connection is reused for before it is replaced (default 60; `nil` for no limit; 0 or less opens a new connection for every enqueue). |
+
+A tube's name is the namespace, the separator and the queue name, dasherized (`some.app.production.newsletter-sender`).
+Don't start queue names with the namespace: that part of the queue name is dropped, so with the namespace `app` the queues
+`mail` and `appmail` share the tube `app.mail` and `apple-sync` uses `app.le-sync` (an ActiveJob `queue_name_prefix` equal
+to the namespace does the same: `production_mailers` uses `production.-mailers`). It stays that way because changing it
+would leave queued jobs in the old tubes. A worker given no tubes watches the tubes on the server whose names start with the
+namespace and separator and whose rest has no capitals or underscores (the names this configuration could have produced),
+the tubes of the job classes it has loaded and the primary queue. On a shared beanstalkd, don't give one app a namespace
+that starts with another's namespace and separator (`app` and `app.prod`, `MyApp` and `MyApp.admin`): workers of the
+shorter one given no tubes reserve the other's jobs, which they can't run.
 
 ## Breaking Changes
 
@@ -165,7 +179,7 @@ class NewsletterJob
 
   # optional, defaults to respond_timeout in config
   def self.queue_respond_timeout
-    300 # number of seconds before job times out, 0 to avoid timeout. NB: A timeout of 1 second will likely lead to race conditions between Backburner and beanstalkd and should be avoided
+    300 # number of seconds before job times out; the minimum is 1 (beanstalkd turns 0 into 1), so give long jobs a generous value. NB: A timeout of 1 second will likely lead to race conditions between Backburner and beanstalkd and should be avoided
   end
 
   # optional, defaults to retry_delay_proc in config
@@ -192,7 +206,7 @@ class NewsletterJob
   include Backburner::Queue
   queue "newsletter-sender"  # defaults to 'backburner-jobs' tube
   queue_priority 1000 # most urgent priority is 0
-  queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout
+  queue_respond_timeout 300 # number of seconds before job times out (minimum 1; 0 does not disable it)
 
   def self.perform(email, body)
     NewsletterMailer.deliver_text_to_email(email, body)
@@ -200,6 +214,11 @@ class NewsletterJob
 end
 ```
 
+A subclass of such a job uses its parent's settings (including its tube) unless it sets its own.
+It can't unset one: the setters ignore `nil` and `false`. For the Threading and ThreadsOnFork workers, a shared
+tube's limits (`queue_jobs_limit`, `queue_garbage_limit`, `queue_retry_limit`) come from the class that declares the
+tube; subclasses only fill in the limits it leaves unset.
+
 Jobs can be enqueued with:
 
 ```ruby
@@ -222,9 +241,9 @@ includes `Backburner::Performable`. Async enqueuing works for both instance and
 ```ruby
 class User
   include Backburner::Performable
-  queue "user-jobs"  # defaults to 'user'
+  queue "user-jobs"  # defaults to the primary queue, 'backburner-jobs'
   queue_priority 500 # most urgent priority is 0
-  queue_respond_timeout 300 # number of seconds before job times out, 0 to avoid timeout
+  queue_respond_timeout 300 # number of seconds before job times out (minimum 1; 0 does not disable it)
 
   def activate(device_id)
     @device = Device.find(device_id)
@@ -247,6 +266,10 @@ This automatically enqueues a job for that user record that will run `activate`
 Note that you can set the queue name and queue priority at the class level and
 you are also able to pass `pri`, `ttr`, `delay` and `queue` directly as options into `async`.
 
+The worker calls only methods your app defines on the class, its superclasses or the modules they include (private
+ones too). A job naming a method every object or class has (`frozen?`, `instance_variable_get`, `name`, ...), or one
+of Backburner's own (`queue`, `perform`), fails with `NoMethodError`.
+
 The queue name used by default is `{namespace}.backburner-jobs` if not otherwise
 specified.
 
@@ -356,12 +379,12 @@ example from above. We'll run the following code to create a job:
 User.async.reset_password(@user.id)
 ```
 
-The following JSON will be put on the `{namespace}.backburner-jobs` queue:
+The following JSON will be put on the `{namespace}.user-jobs` queue (the class's queue; `{namespace}.backburner-jobs` for a class without one):
 
 ``` javascript
 {
-    'class': 'User',
-    'args': [nil, 'reset_password', 123]
+    "class": "User",
+    "args": [null, "reset_password", 123]
 }
 ```
 
@@ -377,8 +400,8 @@ would be stored as:
 
 ``` javascript
 {
-    'class': 'User',
-    'args': [246, 'activate', 987]
+    "class": "User",
+    "args": [246, "activate", 987]
 }
 ```
 
@@ -408,6 +431,8 @@ Backburner::Worker.enqueue NewsletterJob, ["foo", "bar"], :pri => :custom
 User.async(:pri => :useful, :delay => 10.seconds).reset_password(@user.id)
 ```
 
+They work for `config.default_priority` too (e.g. `config.default_priority = :custom`).
+
 Using named priorities can greatly simplify priority management.
 
 ### Processing Strategies
@@ -456,7 +481,7 @@ For more information on the threads_on_fork worker, check out the
 
 #### More info: Threading Worker (thread-pool-based)
 
-Configuration options for the Threading worker are similar to the threads_on_fork worker, sans the garbage option. When running via the `backburner` CLI, it's simplest to provide the queue names and maximum number of threads in the format "{queue name}:{max threads in pool}[,{name}:{threads}]":
+Configuration options for the Threading worker are similar to the threads_on_fork worker, sans the garbage option. The `backburner` CLI runs the default worker, so set `config.default_worker = Backburner::Workers::Threading` in the app it loads; then it's simplest to provide the queue names and maximum number of threads in the format "{queue name}:{max threads in pool}[,{name}:{threads}]" (the Simple and Forking workers ignore the thread counts):
 
 ```
 $ bundle exec backburner -q queue1:4,queue2:4  # and then other options, like environment, pidfile, app root, etc. See docs for the CLI
@@ -555,9 +580,58 @@ Often using hooks is much easier then trying to monkey patch the externals.
 
 Check out [HOOKS.md](https://github.com/nesquena/backburner/blob/master/HOOKS.md) for a detailed overview on using hooks.
 
+### Active Job
+
+Backburner is an [Active Job](https://guides.rubyonrails.org/active_job_basics.html) adapter. Set it in
+`config/application.rb` (or an environment file):
+
+```ruby
+config.active_job.queue_adapter = :backburner
+```
+
+An Active Job queue is a tube named like any other queue: the `tube_namespace`, the separator and the queue name,
+dasherized. With the namespace `myapp.production`, `queue_as :low_priority` uses the tube `myapp.production.low-priority`;
+a `queue_name_prefix` is part of the queue name (`production_mailers` uses `myapp.production.production-mailers`).
+A job's priority (`queue_with_priority`, `set(priority: 10)`) is its beanstalkd priority (lower runs first), and
+`set(wait: ...)` or `set(wait_until: ...)` delays it (rounded up to whole seconds).
+
+Start workers with `-q` listing the Active Job queues, named as Active Job names them (with any prefix):
+
+```
+bundle exec backburner -q default,mailers,low_priority
+```
+
+A worker started without `-q` only watches the tubes that are on the server when it starts (and those of the
+`Backburner::Queue` classes it has loaded, and the primary queue), so the jobs of a queue first used after it started
+aren't worked until it restarts.
+
+A job's time to run (TTR) is its class's `queue_respond_timeout` when it defines one, otherwise `config.respond_timeout`
+(120 seconds by default). A job still running when its TTR is up is stopped with `Backburner::Job::JobTimeout`:
+
+```ruby
+class ReportJob < ApplicationJob
+  def self.queue_respond_timeout
+    3600 # seconds; subclasses inherit it
+  end
+end
+```
+
+Active Job's `retry_on` and Backburner's `max_job_retries` add up: once `retry_on` gives up and the error reaches
+Backburner, it retries the job `max_job_retries` more times before burying it, calling `on_error` each time. Keep
+`max_job_retries` at 0 (the default) for Active Job jobs and retry with `retry_on`, or expect both.
+
+Backburner's [hooks](#hooks) don't run for Active Job jobs (they are all enqueued as the adapter's wrapper class):
+use Active Job's callbacks (`before_perform`, `around_perform`, `after_enqueue` and so on) instead. `config.on_error`
+is still called when a job raises.
+
+In a Rails app, a worker runs each job (Active Job or not) in the app's reloader, as Rails does: `CurrentAttributes` are
+reset between jobs, a job's database connections go back to the pool when it's done, Active Record's query cache is on
+while a job runs (a job that waits for a change another process makes reads it in `ActiveRecord::Base.uncached { ... }`),
+and in development jobs run the latest code (a reload waits for the jobs running in other threads).
+
 ### Workers in Production
 
-Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](http://kr.github.com/beanstalkd/download.html)
+Once you have Backburner setup in your application, starting workers is really easy. Once [beanstalkd](https://beanstalkd.github.io/download.html)
 is installed, your best bet is to use the built-in rake task that comes with Backburner. Simply add the task to your Rakefile:
 
 ```ruby
@@ -611,7 +685,12 @@ backburner -e $environment
 
 In Backburner, if the beanstalkd connection is temporarily severed, several retries to establish the connection will be attempted.
 After several retries, if the connection is still not able to be made, a `Beaneater::NotConnected` exception will be raised.
-You can manually catch this exception, and attempt another manual retry using `Backburner::Worker.retry_connection!`.
+Workers log the exception and try again on their next reserve, and watch their tubes again once the connection is back.
+Elsewhere (when enqueuing, for example) you can catch the exception and retry the call yourself.
+
+Enqueueing reuses a small pool of connections (up to 5 idle per process). While beanstalkd is down, each enqueue
+that picks up one of those idle connections takes about 2 seconds of retries before raising `Beaneater::NotConnected`;
+once they are used up, enqueues fail at once until beanstalkd is back.
 
 ### Web Front-end
 
@@ -651,8 +730,8 @@ Thanks to these projects for inspiration and certain design and implementation d
 
 ## Links
 
- * Code: `git clone git://github.com/nesquena/backburner.git`
- * Home: <http://github.com/nesquena/backburner>
- * Docs: <http://rdoc.info/github/nesquena/backburner/master/frames>
- * Bugs: <http://github.com/nesquena/backburner/issues>
- * Gems: <http://gemcutter.org/gems/backburner>
+ * Code: `git clone https://github.com/nesquena/backburner.git`
+ * Home: <https://github.com/nesquena/backburner>
+ * Docs: <https://www.rubydoc.info/gems/backburner>
+ * Bugs: <https://github.com/nesquena/backburner/issues>
+ * Gems: <https://rubygems.org/gems/backburner>

data/bin/backburner

@@ -3,5 +3,5 @@
 require 'backburner'
 require 'backburner/cli'
 
-# bundle exec backburner foo,bar
+# bundle exec backburner -q foo,bar
 Backburner::CLI.start(ARGV)

data/lib/active_job/queue_adapters/backburner_adapter.rb

@@ -1,23 +1,37 @@
+# Rails' own adapter files require backburner too, for apps that don't require it themselves
+require 'backburner' unless defined?(Backburner::Worker)
+
 module ActiveJob
   module QueueAdapters
-    # Explicitly remove the implementation existing in older rails'.
-    remove_const(:BackburnerAdapter) if const_defined?(:BackburnerAdapter)
+    # Rails 7.2 added AbstractAdapter; the adapters of older Rails have no superclass
+    rails_7_2_or_newer = const_defined?(:AbstractAdapter, false)
+
+    # Explicitly remove the implementation rails' 7.2+ ship. Older rails' keep theirs, unless their autoload
+    # loads this file instead (it shadows theirs when this gem's lib comes first on the load path).
+    remove_const(:BackburnerAdapter) if rails_7_2_or_newer && const_defined?(:BackburnerAdapter)
 
     # = Backburner adapter for Active Job
     #
     # To use Backburner set the queue_adapter config to +:backburner+.
     #
     #   Rails.application.config.active_job.queue_adapter = :backburner
-    class BackburnerAdapter < ::ActiveJob::QueueAdapters::AbstractAdapter
+    #
+    # A job's TTR is its class's +queue_respond_timeout+ when it has one, as for a Backburner::Queue job
+    # (+ttr: job.class+ resolves it), else +config.respond_timeout+:
+    #
+    #   class ReportJob < ApplicationJob
+    #     def self.queue_respond_timeout; 3600; end
+    #   end
+    class BackburnerAdapter < (rails_7_2_or_newer ? ::ActiveJob::QueueAdapters::AbstractAdapter : Object)
       def enqueue(job)
-        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority)
+        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority, ttr: job.class)
         job.provider_job_id = response[:id] if response.is_a?(Hash)
         response
       end
 
       def enqueue_at(job, timestamp)
         delay = timestamp - Time.current.to_f
-        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority, delay: delay)
+        response = Backburner::Worker.enqueue(JobWrapper, [job.serialize], queue: job.queue_name, pri: job.priority, delay: delay, ttr: job.class)
         job.provider_job_id = response[:id] if response.is_a?(Hash)
         response
       end

data/lib/backburner/cli.rb

@@ -10,7 +10,7 @@ module Backburner
         opts.on("-r", "--require PATH", String, "The path to load as the environment.") do |req|
           options[:require] = req
         end
-        opts.on("-q", "--queues PATH", String, "The specific queues to work.") do |queues|
+        opts.on("-q", "--queues QUEUES", String, "The specific queues to work (comma-separated).") do |queues|
           options[:queues] = queues
         end
         opts.on("-e", "--environment ENVIRONMENT", String, "The environment to run Backburner within") do |environment|

data/lib/backburner/configuration.rb

@@ -19,6 +19,7 @@ module Backburner
     attr_accessor :reserve_timeout     # duration to wait to reserve on a single server
     attr_accessor :job_serializer_proc # proc to write the job body to a string
     attr_accessor :job_parser_proc     # proc to parse a job body from a string
+    attr_reader   :enqueue_connection_max_idle # seconds an idle enqueue connection is reused for (nil: no limit)
 
     def initialize
       @beanstalk_url       = "beanstalk://127.0.0.1"
@@ -38,6 +39,12 @@ module Backburner
       @reserve_timeout     = nil
       @job_serializer_proc = lambda { |body| body.to_json }
       @job_parser_proc     = lambda { |body| JSON.parse(body) }
+      @enqueue_connection_max_idle = 60
+    end
+
+    # Accepts a number or a numeric string (from ENV, say), or nil for no limit
+    def enqueue_connection_max_idle=(seconds)
+      @enqueue_connection_max_idle = seconds.nil? ? nil : Float(seconds)
     end
 
     def namespace_separator=(val)

data/lib/backburner/connection.rb

@@ -44,7 +44,9 @@ module Backburner
     # completely new connection)
     # @raise [Beaneater::NotConnected] If beanstalk fails to connect
     def reconnect!
-      close
+      # Close the socket but keep the delegate: if connecting fails, the next call
+      # must still reach ensure_connected! and try again (unlike after #close)
+      @beanstalk.close if @beanstalk
       connect!
       @on_reconnect.call(self) if @on_reconnect.respond_to?(:call)
     end
@@ -67,7 +69,7 @@ module Backburner
       begin
         yield
 
-      rescue Beaneater::NotConnected
+      rescue Beaneater::NotConnected => e
         if retry_count > 0
           reconnect!
           retry_count -= 1

data/lib/backburner/enqueue_pool.rb

@@ -0,0 +1,90 @@
+module Backburner
+  # The connections Worker.enqueue puts jobs with, shared by the process's threads and fibers,
+  # so enqueueing doesn't open a TCP connection per job (that ran out of ports, #125).
+  # A connection is checked out for one enqueue and checked back in after it; up to
+  # MAX_IDLE_CONNECTIONS stay open between enqueues and the rest are closed.
+  #
+  # ceiling: idle connections are only closed when they are next checked out (or when the URL
+  # changes or the process exits); a reaper thread would close them sooner.
+  class EnqueuePool
+    MAX_IDLE_CONNECTIONS = 5
+
+    @current = nil
+
+    # The pool of this process: a forked child gets a new one and leaves its parent's
+    # sockets (and a lock another thread may have held at the fork) alone.
+    # ceiling: no lock, so threads that race on the first enqueue (or the first after a fork) may
+    # each make a pool; only the last one is kept, and the others' connections stay open until
+    # garbage collected. A class-level lock would need replacing after a fork, as a thread that
+    # doesn't exist in the child may hold it.
+    def self.current
+      pool = @current
+      pool && pool.pid == Process.pid ? pool : (@current = new)
+    end
+
+    attr_reader :pid
+
+    def initialize
+      @pid  = Process.pid
+      @lock = Mutex.new
+      @url  = nil
+      @idle = [] # [connection, idle since], most recently used last
+    end
+
+    # Yields a connection to the configured beanstalk_url. If the block doesn't finish (it raises,
+    # or its thread is killed, or it is left with throw), the connection is closed instead of
+    # checked back in, since an interrupted command may have left a reply unread on it.
+    def with_connection
+      url, connection = checkout
+      finished = false
+      begin
+        result = yield connection
+        finished = true
+        result
+      ensure
+        if finished
+          checkin(url, connection)
+        else
+          connection.close rescue nil
+        end
+      end
+    end
+
+    private
+
+    def checkout
+      url = Backburner.configuration.beanstalk_url
+      max_idle = Backburner.configuration.enqueue_connection_max_idle
+      to_close = []
+      connection = @lock.synchronize do
+        unless @url == url # the URL changed: its old connections go
+          to_close.concat(@idle)
+          @idle = []
+          @url = url
+        end
+        # Connections idle for too long may have been dropped by a firewall or load balancer
+        # without either end knowing, and a put on one would hang until TCP gives up
+        now = monotonic_now
+        expired, @idle = @idle.partition { |_conn, since| max_idle && now - since > max_idle }
+        to_close.concat(expired)
+        conn, _since = @idle.pop
+        conn
+      end
+      to_close.each { |conn, _since| conn.close rescue nil }
+      [url, connection || Backburner::Connection.new(url)]
+    end
+
+    def checkin(url, connection)
+      kept = @lock.synchronize do
+        if @url == url && @idle.size < MAX_IDLE_CONNECTIONS
+          @idle.push([connection, monotonic_now])
+        end
+      end
+      connection.close rescue nil unless kept
+    end
+
+    def monotonic_now
+      Process.clock_gettime(Process::CLOCK_MONOTONIC)
+    end
+  end # EnqueuePool
+end # Backburner