postburner 0.7.2 → 0.9.0.rc.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 31329ed53b5479ef4b3963ce4f6faa7ae103986cb68af6777db9c5dbb46a5b28
-  data.tar.gz: 2c115fcdea67de746f3544684377ff66f8f85ec48241b93a6c8f107e9c5bf052
+  metadata.gz: 5bef243523d66bb1c5d803e826f55fa047c20b7876b804cb82ecb6a82b235a38
+  data.tar.gz: f97a82f504e19e297fcb2f8b99b238b8804fc7e566fba23f082d092c79b88101
 SHA512:
-  metadata.gz: 213abba743144280c99e4b65b0ad54fa26189a3c7f60c82ac9258642ca50803769b297abb53619f89bd88152e43ac6720881178b154ed45771c1fb252eeda99e
-  data.tar.gz: c01eb3f31553159d442cf9c82e984ca084c83b301df918fe2cfedc4d6d2c508d15ae114ff5402361acf9a2bdeefe0db4ed508183e366ed144ae467aad422a0c3
+  metadata.gz: e990e976405963feeb3b5c8f9de98e1a2553f2820f363ef67ffc8c31cf54fa1e6cb6a9e9a8cb53b01ef9f8c644fe0fcc5a13fd7dc2677796401f66f2c7318f93
+  data.tar.gz: c2e829e9f8fc65020cfc080e19d4f26c4d5befa5c5855a73c4225cfaec557144a5237451d129a1e3cae554575bd3053401ead7bcc4c79c524282f6d3dfb4c370

data/CHANGELOG.md

@@ -1,18 +1,5 @@
 # Changelog
 
-## v0.7.1 - 2022-10-30
-
-### Required Changes
-- rename `enqueue` callbacks to `insert`.
-
-### Changed
-- convert to ActiveModel callbacks.
-
-## v0.7.1 - 2022-10-30
-
-### Changed
-- return true from insert_if_queued! and log response to Rails logger.
-
 ## v0.7.0 - 2022-10-24
 
 ### Added

data/README.md

@@ -11,6 +11,27 @@ If you need something faster, check out Que - we love it! Postburner is built fo
 simple, safe, and inspectable. All of which Que has (or can be added), but are included by default with some architecture
 differences. See [Comparison to Que](#comparison-to-que) for more.
 
+## Queue Strategies
+
+Postburner uses different strategies to control how jobs are queued and executed:
+
+| Strategy | When to Use | Behavior | Requires Beanstalkd |
+|----------|-------------|----------|---------------------|
+| **NiceQueue** (default) | Production | Async via Beanstalkd, gracefully re-queues premature jobs | Yes |
+| **Queue** (suggested) | Production | Async via Beanstalkd, raises error on premature execution | Yes |
+| **TestQueue** | Testing with explicit time control | Inline execution, raises error for scheduled jobs | No |
+| **ImmediateTestQueue** | Testing with automatic time travel | Inline execution, auto time-travels for scheduled jobs | No |
+| **NullQueue** | Batch processing / deferred execution | Jobs created but not queued, manual execution via `Postburner::Job.perform(id)` | No |
+
+```ruby
+# Switch strategies
+Postburner.nice_async_strategy!           # Default production (NiceQueue)
+Postburner.async_strategy!                # Strict production (Queue)
+Postburner.inline_test_strategy!          # Testing (TestQueue)
+Postburner.inline_immediate_test_strategy! # Testing with time travel (ImmediateTestQueue)
+Postburner.null_strategy!                 # Deferred execution (NullQueue)
+```
+
 ## Usage
 
 ```ruby
@@ -47,6 +68,57 @@ RunDonation.create!(args: {donation_id: 123}).queue! delay: 1.hour
 => {:status=>"INSERTED", :id=>"1141"}
 ```
 
+### Backburner Queue Configuration
+
+Postburner jobs inherit from `Backburner::Queue`, giving you access to Backburner's queue configuration methods. These are **optional** - Backburner provides sensible defaults.
+
+```ruby
+class CriticalJob < Postburner::Job
+  # Specify which Beanstalkd tube to use (default: 'backburner-jobs')
+  queue 'critical'
+
+  # Set job priority - lower numbers = higher priority (default: 65536)
+  # Most urgent priority is 0
+  queue_priority 0
+
+  # Set maximum retry attempts before burying (default: from Backburner config)
+  queue_max_job_retries 3
+
+  # Set job timeout in seconds (default: from Backburner config)
+  queue_respond_timeout 300
+
+  def perform(args)
+    # your job logic
+  end
+end
+```
+
+**Common configurations:**
+
+```ruby
+# High priority job with no retries
+class PaymentJob < Postburner::Job
+  queue 'payments'
+  queue_priority 0
+  queue_max_job_retries 0  # Don't retry - handle failures manually
+end
+
+# Background job with retries
+class EmailJob < Postburner::Job
+  queue 'emails'
+  queue_priority 1000  # Lower priority
+  queue_max_job_retries 5  # Retry up to 5 times
+end
+
+# Long-running job
+class DataExportJob < Postburner::Job
+  queue 'exports'
+  queue_respond_timeout 3600  # 1 hour timeout
+end
+```
+
+These methods come from [Backburner](https://github.com/nesquena/backburner#configure-backburner) and work seamlessly with Postburner. See Backburner's documentation for all available options.
+
 ### Mailers
 
 ```ruby
@@ -58,31 +130,96 @@ j.queue!
 => {:status=>"INSERTED", :id=>"1139"}
 ```
 
-### [Beaneater](https://github.com/beanstalkd/beaneater) and [beanstalkd](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt) attributes and methods
-```ruby
-# get the beanstalkd job id
-job.bkid
-=> 1104
+### Job Management Methods
 
-# get the beaneater job, call any beaneater methods on this object
-job.beanstalk_job
+Postburner provides several methods for managing jobs in Beanstalkd and the database:
 
-# get the beanstald stats
-job.beanstalk_job.stats
+#### delete! - Remove from Beanstalkd only
 
-# kick the beanstalk job
-job.kick!
+Deletes the job from the Beanstalkd queue but **keeps the database record**. Use this when you want to cancel a queued job while maintaining its audit trail.
 
-# delete beankstalkd job, retain the job model
+```ruby
+job = MyJob.create!(args: {})
+job.queue!(delay: 1.hour)
+job.bkid  # => 12345
+
+# Remove from Beanstalkd but keep in database
 job.delete!
 
-# delete beankstalkd job, retain the job model, but set `removed_at` on model.
+job.reload
+job.bkid  # => 12345 (still has bkid)
+job.persisted?  # => true (still in database)
+# Job won't execute, but you can still inspect it
+```
+
+#### remove! - Soft delete (Beanstalkd + removed_at)
+
+Removes the job from Beanstalkd **and** sets `removed_at` timestamp. The database record is preserved for audit trails. This is the recommended way to cancel jobs.
+
+```ruby
+job = MyJob.create!(args: {})
+job.queue!(delay: 1.hour)
+
+# Soft delete - removes from queue and marks as removed
 job.remove!
 
-# or simply remove the model, which will clean up the beanstalkd job in a before_destroy hook
-job.destroy # OR job.destroy!
+job.reload
+job.removed_at  # => 2025-11-01 12:34:56 UTC
+job.persisted?  # => true (still in database)
+# Job won't execute and is marked as removed
+```
+
+#### destroy / destroy! - Delete from both
+
+Destroys the database record **and** removes from Beanstalkd via `before_destroy` callback. Use when you want to completely eliminate the job.
+
+```ruby
+job = MyJob.create!(args: {})
+job.queue!(delay: 1.hour)
+
+# Completely remove job from database and Beanstalkd
+job.destroy!
+
+# Job no longer exists in database or Beanstalkd
+```
+
+#### kick! - Retry buried jobs
+
+Moves a buried job back into the ready queue. Buried jobs are those that failed too many times or were explicitly buried by Beanstalkd.
+
+```ruby
+# Job failed and was buried
+job.beanstalk_job.stats['state']  # => 'buried'
+
+# Move it back to ready queue for retry
+job.kick!
+
+job.beanstalk_job.stats['state']  # => 'ready'
+```
 
-# get a cached Backburner connection and inspect it (or even use it directly)
+**Summary:**
+- **`delete!`** - Removes from Beanstalkd, keeps full database record (keeps `bkid`)
+- **`remove!`** - Removes from Beanstalkd, marks `removed_at`, keeps database record (**recommended for canceling jobs**)
+- **`destroy!`** - Removes from both Beanstalkd and database (complete deletion)
+- **`kick!`** - Moves buried job back to ready queue (for retrying failed jobs)
+
+### [Beaneater](https://github.com/beanstalkd/beaneater) and [beanstalkd](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt) Integration
+
+Access Beanstalkd directly for advanced queue operations:
+
+```ruby
+# Get the beanstalkd job id
+job.bkid
+=> 1104
+
+# Get the beaneater job object - call any beaneater methods
+job.beanstalk_job
+
+# Get beanstalkd stats for this job
+job.beanstalk_job.stats
+=> {"id"=>1104, "tube"=>"critical", "state"=>"ready", "pri"=>0, ...}
+
+# Get a cached Backburner connection
 c = Postburner.connection
 c.beanstalk.tubes.to_a
 c.beanstalk.tubes.to_a.map{|t| c.tubes[t.name].peek(:buried)}
@@ -91,10 +228,12 @@ c.beanstalk.tubes['ss.development.caching'].peek(:buried).kick
 c.beanstalk.tubes['ss.development.caching'].kick(3)
 c.close
 
-# automatically close
+# Automatically close connection (recommended)
 Postburner.connected do |connection|
   # do stuff with connection
+  connection.tubes['my.tube'].stats
 end
+# Connection automatically closed
 ```
 
 Read about the [beanstalkd protocol](https://raw.githubusercontent.com/beanstalkd/beanstalkd/master/doc/protocol.txt).
@@ -188,6 +327,52 @@ class RunDonation < Postburner::Job
 end
 ```
 
+### Callbacks
+
+Postburner provides ActiveJob-style lifecycle callbacks for job execution:
+
+```ruby
+class ProcessPayment < Postburner::Job
+  queue 'critical'
+
+  before_enqueue :validate_payment
+  after_enqueue :send_confirmation
+
+  before_attempt :log_attempt
+  after_attempt :update_metrics
+
+  before_processing :acquire_lock
+  after_processing :release_lock
+
+  after_processed :send_receipt  # Only runs on success
+
+  def perform(args)
+    payment = Payment.find(args['payment_id'])
+    payment.process!
+  end
+
+  private
+
+  def validate_payment
+    raise "Invalid payment" unless args['payment_id'].present?
+  end
+
+  def send_confirmation
+    PaymentMailer.queued(args['payment_id']).deliver_later
+  end
+
+  def send_receipt
+    PaymentMailer.receipt(args['payment_id']).deliver_later
+  end
+end
+```
+
+**Available callbacks:**
+- `before_enqueue`, `around_enqueue`, `after_enqueue` - Run when `queue!` is called
+- `before_attempt`, `around_attempt`, `after_attempt` - Run on every execution attempt (including retries)
+- `before_processing`, `around_processing`, `after_processing` - Run during job execution
+- `after_processed` - Only runs after successful completion (not on errors)
+
 ### Optionally, mount the engine
 
 ```ruby
@@ -203,6 +388,199 @@ to add your own authentication or changes - or just create your own routes, cont
 [Override the views](https://guides.rubyonrails.org/engines.html#overriding-views) to make them
 prettier - or follow the suggestion above and use your own.
 
+## Testing
+
+Postburner provides multiple testing strategies to make your tests fast and predictable without requiring Beanstalkd. Tests automatically use `TestQueue` when Rails test mode is detected.
+
+### Rails and ActiveJob Test Configuration
+
+Rails provides several configuration options that affect how background jobs behave in test environments:
+
+#### ActiveJob Queue Adapter
+
+The `config.active_job.queue_adapter` setting controls how ActiveJob queues jobs. Rails supports several built-in adapters:
+
+```ruby
+# config/environments/test.rb (or config/application.rb)
+
+# :test adapter (Rails default for test environment)
+# - Jobs are NOT executed automatically
+# - Jobs accumulate in ActiveJob::Base.queue_adapter.enqueued_jobs
+# - You must explicitly call perform_enqueued_jobs to execute them
+config.active_job.queue_adapter = :test
+
+# :inline adapter
+# - Jobs execute immediately in the same process
+# - No queueing, synchronous execution like Postburner's test strategies
+config.active_job.queue_adapter = :inline
+
+# :async adapter
+# - Jobs execute asynchronously in a thread pool
+# - Not recommended for tests due to race conditions
+config.active_job.queue_adapter = :async
+
+# :backburner adapter (for production use with Postburner)
+# - Requires Beanstalkd to be running
+# - Jobs queued to Beanstalkd tubes
+config.active_job.queue_adapter = :backburner
+```
+
+#### ActiveJob Test Helpers
+
+When using the `:test` adapter, Rails provides test helpers in `ActiveJob::TestHelper`:
+
+```ruby
+# test/test_helper.rb
+require 'test_helper'
+
+class MyTest < ActiveSupport::TestCase
+  include ActiveJob::TestHelper
+
+  test "job is enqueued" do
+    assert_enqueued_with(job: MyJob, args: [{user_id: 123}]) do
+      MyJob.perform_later(user_id: 123)
+    end
+  end
+
+  test "job is performed" do
+    perform_enqueued_jobs do
+      MyJob.perform_later(user_id: 123)
+    end
+    # Job has now executed
+  end
+end
+```
+
+**Available helpers:**
+- `perform_enqueued_jobs` - Executes all enqueued jobs
+- `assert_enqueued_jobs` - Asserts number of jobs enqueued
+- `assert_enqueued_with` - Asserts specific job was enqueued with args
+- `assert_no_enqueued_jobs` - Asserts no jobs were enqueued
+- `assert_performed_jobs` - Asserts number of jobs performed
+- `queue_adapter.enqueued_jobs` - Array of enqueued jobs for inspection
+
+#### Other ActiveJob Configuration Options
+
+```ruby
+# config/application.rb or config/environments/test.rb
+
+# Set default queue name for all jobs
+config.active_job.queue_name_prefix = "myapp_#{Rails.env}"
+
+# Skip after_enqueue callbacks if before_enqueue halts
+config.active_job.skip_after_callbacks_if_terminated = true
+
+# Log arguments when jobs are enqueued (useful for debugging)
+config.active_job.log_arguments = true  # default: true
+
+# Set queue priority (if adapter supports it)
+config.active_job.default_queue_name = 'default'
+
+# Configure job serialization
+config.active_job.custom_serializers << MyCustomSerializer
+```
+
+### Postburner Test Modes vs ActiveJob Adapters
+
+**Key Difference:** Postburner jobs (subclasses of `Postburner::Job`) do NOT use ActiveJob's queue adapter. Postburner has its own queue strategies that control execution independently of ActiveJob settings.
+
+| Postburner Strategy | Execution | Use Case | ActiveJob Equivalent |
+|---------------------|-----------|----------|----------------------|
+| `TestQueue` | Inline, raises on scheduled | Explicit time control | `:inline` adapter |
+| `ImmediateTestQueue` | Inline with auto time-travel | Convenience testing | `:inline` adapter + `travel_to` |
+| `NullQueue` | Deferred, manual execution | Batch processing | `:test` adapter (manual `perform_enqueued_jobs`) |
+| `Queue` / `NiceQueue` | Async via Beanstalkd | Production | `:backburner` adapter |
+
+**When to use what:**
+- **ActiveJob with `:test` adapter:** For testing standard ActiveJob classes (mailers, ActiveStorage, etc.)
+- **Postburner test strategies:** For testing `Postburner::Job` subclasses
+- Both can coexist in the same test suite with different behaviors
+
+### Automatic Test Mode Detection
+
+In Rails test environments, Postburner automatically switches to `TestQueue`:
+
+```ruby
+# Happens automatically when:
+# - Rails.env.test? is true
+# - ActiveJob::Base.queue_adapter_name == :test
+
+# Verify you're in test mode
+Postburner.testing?  # => true in tests
+```
+
+**Note:** Even though `TestQueue` is set automatically in Rails test environments, you can override it at any time to use a different test strategy:
+
+```ruby
+# In test_helper.rb or specific tests
+Postburner.inline_immediate_test_strategy!  # Use automatic time travel
+# OR
+Postburner.null_strategy!  # Use deferred execution
+
+# The auto-detection only sets the initial strategy
+# You can change it anytime during tests
+```
+
+### TestQueue: Explicit Time Control (Default for Tests)
+
+The `TestQueue` strategy forces explicit time management, helping catch scheduling bugs:
+
+```ruby
+test "processes job immediately" do
+  job = MyJob.create!(args: { user_id: 123 })
+  job.queue!
+  assert job.reload.processed_at
+end
+
+test "processes scheduled job with time travel" do
+  job = MyJob.create!(args: { user_id: 123 })
+  job.queue!(delay: 2.hours)
+
+  # Must use travel_to for scheduled jobs
+  travel_to(job.run_at) do
+    assert job.reload.processed_at
+  end
+end
+```
+
+### ImmediateTestQueue: Automatic Time Travel
+
+For convenience in integration tests, `ImmediateTestQueue` automatically handles scheduled jobs:
+
+```ruby
+test "scheduled jobs execute immediately" do
+  Postburner.inline_immediate_test_strategy!
+
+  job = SendReminderEmail.create!(args: { user_id: 123 })
+  job.queue!(delay: 24.hours)
+
+  # Automatically time-travels to scheduled time
+  assert job.reload.processed_at
+  assert_emails 1
+end
+```
+
+### NullQueue: Deferred Execution
+
+Create jobs without queueing, then manually execute them:
+
+```ruby
+test "batch processes jobs" do
+  Postburner.null_strategy!
+
+  jobs = 5.times.map do |i|
+    job = ProcessRecord.create!(args: { id: i })
+    job.queue!
+    job
+  end
+
+  # Manually execute when ready
+  jobs.each { |job| Postburner::Job.perform(job.id) }
+
+  assert jobs.all? { |j| j.reload.processed_at }
+end
+```
+
 ## Installation
 
 First [install beanstalkd](https://beanstalkd.github.io/download.html). On Debian-based systems, that goes like this:
@@ -343,5 +721,51 @@ bundle exec backburner
   Where <nac> is an authorized key with push capabilities.
 
 
+## FAQ
+
+### Can I save a Postburner::Job without enqueuing it?
+
+Yes! You can absolutely save a `Postburner::Job` without enqueuing it. Looking at the code, here's how it works:
+
+#### Creating vs. Queueing
+
+When you use `create!` or `save!`, the job is persisted to the database but **not** sent to Beanstalkd:
+
+```ruby
+# Creates job in database but does NOT enqueue
+job = RunDonation.create!(args: {donation_id: 123})
+job.persisted?  # => true
+job.queued_at   # => nil
+job.bkid        # => nil (no Beanstalkd ID)
+```
+
+To actually enqueue it, you must explicitly call `queue!`:
+
+```ruby
+# Now it's enqueued to Beanstalkd
+job.queue!(delay: 1.hour)
+job.queued_at  # => Time.zone.now
+job.bkid       # => 12345 (Beanstalkd job ID)
+```
+
+#### How it Works
+
+The implementation uses a clever pattern (app/models/postburner/job.rb:86-724):
+
+1. **after_save_commit callback** (`insert_if_queued!` at line 86) runs on every save
+2. **Conditional insertion** (line 722) - only inserts if `will_insert?` returns true
+3. **Flag check** (lines 242-244) - `will_insert?` only returns true if `@_insert_options` is set
+4. **Flag set** (line 199) - `@_insert_options` is only set inside the `queue!` method
+
+So the job is safely persisted without queueing until you explicitly call `queue!`.
+
+#### Use Cases
+
+This is useful for:
+- Creating jobs conditionally queued based on business logic
+- Building jobs within transactions that might roll back
+- Testing job creation without running workers
+- Creating foreign key relationships before queueing
+
 ## License
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -3,7 +3,7 @@ require "bundler/setup"
 APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
 load "rails/tasks/engine.rake"
 
-load "rails/tasks/statistics.rake"
+#load "rails/tasks/statistics.rake"
 
 require "bundler/gem_tasks"