postburner 1.0.0.pre.16 → 1.0.0.pre.18

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8e032f7d37f1680fe34a1402f8eb9f0b3ca61aa9231f6d540c35c3a919e3f58d
-  data.tar.gz: bd35338dd14bc499c4c7d9f57d30eff3902615cd72876f766f1e379f12baba77
+  metadata.gz: 4af263bde4adb664e58df7f7045152e0fee7c29c953227aeb962b385b68de21d
+  data.tar.gz: 976c3a47d49bf83c26719e7b8adf17d05c827c0cd982855c5258307d703c4acf
 SHA512:
-  metadata.gz: f207c1d4dd840bee6b00ad71de9cc1b774d9b2376f7f34125fcc3a164220a340badc6b1ef17931d3dc09a0bb78a09c66b1f1f9ffdf1ead3e9737c44a39fe9a29
-  data.tar.gz: e0212499c62223a87720462226660045585ac4bde21258eb54aeed9d85683c0c9652477a6cf5824bdffa7e30314efb1e1cbb9d146eae8ef3ade2202af73f2abe
+  metadata.gz: 30e9379f4c7e05d18659f9864fb58c9e8a0f564575d323a9d3865b352ba3e0d08975bc9a3859d8044602f5c657c16bdd7dd32098944d07b6ad8007124f868a26
+  data.tar.gz: 1c0eb570bf338995bfa385a95c6d8ded11a937fb6155481b199e6a94f5992104d6ebc9524601056397f6251c89841d9148d7d45434e583d2e8f91e1e8f14aa66

data/README.md

@@ -1,12 +1,8 @@
 # Postburner
 
-Fast Beanstalkd-backed job queue with **optional PostgreSQL records**.
+Fast Beanstalkd-backed job queue with **optional PostgreSQL records via ActiveRecord**.
 
-Postburner provides dual-mode job execution:
-- **Fast jobs**: Fast execution via Beanstalkd
-- **Tracked jobs**: Audited jobs logs, timing, errors, and statistics
-
-Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
+Built for the real world where you may want fast background processing for most jobs, but comprehensive auditing for critical operations.
 
 - **ActiveJob Adapter** - To use with Rails, ActionMailer, ActiveStorage
 - **Dual-mode execution** - Beanstalkd only or tracked (database backed)
@@ -104,6 +100,7 @@ bundle exec rake postburner:work WORKER=default
 - [Configuration](#configuration)
 - [Callbacks](#callbacks)
 - [Instrumentation](#instrumentation)
+- [Logging](#logging)
 - [Why Beanstalkd?](#why-beanstalkd)
 - [Beanstalkd Integration](#beanstalkd-integration)
 - [Installation](#installation)
@@ -216,6 +213,38 @@ then queue it!
 | **Set priority** | `.set(priority: 0)` | `job.queue!(priority: 0)` |
 | **Set TTR** | `.set(ttr: 300)` | `job.queue!(ttr: 300)` |
 
+
+| Retries | ActiveJob | Postburner::Job |
+|---------|-----------|-----------------|
+| **Default** | No retries (discarded) | No retries (buried) |
+| **Disable retries** | `discard_on StandardError` | *(default behavior)* |
+| **Enable retries** | `retry_on StandardError` | `max_retries 5` |
+
+- **ActiveJob**: No automatic worker-level retries. Use ActiveJob's `retry_on`/`discard_on` for retry behavior. Failed jobs without retry configuration are discarded.
+- **Postburner::Job**: No automatic retries by default. On failure, the job is buried in Beanstalkd for inspection. Use `max_retries` (1-32) to enable retries with exponential backoff (2^n seconds).
+
+```ruby
+# ActiveJob: Use retry_on for retries
+class MyActiveJob < ApplicationJob
+  retry_on StandardError, wait: :polynomially_longer, attempts: 5
+
+  def perform(args)
+    # ...
+  end
+end
+
+# Postburner::Job: Use max_retries for retries
+class MyJob < Postburner::Job
+  max_retries 5  # Exponential backoff: 1s, 2s, 4s, 8s, 16s
+end
+
+# Or use a fixed delay
+class MyJob < Postburner::Job
+  max_retries 5
+  retry_delay 10  # 10 seconds between retries
+end
+```
+
 ## Usage
 
 ### Default Jobs
@@ -273,6 +302,7 @@ end
 Jobs without `Postburner::Beanstalkd` use defaults from `config/postburner.yml`:
 - `default_priority: 65536`
 - `default_ttr: 300`
+- `default_queue: default` (if not specified, defaults to `'default'`)
 
 #### Configuring Third-Party Jobs
 
@@ -1249,6 +1279,8 @@ production:  # <- environment config
 # config/postburner.yml
 default: &default
   beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
+  # default_queue: default       # Queue for jobs (defaults to 'default')
+  default_mailer_queue: mailers  # Queue for Postburner::Mailer (defaults to default_queue)
 
 development:  # <- environment config
   <<: *default
@@ -1459,9 +1491,9 @@ Postburner emits ActiveSupport::Notifications events following Rails conventions
 |-------|------|--------------|
 | `perform_start.job.postburner` | Before job execution begins | `:job`, `:beanstalk_job_id` |
 | `perform.job.postburner` | Around job execution (includes duration) | `:job`, `:beanstalk_job_id` |
-| `retry.job.postburner` | When job is retried after error | `:job`, `:beanstalk_job_id`, `:error`, `:wait`, `:attempt` |
-| `retry_stopped.job.postburner` | When tracked job exhausts retries | `:job`, `:beanstalk_job_id`, `:error` |
-| `discard.job.postburner` | When default job exhausts retries | `:job`, `:beanstalk_job_id`, `:error` |
+| `retry.job.postburner` | When Postburner::Job is retried | `:job`, `:beanstalk_job_id`, `:error`, `:wait`, `:attempt` |
+| `retry_stopped.job.postburner` | When Postburner::Job exhausts retries (buried) | `:job`, `:beanstalk_job_id`, `:error` |
+| `discard.job.postburner` | When default ActiveJob fails (discarded) | `:job`, `:beanstalk_job_id`, `:error` |
 | `enqueue.job.postburner` | When job is enqueued for immediate execution | `:job` |
 | `enqueue_at.job.postburner` | When job is enqueued with delay | `:job`, `:scheduled_at` |
 
@@ -1572,6 +1604,46 @@ end
 
 **Note:** These events complement (don't replace) ActiveJob's built-in instrumentation events like `enqueue.active_job` and `perform.active_job`.
 
+## Logging
+
+### Configuration
+
+Set the log level in your Rails environment configuration:
+
+```ruby
+# config/environments/production.rb
+config.log_level = :info  # Default, recommended for production
+
+# config/environments/development.rb
+config.log_level = :debug  # Verbose logging for development
+```
+
+Postburner uses `Rails.logger`, so standard Rails log level configuration applies.
+
+### Debugging
+
+Set the log level to `debug` for verbose logging:
+
+```ruby
+config.log_level = :debug
+```
+
+### Custom Job Logging
+
+```ruby
+class ProcessPayment < ApplicationJob
+  include Postburner::Tracked
+
+  def perform(payment_id)
+    log "Starting payment processing for $#{payment.amount}"  # Stored in database
+    payment.charge!
+    log! "Payment charged successfully"  # Saved immediately
+  end
+end
+```
+
+These job-specific logs are stored in the database (for tracked jobs only) and are separate from Rails application logs, providing a complete audit trail for critical operations.
+
 ## Why Beanstalkd?
 
 Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
@@ -1696,7 +1768,7 @@ class BackgroundTask < Postburner::Job
 end
 ```
 
-**Recommended Priority Ranges:**
+**Example Priority Ranges:**
 
 | Priority Range | Use Case | Examples |
 |---------------|----------|----------|
@@ -1712,7 +1784,9 @@ Set default priority in `config/postburner.yml`:
 
 ```yaml
 production:
-  default_priority: 65536  # Default for jobs without explicit priority
+  default_queue: default           # Queue for jobs (defaults to 'default')
+  default_mailer_queue: mailers    # Queue for Postburner::Mailer (defaults to default_queue)
+  default_priority: 65536          # Default without explicit priority set
   default_ttr: 300
 ```
 

data/app/concerns/postburner/execution.rb

@@ -162,7 +162,7 @@ module Postburner
 
         rescue Exception => exception
           self.log_exception!(exception)
-          raise exception
+          handle_retry_or_raise(exception)
         end
       end # run_callbacks :attempt
 
@@ -170,6 +170,43 @@ module Postburner
 
     private
 
+    # Handles retry logic or re-raises exception.
+    #
+    # If job has retries configured and hasn't exceeded max_retries, requeues
+    # the job with the configured delay. Otherwise, re-raises the exception
+    # for the worker to handle (typically by burying).
+    #
+    # Instruments with ActiveSupport::Notifications:
+    # - retry.job.postburner: When job is requeued for retry
+    #
+    # @param exception [Exception] The exception that caused the failure
+    #
+    # @raise [Exception] Re-raises if no retries configured or max exceeded
+    #
+    # @api private
+    #
+    def handle_retry_or_raise(exception)
+      if should_retry?
+        delay = retry_delay_for_attempt(attempt_count - 1)
+
+        self.log!("RETRY: attempt #{attempt_count}/#{self.class.max_retries}, delay #{delay}s")
+
+        # Instrument retry event
+        job_payload = Postburner::Instrumentation.job_payload_from_model(self, beanstalk_job_id: self.bkid)
+        ActiveSupport::Notifications.instrument('retry.job.postburner', {
+          job: job_payload,
+          beanstalk_job_id: self.bkid,
+          error: exception,
+          wait: delay,
+          attempt: attempt_count
+        })
+
+        requeue!(delay: delay)
+      else
+        raise exception
+      end
+    end
+
     # Records an attempt and calculates execution lag.
     #
     # Appends current time to attempts array, sets attempting_at on first attempt,

data/app/concerns/postburner/properties.rb

@@ -6,6 +6,26 @@ module Postburner
   # Provides DSL methods for configuring queue behavior (name, priority, TTR, retries).
   # Defines configurable properties for job queue management.
   #
+  # == Retry Behavior
+  #
+  # By default, Postburner::Job does NOT retry failed jobs. When a job raises an
+  # exception, it is buried in Beanstalkd for inspection. This differs from default
+  # ActiveJob behavior (5 retries with 2^n second backoff, ~31s total).
+  #
+  # Use +max_retries+ to enable automatic retries:
+  #
+  #   class MyJob < Postburner::Job
+  #     max_retries 5  # Retry up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s)
+  #   end
+  #
+  # The default +retry_delay+ is exponential backoff (2^n seconds), matching
+  # Postburner's default ActiveJob behavior. Override with a fixed delay or custom proc:
+  #
+  #   retry_delay 10  # Fixed 10 second delay between retries
+  #
+  #   # Match ActiveJob's :exponentially_longer (polynomial: (n+1)^4 + 2)
+  #   retry_delay ->(n) { ((n + 1) ** 4) + 2 }  # 3s, 18s, 83s, 258s, 627s
+  #
   # @example Basic usage
   #   class ProcessPayment < Postburner::Job
   #     queue 'critical'
@@ -25,7 +45,9 @@ module Postburner
       # Instance-level queue configuration (overrides class-level defaults)
       attr_writer :priority, :ttr, :queue_name
 
-      class_attribute :postburner_queue_name, default: 'default'
+      # Class-level overrides (all default to nil).
+      # Global defaults are defined in Postburner::Configuration (lib/postburner/configuration.rb).
+      class_attribute :postburner_queue_name, default: nil
       class_attribute :postburner_priority, default: nil
       class_attribute :postburner_ttr, default: nil
       class_attribute :postburner_max_retries, default: nil
@@ -106,7 +128,13 @@ module Postburner
 
       # Sets or returns maximum number of job retries.
       #
-      # @param retries [Integer, nil] Max retries, or nil to get current value
+      # By default, Postburner::Job does NOT retry (max_retries is nil).
+      # Failed jobs are buried in Beanstalkd for inspection.
+      # Set max_retries to enable automatic retries on failure.
+      #
+      # Values of nil or 0 disable retries. Maximum allowed value is 32.
+      #
+      # @param retries [Integer, nil] Max retries (0-32), or nil to get current value
       #
       # @return [Integer, nil] Current max retries when getting, nil when setting
       #
@@ -116,9 +144,11 @@ module Postburner
       # @example Get max retries
       #   ProcessPayment.max_retries  # => 3
       #
+      # @see retry_delay
+      #
       def max_retries(retries = nil)
         if retries
-          self.postburner_max_retries = retries
+          self.postburner_max_retries = [[retries.to_i, 0].max, 32].min
           nil
         else
           postburner_max_retries
@@ -172,10 +202,10 @@ module Postburner
     #   job.queue_name  # => 'urgent'
     #
     def queue_name
-      @queue_name || self.class.queue
+      @queue_name || self.class.queue || Postburner.configuration.default_queue
     end
 
-    # Returns the full tube name with environment prefix.
+    # Returns the full Beanstalkd tube name with environment prefix.
     #
     # Expands the queue name to include the environment prefix
     # (e.g., 'critical' becomes 'postburner.development.critical').
@@ -183,12 +213,18 @@ module Postburner
     # @return [String] Full tube name with environment prefix
     #
     # @example
-    #   job.expanded_tube_name  # => 'postburner.development.critical'
+    #   job.tube_name  # => 'postburner.development.critical'
     #
-    def expanded_tube_name
+    def tube_name
       Postburner.configuration.expand_tube_name(queue_name)
     end
 
+    # @!method expanded_tube_name
+    #   Alias for {#tube_name}.
+    #   @return [String] Full tube name with environment prefix
+    #   @see #tube_name
+    alias_method :expanded_tube_name, :tube_name
+
     # Returns the priority for this job instance.
     #
     # Checks instance-level override first, then falls back to class-level configuration.
@@ -222,5 +258,67 @@ module Postburner
       @ttr || self.class.ttr
     end
 
+    # Checks if this job should retry after a failure.
+    #
+    # Returns true if max_retries is configured and the current attempt count
+    # is less than max_retries.
+    #
+    # @return [Boolean] true if job should retry, false otherwise
+    #
+    # @example
+    #   class MyJob < Postburner::Job
+    #     max_retries 3
+    #   end
+    #
+    #   job = MyJob.create!(args: {})
+    #   job.should_retry?  # => true (attempt_count is 0)
+    #   # After 3 failed attempts...
+    #   job.should_retry?  # => false
+    #
+    def should_retry?
+      max = self.class.max_retries || Postburner.configuration.default_max_retries
+      attempt_count.to_i < max.to_i
+    end
+
+    # Calculates the retry delay for the given attempt.
+    #
+    # Uses the class-level retry_delay configuration, which can be:
+    # - Integer: Fixed delay in seconds
+    # - Proc: Called with attempt number (0-based), returns delay in seconds
+    # - nil: Defaults to 5 seconds
+    #
+    # @param attempt [Integer] The attempt number (0-based: 0 for first retry)
+    #
+    # @return [Integer] Delay in seconds before next retry
+    #
+    # @example Fixed delay
+    #   class MyJob < Postburner::Job
+    #     max_retries 3
+    #     retry_delay 10
+    #   end
+    #   job.retry_delay_for_attempt(0)  # => 10
+    #
+    # @example Exponential backoff (2s, 4s, 8s...)
+    #   class MyJob < Postburner::Job
+    #     max_retries 5
+    #     retry_delay ->(n) { 2 ** (n + 1) }
+    #   end
+    #   job.retry_delay_for_attempt(0)  # => 2
+    #   job.retry_delay_for_attempt(1)  # => 4
+    #   job.retry_delay_for_attempt(2)  # => 8
+    #
+    def retry_delay_for_attempt(attempt)
+      delay_config = self.class.retry_delay || Postburner.configuration.default_retry_delay
+
+      case delay_config
+      when Proc
+        delay_config.call(attempt).to_i
+      when Integer
+        delay_config
+      else
+        2 ** attempt # Fallback to default exponential backoff
+      end
+    end
+
   end
 end

data/app/concerns/postburner/statistics.rb

@@ -73,13 +73,13 @@ module Postburner
     #
     def stats
       # Get configured watched tubes (expanded with environment prefix)
-      watched = Postburner.watched_tube_names.include?(self.expanded_tube_name)
+      watched = Postburner.watched_tube_names.include?(self.tube_name)
 
       {
         id: self.id,
         bkid: self.bkid,
         queue: queue_name,
-        tube: expanded_tube_name,
+        tube: tube_name,
         watched: watched,
         beanstalk: self.bk.stats.to_h.symbolize_keys,
       }

data/app/models/postburner/mailer.rb

@@ -1,12 +1,90 @@
 module Postburner
-  # Send a mailer, tracked.
+  # A tracked job for sending ActionMailer emails via Beanstalkd.
   #
-  # Postburner::Mailer.deliver(UserMailer, :welcome).with(user_id: 1)
-  # Postburner::Mailer.deliver(UserMailer, :welcome).with(user_id: 1).queue! at: Time.current + 1.day
-  # Postburner::Mailer.deliver(UserMailer, :welcome).with(user_id: 1).queue! delay: 5.minutes
+  # Provides a chainable API similar to ActionMailer for building and
+  # queueing email deliveries with full audit trail support.
+  #
+  # == Queue Configuration
+  #
+  # By default, mailer jobs use the +default_queue+ (typically 'default').
+  # You can route mailers to a dedicated queue using several approaches:
+  #
+  # === Via configuration (recommended)
+  #
+  # Set +default_mailer_queue+ in your postburner.yml or configuration:
+  #
+  #   # config/postburner.yml
+  #   production:
+  #     default_mailer_queue: mailers
+  #
+  #   # Or programmatically
+  #   Postburner.configure do |config|
+  #     config.default_mailer_queue = 'mailers'
+  #   end
+  #
+  # === Via queue! option (per-job override)
+  #
+  # Override the queue when queueing a specific job:
+  #
+  #   Postburner::Mailer.delivery(UserMailer, :welcome)
+  #     .with(user_id: 1)
+  #     .queue!(queue: 'priority_mailers')
+  #
+  # === Via instance setter (per-job override)
+  #
+  # Set the queue on the job instance before queueing:
+  #
+  #   job = Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1)
+  #   job.queue_name = 'priority_mailers'
+  #   job.queue!
+  #
+  # @example Basic delivery (queued immediately)
+  #   Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1).queue!
+  #
+  # @example Scheduled delivery at a specific time
+  #   Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1).queue!(at: Time.current + 1.day)
+  #
+  # @example Delayed delivery
+  #   Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1).queue!(delay: 5.minutes)
+  #
+  # @example Queue to specific queue
+  #   Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1).queue!(queue: 'bulk_mailers')
+  #
+  # @see Postburner::Job Base class for tracked jobs
+  # @see Postburner::Configuration#default_mailer_queue Configuration option
   class Mailer < Job
-    #queue 'mailers'
 
+    # Returns the queue name for this mailer job.
+    #
+    # Checks in order: instance override, +default_mailer_queue+ config,
+    # then falls back to +default_queue+.
+    #
+    # @return [String] Queue name
+    #
+    # @example Default behavior (uses default_mailer_queue or default_queue)
+    #   job = Postburner::Mailer.delivery(UserMailer, :welcome)
+    #   job.queue_name  # => 'mailers' (if configured) or 'default'
+    #
+    # @example Instance override
+    #   job = Postburner::Mailer.delivery(UserMailer, :welcome)
+    #   job.queue_name = 'priority_mailers'
+    #   job.queue_name  # => 'priority_mailers'
+    def queue_name
+      @queue_name ||
+        self.class.queue ||
+        Postburner.configuration.default_mailer_queue ||
+        Postburner.configuration.default_queue
+    end
+
+    # Build a new mailer job without persisting.
+    #
+    # @param mailer [Class] The ActionMailer class (e.g., UserMailer)
+    # @param action [Symbol, String] The mailer action/method name (e.g., :welcome)
+    # @return [Postburner::Mailer] An unsaved mailer job instance
+    #
+    # @example
+    #   job = Postburner::Mailer.delivery(UserMailer, :welcome)
+    #   job.with(user_id: 1).queue!
     def self.delivery(mailer, action)
       job = self.new(
         args: {
@@ -17,14 +95,33 @@ module Postburner
       job
     end
 
+    # Build and persist a new mailer job.
+    #
+    # @param mailer [Class] The ActionMailer class (e.g., UserMailer)
+    # @param action [Symbol, String] The mailer action/method name (e.g., :welcome)
+    # @return [Postburner::Mailer] A persisted mailer job instance
+    #
+    # @example
+    #   job = Postburner::Mailer.delivery!(UserMailer, :password_reset)
+    #   job.with!(token: reset_token)
     def self.delivery!(mailer, action)
       job = self.delivery(mailer, action)
       job.save!
       job
     end
 
-    # Similar to ActionMailer #with - set the parameters
+    # Set the mailer parameters without persisting.
+    #
+    # Similar to ActionMailer's +#with+ method for parameterized mailers.
+    # Parameters are serialized using ActiveJob::Arguments for safe storage.
+    #
+    # @param params [Hash] Parameters to pass to the mailer
+    # @return [self] Returns self for method chaining
     #
+    # @example
+    #   Postburner::Mailer.delivery(UserMailer, :welcome)
+    #     .with(user_id: 1, locale: :en)
+    #     .queue!
     def with(params={})
       self.args.merge!(
         'params' => ::ActiveJob::Arguments.serialize(params)
@@ -32,16 +129,31 @@ module Postburner
       self
     end
 
+    # Set the mailer parameters and persist immediately.
+    #
+    # @param params [Hash] Parameters to pass to the mailer
+    # @return [self] Returns self for method chaining
+    #
+    # @example
+    #   job = Postburner::Mailer.delivery!(OrderMailer, :confirmation)
+    #   job.with!(order_id: order.id)
     def with!(params={})
       self.with(params)
       self.save!
       self
     end
 
-    # Build the mail but don't send.
+    # Build the Mail::Message without sending.
+    #
+    # Useful for testing or inspecting the email before delivery.
     #
-    # Optional `args` argument for testing convenience.
+    # @return [Mail::Message] The assembled email message
     #
+    # @example Inspect email before sending
+    #   job = Postburner::Mailer.delivery(UserMailer, :welcome).with(user_id: 1)
+    #   mail = job.assemble
+    #   puts mail.subject  # => "Welcome!"
+    #   puts mail.to       # => ["user@example.com"]
     def assemble
       mail = self.mailer.with(self.params).send(self.action)
       mail
@@ -49,22 +161,43 @@ module Postburner
 
     # Get the mailer class.
     #
+    # @return [Class] The ActionMailer class
+    #
+    # @example
+    #   job.mailer  # => UserMailer
     def mailer
       self.args['mailer'].constantize
     end
 
-    # Get the mailer action as a symbol.
+    # Get the mailer action name.
+    #
+    # @return [Symbol, nil] The mailer action as a symbol
     #
+    # @example
+    #   job.action  # => :welcome
     def action
       self.args['action']&.to_sym
     end
 
-    # Get the deserialized params.
+    # Get the deserialized mailer parameters.
     #
+    # @return [Hash] The parameters passed via {#with}
+    #
+    # @example
+    #   job.params  # => { user_id: 1, locale: :en }
     def params
       ::ActiveJob::Arguments.deserialize(self.args['params']).to_h
     end
 
+    # Execute the mailer job.
+    #
+    # Called by the Postburner worker. Assembles and delivers the email,
+    # logging progress at each step.
+    #
+    # @param args [Hash] Job arguments (unused, params stored in {#args})
+    # @return [void]
+    #
+    # @api private
     def perform(args)
       self.log! "Building"
       mail = self.assemble

data/lib/postburner/configuration.rb

@@ -22,6 +22,7 @@ module Postburner
   class Configuration
     # Global settings
     attr_accessor :beanstalk_url, :logger, :default_priority, :default_ttr
+    attr_accessor :default_queue, :default_mailer_queue, :default_max_retries, :default_retry_delay
     attr_accessor :default_scheduler_interval, :default_scheduler_priority
     attr_accessor :enqueue_options
 
@@ -33,6 +34,7 @@ module Postburner
     # @option options [Logger] :logger Logger instance (default: Rails.logger)
     # @option options [Integer] :default_priority Default job priority (default: 65536, lower = higher priority)
     # @option options [Integer] :default_ttr Default time-to-run in seconds (default: 300)
+    # @option options [String] :default_mailer_queue Queue name for Postburner::Mailer jobs (default: nil, uses default_queue)
     # @option options [Integer] :default_scheduler_interval Scheduler check interval in seconds (default: 300)
     # @option options [Integer] :default_scheduler_priority Scheduler job priority (default: 100)
     # @option options [Proc] :enqueue_options Proc that receives job and returns options hash
@@ -52,6 +54,10 @@ module Postburner
       @logger = options[:logger] || (defined?(Rails) ? Rails.logger : Logger.new(STDOUT))
       @default_priority = options[:default_priority] || 65536
       @default_ttr = options[:default_ttr] || 300
+      @default_queue = options[:default_queue] || 'default'
+      @default_mailer_queue = options[:default_mailer_queue]
+      @default_max_retries = options[:default_max_retries] || 0
+      @default_retry_delay = options[:default_retry_delay] || ->(n) { 2 ** n }
       @default_scheduler_interval = options[:default_scheduler_interval] || 300
       @default_scheduler_priority = options[:default_scheduler_priority] || 100
       @enqueue_options = options[:enqueue_options]
@@ -93,6 +99,7 @@ module Postburner
     #   default: &default
     #     beanstalk_url: <%= ENV.fetch('BEANSTALK_URL', 'beanstalk://localhost:11300') %>
     #     default_priority: 131072 # change default priority from 65536 to 131072
+    #     default_mailer_queue: mailers  # optional: route Postburner::Mailer to separate queue
     #
     #   production: # <- environment config, i.e. defaults
     #     <<: *default
@@ -162,6 +169,8 @@ module Postburner
         beanstalk_url: env_config['beanstalk_url'],
         default_priority: env_config['default_priority'],
         default_ttr: env_config['default_ttr'],
+        default_queue: env_config['default_queue'],
+        default_mailer_queue: env_config['default_mailer_queue'],
         default_scheduler_interval: env_config['scheduler_interval'],
         default_scheduler_priority: env_config['scheduler_priority'],
         worker_config: worker_config

data/lib/postburner/strategies/strict_queue.rb

@@ -76,7 +76,7 @@ module Postburner
         #debugger
         Postburner::Job.transaction do
           Postburner.connected do |conn|
-            tube_name = job.expanded_tube_name
+            tube_name = job.tube_name
             data = { class: job.class.name, args: [job.id] }
 
             # Get priority, TTR from job instance (respects instance overrides) or options

data/lib/postburner/tube.rb

@@ -1,35 +1,96 @@
 module Postburner
+  # A wrapper around Beaneater tubes for inspecting Beanstalkd queues.
+  #
+  # Provides methods to enumerate tubes and paginate through jobs,
+  # primarily used by the Postburner web UI for job inspection.
+  #
+  # @note Beanstalkd is not designed for job enumeration or searching.
+  #   It is optimized for fast job reservation, not inspection. This class
+  #   iterates through job IDs sequentially which is inherently inefficient.
+  #   Use this only for debugging and development purposes, not for
+  #   production workloads or high-volume introspection of queues.
+  #
+  # @example List all tubes
+  #   tubes = Postburner::Tube.all
+  #   tubes.each { |tube| puts tube.peek_ids }
+  #
+  # @example Paginate through jobs in a tube
+  #   tube = Postburner::Tube.all.first
+  #   first_page = tube.jobs(20)
+  #   second_page = tube.jobs(20, after: first_page.last.id)
+  #
+  # @see Postburner.connection For accessing the Beaneater connection
   class Tube
+    # Initialize a new Tube wrapper.
+    #
+    # @param tube [Beaneater::Tube] The underlying Beaneater tube object
     def initialize(tube)
       @tube = tube
     end
 
     # Get all tubes as Postburner::Tube instances.
     #
+    # @return [Array<Postburner::Tube>] All tubes from the Beanstalkd server
+    #
+    # @example
+    #   Postburner::Tube.all
+    #   # => [#<Postburner::Tube>, #<Postburner::Tube>, ...]
     def self.all
       Postburner.connection.tubes.all.map { |tube| self.new(tube) }
     end
 
-    # Get all peeked ids across all known tubes.
+    # Get all peeked job IDs across all known tubes.
+    #
+    # Useful for finding the minimum known job ID when starting pagination.
+    #
+    # @return [Array<Integer>] Sorted array of job IDs from all tubes
     #
+    # @example
+    #   Postburner::Tube.peek_ids
+    #   # => [1, 5, 12, 47, 89, 102]
     def self.peek_ids
       self.all.map(&:peek_ids).flatten.sort
     end
 
-    # Get all peeked ids.
+    # Get peeked job IDs from this tube.
     #
+    # Peeks at buried, ready, and delayed states to find known job IDs.
+    #
+    # @return [Array<Integer>] Job IDs visible via peek operations
+    #
+    # @example
+    #   tube = Postburner::Tube.all.first
+    #   ids = tube.peek_ids  # => [1, 5, 12]
     def peek_ids
       [ :buried, :ready, :delayed ].map { |type| @tube.peek(type) }.
         reject(&:nil?).map(&:id).map(&:to_i)
     end
 
-    # Get paginated array of jobs.
+    # Get a paginated array of jobs from this tube.
+    #
+    # Efficiently retrieves jobs by starting from known IDs and iterating
+    # until the requested count is fulfilled. Supports cursor-based pagination
+    # via the +after+ parameter.
+    #
+    # @param count [Integer] Number of jobs to return (default: 20)
+    # @param limit [Integer] Maximum ID range to scan (default: 1000)
+    # @param after [Integer, nil] Return jobs after this ID for pagination
+    # @return [Array<Beaneater::Job>] Jobs from this tube
     #
-    # Attempts to do this efficiently as possible, by peeking at known
-    # ids and exiting when count has been fulfilled.
+    # @example Get first page of jobs
+    #   tube.jobs(20)
+    #   # => [#<Beaneater::Job id=1>, #<Beaneater::Job id=5>, ...]
     #
-    # Just pass the last known id to after for the next batch.
+    # @example Access job properties
+    #   jobs = tube.jobs(5)
+    #   jobs.first.id      # => 1
+    #   jobs.first.body    # => "{\"job_class\":\"MyJob\",...}"
+    #   jobs.first.stats   # => {"tube"=>"default", "state"=>"ready", ...}
     #
+    # @example Paginate through jobs
+    #   page1 = tube.jobs(20)
+    #   page2 = tube.jobs(20, after: page1.last.id)
+    #   # => [#<Beaneater::Job id=25>, #<Beaneater::Job id=26>, ...]
     def jobs(count=20, limit: 1000, after: nil)
       # Note: beaneater transforms hyphenated beanstalkd stats to underscores
       stats = @tube.stats

data/lib/postburner/version.rb

@@ -1,3 +1,3 @@
 module Postburner
-  VERSION = '1.0.0.pre.16'
+  VERSION = '1.0.0.pre.18'
 end

data/lib/postburner/worker.rb

@@ -136,6 +136,28 @@ module Postburner
       @shutdown
     end
 
+    # Checks if this process has been orphaned (parent died).
+    #
+    # When the parent process dies, the kernel re-parents children to init (PID 1).
+    # Detecting this allows forked children to exit gracefully instead of running
+    # indefinitely as orphans.
+    #
+    # @return [Boolean] true if parent PID is 1, false otherwise
+    def orphaned?
+      Process.ppid == 1
+    end
+
+    # Calculates exponential backoff sleep duration for reconnection attempts.
+    #
+    # Uses exponential backoff starting at 1 second and doubling each attempt,
+    # capped at 32 seconds to prevent excessively long waits.
+    #
+    # @param attempts [Integer] Number of consecutive failed attempts (0-based)
+    # @return [Integer] Sleep duration in seconds (1, 2, 4, 8, 16, or 32)
+    def reconnect_backoff(attempts)
+      [2**attempts, 32].min
+    end
+
     private
 
     # Returns the worker configuration hash.
@@ -148,13 +170,17 @@ module Postburner
 
     # Sets up signal handlers for graceful shutdown.
     #
-    # Traps TERM and INT signals to initiate graceful shutdown.
+    # Trapped signals:
+    # - TERM: Graceful termination request (systemd, kill, process managers)
+    # - INT: Interrupt from keyboard (Ctrl+C)
+    # - HUP: Hangup signal when controlling terminal dies (prevents orphaned children)
     #
     # @return [void]
     # @api private
     def setup_signal_handlers
       Signal.trap('TERM') { shutdown }
       Signal.trap('INT') { shutdown }
+      Signal.trap('HUP') { shutdown }
     end
 
     # Expands queue name to full tube name with environment prefix.
@@ -245,17 +271,19 @@ module Postburner
     def process_jobs
       connection = Postburner::Connection.new
       timeout = worker_config[:timeout]
+      reconnect_attempts = 0
 
       watch_queues(connection, config.queue_names)
 
       until shutdown? || (@gc_limit && @jobs_processed.value >= @gc_limit)
         begin
-          job = connection.beanstalk.tubes.reserve(timeout: timeout)
+          job = connection.beanstalk.tubes.reserve(timeout)
 
           if job
             logger.debug "[Postburner::Worker] Thread #{Thread.current.object_id} reserved job #{job.id}"
             execute_job(job)
             @jobs_processed.increment
+            reconnect_attempts = 0  # Reset backoff on successful job execution
           else
             ensure_scheduler_watchdog!(connection)
           end
@@ -263,8 +291,10 @@ module Postburner
           ensure_scheduler_watchdog!(connection)
           next
         rescue Beaneater::NotConnected => e
-          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}"
-          sleep 1
+          backoff = reconnect_backoff(reconnect_attempts)
+          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}, reconnecting in #{backoff}s (attempt #{reconnect_attempts + 1})"
+          sleep backoff
+          reconnect_attempts += 1
           connection.reconnect!
           watch_queues(connection, config.queue_names)
         rescue => e
@@ -359,6 +389,11 @@ module Postburner
       end
 
       until shutdown? || (gc_limit && jobs_processed.value >= gc_limit)
+        if orphaned?
+          logger.error "[Postburner::Worker] Fork #{fork_num} detected parent died (orphaned), initiating shutdown"
+          shutdown
+          break
+        end
         sleep 0.5
       end
 
@@ -391,17 +426,19 @@ module Postburner
     def process_jobs_in_fork(fork_num, jobs_processed, gc_limit)
       connection = Postburner::Connection.new
       timeout = worker_config[:timeout]
+      reconnect_attempts = 0
 
       watch_queues(connection, config.queue_names)
 
       until shutdown? || (gc_limit && jobs_processed.value >= gc_limit)
         begin
-          job = connection.beanstalk.tubes.reserve(timeout: timeout)
+          job = connection.beanstalk.tubes.reserve(timeout)
 
           if job
             logger.debug "[Postburner::Worker] Fork #{fork_num} thread #{Thread.current.object_id} reserved job #{job.id}"
             execute_job(job)
             jobs_processed.increment
+            reconnect_attempts = 0  # Reset backoff on successful job execution
           else
             ensure_scheduler_watchdog!(connection)
           end
@@ -409,8 +446,10 @@ module Postburner
           ensure_scheduler_watchdog!(connection)
           next
         rescue Beaneater::NotConnected => e
-          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}"
-          sleep 1
+          backoff = reconnect_backoff(reconnect_attempts)
+          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}, reconnecting in #{backoff}s (attempt #{reconnect_attempts + 1})"
+          sleep backoff
+          reconnect_attempts += 1
           connection.reconnect!
           watch_queues(connection, config.queue_names)
         rescue => e
@@ -549,13 +588,16 @@ module Postburner
 
     # Handles job execution errors with retry logic.
     #
-    # For tracked jobs and legacy Postburner::Job: Buries the job for inspection,
-    # reports to Rails.error, and emits retry_stopped.job.postburner event.
+    # For Postburner::Job: The job handles its own retries in perform!. If an
+    # exception bubbles up here, it means no retries configured or max exceeded.
+    # Buries the job for inspection, reports to Rails.error, and emits
+    # retry_stopped.job.postburner event.
+    #
     # For default ActiveJob: Applies exponential backoff retry with max 5 attempts,
     # reporting to Rails.error only on final discard.
     #
     # Instruments with ActiveSupport::Notifications:
-    # - retry_stopped.job.postburner: When tracked job is buried after failure
+    # - retry_stopped.job.postburner: When Postburner::Job is buried after failure
     #
     # @param beanstalk_job [Beaneater::Job] Failed job
     # @param error [Exception] The error that caused the failure
@@ -563,39 +605,14 @@ module Postburner
     # @api private
     def handle_error(beanstalk_job, error)
       logger.error "[Postburner] Job failed: #{error.class} - #{error.message}"
-      logger.error error.backtrace.join("\n")
 
       begin
         payload = JSON.parse(beanstalk_job.body)
 
         if payload['tracked'] || Postburner::ActiveJob::Payload.legacy_format?(payload)
-          # Extract job ID from payload
-          job_id = if Postburner::ActiveJob::Payload.legacy_format?(payload)
-                     payload['args']&.first
-                   else
-                     payload['postburner_job_id']
-                   end
-
-          logger.info "[Postburner] Burying tracked/legacy job for inspection (bkid: #{beanstalk_job.id}, job_id: #{job_id})"
-
-          # Report to Rails error reporter for integration with error tracking services
-          Rails.error.report(error, handled: false, context: {
-            job_class: payload['job_class'] || payload['class'],
-            job_id: job_id,
-            beanstalk_job_id: beanstalk_job.id,
-            queue_name: payload['queue_name']
-          })
-
-          job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
-          ActiveSupport::Notifications.instrument('retry_stopped.job.postburner', {
-            job: job_payload,
-            beanstalk_job_id: beanstalk_job.id,
-            error: error
-          })
-
-          beanstalk_job.bury
+          handle_postburner_job_error(beanstalk_job, payload, error)
         else
-          handle_default_retry(beanstalk_job, payload, error)
+          handle_default_job_error(beanstalk_job, payload, error)
         end
       rescue => retry_error
         logger.error "[Postburner] Error handling failure: #{retry_error.message}"
@@ -603,72 +620,85 @@ module Postburner
       end
     end
 
-    # Handles retry logic for default jobs.
+    # Handles errors for Postburner::Job (including tracked ActiveJob).
     #
-    # Applies exponential backoff (2^retry_count seconds, max 1 hour).
-    # After 5 failed attempts, discards the job permanently and reports
-    # to Rails.error for integration with error tracking services.
+    # The job handles its own retries in perform!. If the exception bubbles up
+    # here, it means either no retries are configured or max_retries was exceeded.
+    # Buries the job for inspection.
     #
     # Instruments with ActiveSupport::Notifications:
-    # - retry.job.postburner: When job is retried
-    # - discard.job.postburner: When job is discarded after max retries
+    # - retry_stopped.job.postburner: When job is buried
     #
-    # @param beanstalk_job [Beaneater::Job] Failed job to retry
-    # @param payload [Hash] Parsed job body (modified with retry_count)
+    # @param beanstalk_job [Beaneater::Job] Failed job
+    # @param payload [Hash] Parsed job body
     # @param error [Exception] The error that caused the failure
     # @return [void]
     # @api private
-    def handle_default_retry(beanstalk_job, payload, error)
-      retry_count = payload['retry_count'] || 0
-      max_retries = 5
-      job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
+    def handle_postburner_job_error(beanstalk_job, payload, error)
+      job_id = if Postburner::ActiveJob::Payload.legacy_format?(payload)
+                 payload['args']&.first
+               else
+                 payload['postburner_job_id']
+               end
+
+      job_class_name = payload['job_class'] || payload['class']
 
-      if retry_count < max_retries
-        payload['retry_count'] = retry_count + 1
-        payload['executions'] = (payload['executions'] || 0) + 1
+      # Log the error with backtrace (Postburner::Job doesn't use ActiveJob's logging)
+      logger.error "[Postburner] #{job_class_name}##{job_id} failed: #{error.class} - #{error.message}"
+      logger.error error.backtrace.join("\n")
 
-        delay = [2 ** retry_count, 3600].min
+      logger.info "[Postburner] Burying #{job_class_name}##{job_id} for inspection (bkid: #{beanstalk_job.id})"
 
-        beanstalk_job.delete
+      # Report to Rails error reporter for integration with error tracking services
+      Rails.error.report(error, handled: false, context: {
+        job_class: job_class_name,
+        job_id: job_id,
+        beanstalk_job_id: beanstalk_job.id,
+        queue_name: payload['queue_name']
+      })
 
-        Postburner.connected do |conn|
-          tube_name = expand_tube_name(payload['queue_name'])
-          conn.tubes[tube_name].put(
-            JSON.generate(payload),
-            pri: payload['priority'] || config.default_priority,
-            delay: delay,
-            ttr: payload['ttr'] || config.default_ttr
-          )
-        end
+      job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
+      ActiveSupport::Notifications.instrument('retry_stopped.job.postburner', {
+        job: job_payload,
+        beanstalk_job_id: beanstalk_job.id,
+        error: error
+      })
 
-        ActiveSupport::Notifications.instrument('retry.job.postburner', {
-          job: job_payload,
-          beanstalk_job_id: beanstalk_job.id,
-          error: error,
-          wait: delay,
-          attempt: retry_count + 1
-        })
+      beanstalk_job.bury
+    end
 
-        logger.info "[Postburner] Retrying default job #{payload['job_id']}, attempt #{retry_count + 1} in #{delay}s"
-      else
-        # Report to Rails error reporter for integration with error tracking services
-        Rails.error.report(error, handled: false, context: {
-          job_class: payload['job_class'],
-          job_id: payload['job_id'],
-          beanstalk_job_id: beanstalk_job.id,
-          queue_name: payload['queue_name'],
-          retry_count: retry_count
-        })
-
-        ActiveSupport::Notifications.instrument('discard.job.postburner', {
-          job: job_payload,
-          beanstalk_job_id: beanstalk_job.id,
-          error: error
-        })
-
-        logger.error "[Postburner] Discarding default job #{payload['job_id']} after #{retry_count} retries"
-        beanstalk_job.delete
-      end
+    # Handles errors for default ActiveJob jobs.
+    #
+    # Discards the job and reports to Rails.error. No automatic retries -
+    # use ActiveJob's retry_on/discard_on for retry behavior.
+    #
+    # Instruments with ActiveSupport::Notifications:
+    # - discard.job.postburner: When job is discarded
+    #
+    # @param beanstalk_job [Beaneater::Job] Failed job
+    # @param payload [Hash] Parsed job body
+    # @param error [Exception] The error that caused the failure
+    # @return [void]
+    # @api private
+    def handle_default_job_error(beanstalk_job, payload, error)
+      job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
+
+      # Report to Rails error reporter for integration with error tracking services
+      Rails.error.report(error, handled: false, context: {
+        job_class: payload['job_class'],
+        job_id: payload['job_id'],
+        beanstalk_job_id: beanstalk_job.id,
+        queue_name: payload['queue_name']
+      })
+
+      ActiveSupport::Notifications.instrument('discard.job.postburner', {
+        job: job_payload,
+        beanstalk_job_id: beanstalk_job.id,
+        error: error
+      })
+
+      logger.error "[Postburner] Discarding #{payload['job_class']} (#{payload['job_id']})"
+      beanstalk_job.delete
     end
 
     # Watches all configured queues in Beanstalkd.

data/lib/postburner.rb

@@ -405,6 +405,11 @@ module Postburner
   #   Postburner.clear_jobs!(Postburner.scheduler_tube_name)
   #   # Clears the scheduler watchdog tube
   #
+  # @example Clear watched tubes AND scheduler tube
+  #   Postburner.clear_jobs!(Postburner.watched_tube_names + [Postburner.scheduler_tube_name])
+  #   # Or use the convenience method:
+  #   Postburner.clear_all!
+  #
   # @example Trying to clear unconfigured tube - RAISES ERROR
   #   Postburner.clear_jobs!(['some-other-app-tube'])
   #   # => ArgumentError: Cannot clear tubes not in configuration
@@ -413,6 +418,7 @@ module Postburner
   #   result = Postburner.clear_jobs!(Postburner.watched_tube_names, silent: true)
   #   result[:totals][:total]  # => 42
   #
+  # @see #clear_all!
   # @see Connection#clear_tubes!
   #
   def self.clear_jobs!(tube_names = nil, silent: false)
@@ -431,21 +437,25 @@ module Postburner
   # Clears all configured tubes including scheduler.
   #
   # Convenience method that clears all watched tubes plus the scheduler tube
-  # in a single call. This is the equivalent of:
+  # in a single call. Use this to completely reset Postburner's Beanstalkd state,
+  # such as during test setup or when recovering from a stuck state.
   #
+  # Equivalent to:
   #   Postburner.clear_jobs!(Postburner.watched_tube_names + [Postburner.scheduler_tube_name])
   #
   # @param silent [Boolean] If true, suppress output to stdout (default: false)
   #
   # @return [Hash] Statistics and results (see Connection#clear_tubes!)
   #
-  # @example Clear everything
+  # @example Clear everything (interactive/console use)
   #   Postburner.clear_all!
   #
-  # @example Silent mode
-  #   result = Postburner.clear_all!(silent: true)
+  # @example Silent mode (test setup or programmatic use)
+  #   Postburner.clear_all!(silent: true)
   #
   # @see #clear_jobs!
+  # @see #watched_tube_names
+  # @see #scheduler_tube_name
   #
   def self.clear_all!(silent: false)
     all_tubes = watched_tube_names + [scheduler_tube_name]

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: postburner
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre.16
+  version: 1.0.0.pre.18
 platform: ruby
 authors:
 - Matt Smith