postburner 1.0.0.pre.15 → 1.0.0.pre.17

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 971531132068ea4bddfdb5b5a8aaae4c788ea576aae268010fb6cd4dbd2b82f1
-  data.tar.gz: 44053985f2bebd385bc2a17eb1eb655d89de17d77a8c25b4a1210177cefead66
+  metadata.gz: 05d87a7d1949eb64ef1918b4bb0f0cdeb24796097c370f90576b3a158d0be5a7
+  data.tar.gz: 857304a0b925574ab66d6ad215e6cbe3bdba8c109ecb9a34e96e8ab0ca40aac3
 SHA512:
-  metadata.gz: 19c023063d9eb06913fcb59eaaf581a537b8a944b8ab8713b95210475517223a42887f239bd8788cd313919ffe9bfb89c6c9f80c8b24869239b943bfd2c300ad
-  data.tar.gz: 0e0ed766a67ddd9ef02c0b97eff3a25c1e923c79068608a4d31db779ae23187eb85a5b075e8100644faf1c196f0bafaf1a7711e09db8a948b0ab09e9138f9b49
+  metadata.gz: af92d06c7d796c0db2bb766986bc9b27ad46e9958103e11871538e3224b43a054d80fb50b7bb0b5f8d93de1a11331e10b10d26058c219047e15530f6a36a1b9e
+  data.tar.gz: 351f86f49998e5350b7b705a4f77f68a91c29340995dc5531b964bbdb40b6bb55f3c4202459cdc7453429c7e3b7ceaa4ca934628fc8503305f9630eb9c287142

data/README.md

@@ -1,12 +1,8 @@
 # Postburner
 
-Fast Beanstalkd-backed job queue with **optional PostgreSQL records**.
+Fast Beanstalkd-backed job queue with **optional PostgreSQL records via ActiveRecord**.
 
-Postburner provides dual-mode job execution:
-- **Fast jobs**: Fast execution via Beanstalkd
-- **Tracked jobs**: Audited jobs logs, timing, errors, and statistics
-
-Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
+Built for the real world where you may want fast background processing for most jobs, but comprehensive auditing for critical operations.
 
 - **ActiveJob Adapter** - To use with Rails, ActionMailer, ActiveStorage
 - **Dual-mode execution** - Beanstalkd only or tracked (database backed)
@@ -97,12 +93,14 @@ bundle exec rake postburner:work WORKER=default
   - [Postburner::Job](#postburnerjob)
   - [Scheduler](#scheduler)
   - [Job Management](#job-management)
+- [Writing Jobs](#writing-jobs)
 - [Queue Strategies](#queue-strategies)
 - [Testing](#testing)
 - [Workers](#workers)
 - [Configuration](#configuration)
 - [Callbacks](#callbacks)
 - [Instrumentation](#instrumentation)
+- [Logging](#logging)
 - [Why Beanstalkd?](#why-beanstalkd)
 - [Beanstalkd Integration](#beanstalkd-integration)
 - [Installation](#installation)
@@ -177,16 +175,77 @@ config.active_job.queue_adapter = :postburner
 #config.active_job.queue_name_prefix = Postburner.tube_prefix(Rails.env) # i.e. "postburner.#{Rails.env}"
 config.action_mailer.deliver_later_queue_name = 'mailers' # gets prefixed by config.active_job.queue_name_prefix
 
-
 bundle exec postburner           # start with bin/postburner
 bundle exec rake postburner:work # or with rake task
 ```
 
-## Usage
+### Enqueueing Jobs
 
-**Job Idempotency:** Jobs should be designed to be idempotent and safely re-runnable. Like all job queues, Postburner provides at-least-once delivery—in rare errant cases outside of Postburner's control, a job may be executed more than once, i.e. network issues, etc.
+```ruby
+# ActiveJob (standard Rails API)
+SendEmailJob.perform_later(user_id)                           # Enqueue immediately
+SendEmailJob.set(wait: 1.hour).perform_later(user_id)         # Delay by duration
+SendEmailJob.set(wait_until: Date.tomorrow.noon).perform_later(user_id)  # Run at specific time
+SendEmailJob.set(queue: 'critical').perform_later(user_id)    # Override queue
+SendEmailJob.set(priority: 0).perform_later(user_id)          # Override priority
+
+# Postburner::Job (always tracked, full API)
+job = ProcessPayment.create!(args: { 'payment_id' => 123 })
+job.queue!                                    # Enqueue immediately
+job.queue!(delay: 1.hour)                     # Delay by duration
+job.queue!(at: Date.tomorrow.noon)            # Run at specific time
+job.queue!(queue: 'critical')                 # Override queue
+job.queue!(priority: 0, ttr: 600)             # Set priority and TTR
+```
 
-**TTR (Time-to-Run):** If a job exceeds its TTR without completion, Beanstalkd releases it back to the queue while still running—causing duplicate execution. For long-running jobs, call `extend!` periodically to reset the TTR, or set a sufficiently high TTR value. You must include the `Postburner::Beanstalkd` or `Postburner::Tracked` module with `ActiveJob` to use `extend!`.
+### ActiveJob vs Postburner::Job TL;DR
+
+`Postburner::Job` as simple subclass of `ActiveRecord`, so the normal
+`ActiveRecord` API applies! Thus the workflow is to create an instance,
+then queue it!
+
+| Operation | ActiveJob | Postburner::Job |
+|-----------|-----------|-----------------|
+| **Enqueue immediately** | `MyJob.perform_later(args)` | `MyJob.create!(args: {}).queue!` |
+| **Delay** | `.set(wait: 1.hour)` | `job.queue!(delay: 1.hour)` |
+| **Run at** | `.set(wait_until: time)` | `job.queue!(at: time)` |
+| **Set queue** | `.set(queue: 'critical')` | `job.queue!(queue: 'critical')` |
+| **Set priority** | `.set(priority: 0)` | `job.queue!(priority: 0)` |
+| **Set TTR** | `.set(ttr: 300)` | `job.queue!(ttr: 300)` |
+
+
+| Retries | ActiveJob | Postburner::Job |
+|---------|-----------|-----------------|
+| **Default** | No retries (discarded) | No retries (buried) |
+| **Disable retries** | `discard_on StandardError` | *(default behavior)* |
+| **Enable retries** | `retry_on StandardError` | `max_retries 5` |
+
+- **ActiveJob**: No automatic worker-level retries. Use ActiveJob's `retry_on`/`discard_on` for retry behavior. Failed jobs without retry configuration are discarded.
+- **Postburner::Job**: No automatic retries by default. On failure, the job is buried in Beanstalkd for inspection. Use `max_retries` (1-32) to enable retries with exponential backoff (2^n seconds).
+
+```ruby
+# ActiveJob: Use retry_on for retries
+class MyActiveJob < ApplicationJob
+  retry_on StandardError, wait: :polynomially_longer, attempts: 5
+
+  def perform(args)
+    # ...
+  end
+end
+
+# Postburner::Job: Use max_retries for retries
+class MyJob < Postburner::Job
+  max_retries 5  # Exponential backoff: 1s, 2s, 4s, 8s, 16s
+end
+
+# Or use a fixed delay
+class MyJob < Postburner::Job
+  max_retries 5
+  retry_delay 10  # 10 seconds between retries
+end
+```
+
+## Usage
 
 ### Default Jobs
 
@@ -335,7 +394,7 @@ job.duration  # Execution time in milliseconds
 job.lag       # Queue lag in milliseconds
 ```
 
-### Direct Postburner::Job Usage
+### Postburner::Job Usage
 
 Direct `Postburner::Job` subclasses are **always tracked**:
 
@@ -363,6 +422,15 @@ job.queue!(delay: 1.hour)
 job.queue!(at: 2.days.from_now)
 ```
 
+> **Note:** The `args` parameter in `perform(args)` is optional. It's a convenience accessor to `self.args`, which is stored in a JSONB column on the job record. You can omit the parameter and access args directly:
+>
+> ```ruby
+> def perform
+>   payment = Payment.find(self.args['payment_id'])
+>   # ...
+> end
+> ```
+
 #### Instance-Level Queue Configuration
 
 Override queue priority and TTR per job instance for dynamic behavior:
@@ -775,6 +843,14 @@ job.errata  # Array of exceptions with backtraces
 job.attempts # Array of attempt timestamps
 ```
 
+## Writing Jobs
+
+Pay attention to the following when writing jobs:
+
+**Job Idempotency:** Jobs should be designed to be idempotent and safely re-runnable. Like all job queues, Postburner provides at-least-once delivery—in rare errant cases outside of Postburner's control, a job may be executed more than once, i.e. network issues, etc.
+
+**TTR (Time-to-Run):** If a job exceeds its TTR without completion, Beanstalkd releases it back to the queue while still running—causing duplicate execution. For long-running jobs, call `extend!` periodically to reset the TTR, or set a sufficiently high TTR value. You must include the `Postburner::Beanstalkd` or `Postburner::Tracked` module with `ActiveJob` to use `extend!`.
+
 ## Queue Strategies
 
 Postburner uses different strategies to control job execution. These affect `Postburner::Job` subclasses (not ActiveJob classes).
@@ -1412,9 +1488,9 @@ Postburner emits ActiveSupport::Notifications events following Rails conventions
 |-------|------|--------------|
 | `perform_start.job.postburner` | Before job execution begins | `:job`, `:beanstalk_job_id` |
 | `perform.job.postburner` | Around job execution (includes duration) | `:job`, `:beanstalk_job_id` |
-| `retry.job.postburner` | When job is retried after error | `:job`, `:beanstalk_job_id`, `:error`, `:wait`, `:attempt` |
-| `retry_stopped.job.postburner` | When tracked job exhausts retries | `:job`, `:beanstalk_job_id`, `:error` |
-| `discard.job.postburner` | When default job exhausts retries | `:job`, `:beanstalk_job_id`, `:error` |
+| `retry.job.postburner` | When Postburner::Job is retried | `:job`, `:beanstalk_job_id`, `:error`, `:wait`, `:attempt` |
+| `retry_stopped.job.postburner` | When Postburner::Job exhausts retries (buried) | `:job`, `:beanstalk_job_id`, `:error` |
+| `discard.job.postburner` | When default ActiveJob fails (discarded) | `:job`, `:beanstalk_job_id`, `:error` |
 | `enqueue.job.postburner` | When job is enqueued for immediate execution | `:job` |
 | `enqueue_at.job.postburner` | When job is enqueued with delay | `:job`, `:scheduled_at` |
 
@@ -1525,6 +1601,46 @@ end
 
 **Note:** These events complement (don't replace) ActiveJob's built-in instrumentation events like `enqueue.active_job` and `perform.active_job`.
 
+## Logging
+
+### Configuration
+
+Set the log level in your Rails environment configuration:
+
+```ruby
+# config/environments/production.rb
+config.log_level = :info  # Default, recommended for production
+
+# config/environments/development.rb
+config.log_level = :debug  # Verbose logging for development
+```
+
+Postburner uses `Rails.logger`, so standard Rails log level configuration applies.
+
+### Debugging
+
+Set the log level to `debug` for verbose logging:
+
+```ruby
+config.log_level = :debug
+```
+
+### Custom Job Logging
+
+```ruby
+class ProcessPayment < ApplicationJob
+  include Postburner::Tracked
+
+  def perform(payment_id)
+    log "Starting payment processing for $#{payment.amount}"  # Stored in database
+    payment.charge!
+    log! "Payment charged successfully"  # Saved immediately
+  end
+end
+```
+
+These job-specific logs are stored in the database (for tracked jobs only) and are separate from Rails application logs, providing a complete audit trail for critical operations.
+
 ## Why Beanstalkd?
 
 Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
@@ -1649,7 +1765,7 @@ class BackgroundTask < Postburner::Job
 end
 ```
 
-**Recommended Priority Ranges:**
+**Example Priority Ranges:**
 
 | Priority Range | Use Case | Examples |
 |---------------|----------|----------|
@@ -1665,7 +1781,7 @@ Set default priority in `config/postburner.yml`:
 
 ```yaml
 production:
-  default_priority: 65536  # Default for jobs without explicit priority
+  default_priority: 65536  # Default without explicit priority set
   default_ttr: 300
 ```
 

data/app/concerns/postburner/execution.rb

@@ -134,7 +134,7 @@ module Postburner
 
           run_callbacks :processing do
             begin
-              self.perform(args)
+              method(:perform).arity == 0 ? self.perform : self.perform(args)
             rescue Exception => exception
               self.persist_metadata!
               self.log! '[Postburner] Exception raised during perform prevented completion.'
@@ -162,7 +162,7 @@ module Postburner
 
         rescue Exception => exception
           self.log_exception!(exception)
-          raise exception
+          handle_retry_or_raise(exception)
         end
       end # run_callbacks :attempt
 
@@ -170,6 +170,43 @@ module Postburner
 
     private
 
+    # Handles retry logic or re-raises exception.
+    #
+    # If job has retries configured and hasn't exceeded max_retries, requeues
+    # the job with the configured delay. Otherwise, re-raises the exception
+    # for the worker to handle (typically by burying).
+    #
+    # Instruments with ActiveSupport::Notifications:
+    # - retry.job.postburner: When job is requeued for retry
+    #
+    # @param exception [Exception] The exception that caused the failure
+    #
+    # @raise [Exception] Re-raises if no retries configured or max exceeded
+    #
+    # @api private
+    #
+    def handle_retry_or_raise(exception)
+      if should_retry?
+        delay = retry_delay_for_attempt(attempt_count - 1)
+
+        self.log!("RETRY: attempt #{attempt_count}/#{self.class.max_retries}, delay #{delay}s")
+
+        # Instrument retry event
+        job_payload = Postburner::Instrumentation.job_payload_from_model(self, beanstalk_job_id: self.bkid)
+        ActiveSupport::Notifications.instrument('retry.job.postburner', {
+          job: job_payload,
+          beanstalk_job_id: self.bkid,
+          error: exception,
+          wait: delay,
+          attempt: attempt_count
+        })
+
+        requeue!(delay: delay)
+      else
+        raise exception
+      end
+    end
+
     # Records an attempt and calculates execution lag.
     #
     # Appends current time to attempts array, sets attempting_at on first attempt,

data/app/concerns/postburner/insertion.rb

@@ -37,10 +37,12 @@ module Postburner
     # @param options [Hash] Queue options
     # @option options [Time, ActiveSupport::Duration] :at Absolute time to run the job
     # @option options [Integer, ActiveSupport::Duration] :delay Seconds to delay execution
-    # @option options [Integer] :pri Beanstalkd priority (lower = higher priority)
-    # @option options [Integer] :ttr Time-to-run in seconds before job times out
+    # @option options [Integer] :priority Priority (0-4294967295, 0 = HIGHEST), sets instance attribute
+    # @option options [Integer] :pri Beanstalkd priority (pass-through, for backwards compatibility)
+    # @option options [Integer] :ttr Time-to-run in seconds (1-4294967295, 0 is silently changed to 1)
+    # @option options [String] :queue Queue name override
     #
-    # @return [void]
+    # @return [true] on success (including if already queued)
     #
     # @raise [ActiveRecord::RecordInvalid] if job is not valid
     # @raise [AlreadyProcessed] if job was already processed
@@ -57,17 +59,25 @@ module Postburner
     #   job.queue!(at: '2025-01-15 09:00:00'.in_time_zone)
     #   job.queue!(at: Time.parse('2025-01-15 09:00:00 EST'))
     #
-    # @example Queue with priority
-    #   job.queue!(pri: 0, delay: 30.minutes)
+    # @example Queue with priority and TTR
+    #   job.queue!(priority: 0, ttr: 600)
+    #
+    # @example Queue to specific queue
+    #   job.queue!(queue: 'critical', delay: 30.minutes)
     #
     # @see #requeue!
     # @see Postburner.queue_strategy
     #
     def queue!(options={})
-      return if self.queued_at.present? && self.bkid.present?
+      return true if self.queued_at.present? && self.bkid.present?
       raise ActiveRecord::RecordInvalid, "Can't queue unless valid." unless self.valid?
       raise AlreadyProcessed, "Processed at #{self.processed_at}" if self.processed_at
 
+      # Extract and set instance-level overrides
+      self.priority = options.delete(:priority) if options.key?(:priority)
+      self.ttr = options.delete(:ttr) if options.key?(:ttr)
+      self.queue_name = options.delete(:queue) if options.key?(:queue)
+
       at = options.delete(:at)
       now = Time.current
 
@@ -86,6 +96,8 @@ module Postburner
       run_callbacks :enqueue do
         self.save!
       end
+
+      true
     end
 
     # Re-queues an existing job by removing it from Beanstalkd and queueing again.
@@ -96,10 +108,11 @@ module Postburner
     # @param options [Hash] Queue options (same as {#queue!})
     # @option options [Time, ActiveSupport::Duration] :at Absolute time to run the job
     # @option options [Integer, ActiveSupport::Duration] :delay Seconds to delay execution
-    # @option options [Integer] :pri Beanstalkd priority
-    # @option options [Integer] :ttr Time-to-run in seconds
+    # @option options [Integer] :priority Priority (0-4294967295, lower = higher priority)
+    # @option options [Integer] :ttr Time-to-run in seconds (1-4294967295)
+    # @option options [String] :queue Queue name override
     #
-    # @return [void]
+    # @return [true] on success
     #
     # @raise [ActiveRecord::RecordInvalid] if job is not valid
     # @raise [Beaneater::NotConnected] if Beanstalkd connection fails

data/app/concerns/postburner/properties.rb

@@ -6,6 +6,26 @@ module Postburner
   # Provides DSL methods for configuring queue behavior (name, priority, TTR, retries).
   # Defines configurable properties for job queue management.
   #
+  # == Retry Behavior
+  #
+  # By default, Postburner::Job does NOT retry failed jobs. When a job raises an
+  # exception, it is buried in Beanstalkd for inspection. This differs from default
+  # ActiveJob behavior (5 retries with 2^n second backoff, ~31s total).
+  #
+  # Use +max_retries+ to enable automatic retries:
+  #
+  #   class MyJob < Postburner::Job
+  #     max_retries 5  # Retry up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s)
+  #   end
+  #
+  # The default +retry_delay+ is exponential backoff (2^n seconds), matching
+  # Postburner's default ActiveJob behavior. Override with a fixed delay or custom proc:
+  #
+  #   retry_delay 10  # Fixed 10 second delay between retries
+  #
+  #   # Match ActiveJob's :exponentially_longer (polynomial: (n+1)^4 + 2)
+  #   retry_delay ->(n) { ((n + 1) ** 4) + 2 }  # 3s, 18s, 83s, 258s, 627s
+  #
   # @example Basic usage
   #   class ProcessPayment < Postburner::Job
   #     queue 'critical'
@@ -23,9 +43,11 @@ module Postburner
 
     included do
       # Instance-level queue configuration (overrides class-level defaults)
-      attr_writer :priority, :ttr
+      attr_writer :priority, :ttr, :queue_name
 
-      class_attribute :postburner_queue_name, default: 'default'
+      # Class-level overrides (all default to nil).
+      # Global defaults are defined in Postburner::Configuration (lib/postburner/configuration.rb).
+      class_attribute :postburner_queue_name, default: nil
       class_attribute :postburner_priority, default: nil
       class_attribute :postburner_ttr, default: nil
       class_attribute :postburner_max_retries, default: nil
@@ -106,7 +128,13 @@ module Postburner
 
       # Sets or returns maximum number of job retries.
       #
-      # @param retries [Integer, nil] Max retries, or nil to get current value
+      # By default, Postburner::Job does NOT retry (max_retries is nil).
+      # Failed jobs are buried in Beanstalkd for inspection.
+      # Set max_retries to enable automatic retries on failure.
+      #
+      # Values of nil or 0 disable retries. Maximum allowed value is 32.
+      #
+      # @param retries [Integer, nil] Max retries (0-32), or nil to get current value
       #
       # @return [Integer, nil] Current max retries when getting, nil when setting
       #
@@ -116,9 +144,11 @@ module Postburner
       # @example Get max retries
       #   ProcessPayment.max_retries  # => 3
       #
+      # @see retry_delay
+      #
       def max_retries(retries = nil)
         if retries
-          self.postburner_max_retries = retries
+          self.postburner_max_retries = [[retries.to_i, 0].max, 32].min
           nil
         else
           postburner_max_retries
@@ -166,8 +196,13 @@ module Postburner
     #   job = MyJob.create!(args: {})
     #   job.queue_name  # => 'critical'
     #
+    # @example Instance-level override
+    #   job = MyJob.create!(args: {})
+    #   job.queue_name = 'urgent'
+    #   job.queue_name  # => 'urgent'
+    #
     def queue_name
-      self.class.queue
+      @queue_name || self.class.queue || Postburner.configuration.default_queue_name
     end
 
     # Returns the full tube name with environment prefix.
@@ -217,5 +252,67 @@ module Postburner
       @ttr || self.class.ttr
     end
 
+    # Checks if this job should retry after a failure.
+    #
+    # Returns true if max_retries is configured and the current attempt count
+    # is less than max_retries.
+    #
+    # @return [Boolean] true if job should retry, false otherwise
+    #
+    # @example
+    #   class MyJob < Postburner::Job
+    #     max_retries 3
+    #   end
+    #
+    #   job = MyJob.create!(args: {})
+    #   job.should_retry?  # => true (attempt_count is 0)
+    #   # After 3 failed attempts...
+    #   job.should_retry?  # => false
+    #
+    def should_retry?
+      max = self.class.max_retries || Postburner.configuration.default_max_retries
+      attempt_count.to_i < max.to_i
+    end
+
+    # Calculates the retry delay for the given attempt.
+    #
+    # Uses the class-level retry_delay configuration, which can be:
+    # - Integer: Fixed delay in seconds
+    # - Proc: Called with attempt number (0-based), returns delay in seconds
+    # - nil: Defaults to 5 seconds
+    #
+    # @param attempt [Integer] The attempt number (0-based: 0 for first retry)
+    #
+    # @return [Integer] Delay in seconds before next retry
+    #
+    # @example Fixed delay
+    #   class MyJob < Postburner::Job
+    #     max_retries 3
+    #     retry_delay 10
+    #   end
+    #   job.retry_delay_for_attempt(0)  # => 10
+    #
+    # @example Exponential backoff (2s, 4s, 8s...)
+    #   class MyJob < Postburner::Job
+    #     max_retries 5
+    #     retry_delay ->(n) { 2 ** (n + 1) }
+    #   end
+    #   job.retry_delay_for_attempt(0)  # => 2
+    #   job.retry_delay_for_attempt(1)  # => 4
+    #   job.retry_delay_for_attempt(2)  # => 8
+    #
+    def retry_delay_for_attempt(attempt)
+      delay_config = self.class.retry_delay || Postburner.configuration.default_retry_delay
+
+      case delay_config
+      when Proc
+        delay_config.call(attempt).to_i
+      when Integer
+        delay_config
+      else
+        2 ** attempt # Fallback to default exponential backoff
+      end
+    end
+
   end
 end

data/app/models/postburner/job.rb

@@ -96,13 +96,13 @@ module Postburner
     #
     # @abstract Subclasses must implement this method
     #
-    # @param args [Hash] Job arguments from the args JSONB column
+    # @param args [Hash] Job arguments from the args JSONB column (optional)
     #
     # @return [void]
     #
     # @raise [NotImplementedError] if subclass does not implement this method
     #
-    # @example
+    # @example With args parameter
     #   class ProcessPayment < Postburner::Job
     #     def perform(args)
     #       payment = Payment.find(args['payment_id'])
@@ -111,14 +111,23 @@ module Postburner
     #     end
     #   end
     #
+    # @example Without args parameter (access via self.args)
+    #   class CleanupJob < Postburner::Job
+    #     def perform
+    #       log "Cleaning up #{self.args['table']}"
+    #       # self.args is always available
+    #     end
+    #   end
+    #
     # @note Use {#log} or {#log!} within perform to add entries to the job's audit trail
     # @note Exceptions will be caught, logged to errata, and re-raised
+    # @note Args are always accessible via self.args regardless of method signature
     #
     # @see #perform!
     # @see #log
     # @see #log_exception
     #
-    def perform(args)
+    def perform(args=nil)
       raise NotImplementedError, "Subclasses must implement the perform method"
     end
 

data/lib/postburner/configuration.rb

@@ -22,6 +22,7 @@ module Postburner
   class Configuration
     # Global settings
     attr_accessor :beanstalk_url, :logger, :default_priority, :default_ttr
+    attr_accessor :default_queue_name, :default_max_retries, :default_retry_delay
     attr_accessor :default_scheduler_interval, :default_scheduler_priority
     attr_accessor :enqueue_options
 
@@ -52,6 +53,9 @@ module Postburner
       @logger = options[:logger] || (defined?(Rails) ? Rails.logger : Logger.new(STDOUT))
       @default_priority = options[:default_priority] || 65536
       @default_ttr = options[:default_ttr] || 300
+      @default_queue_name = options[:default_queue_name] || 'default'
+      @default_max_retries = options[:default_max_retries] || 0
+      @default_retry_delay = options[:default_retry_delay] || ->(n) { 2 ** n }
       @default_scheduler_interval = options[:default_scheduler_interval] || 300
       @default_scheduler_priority = options[:default_scheduler_priority] || 100
       @enqueue_options = options[:enqueue_options]

data/lib/postburner/scheduler.rb

@@ -136,6 +136,16 @@ module Postburner
     #   scheduler.perform
     #
     def perform
+      # Self-deduplicate: if another watchdog exists in the queue, exit early
+      # and let that one handle scheduling. This naturally resolves duplicate
+      # watchdogs that can occur from race conditions.
+      @skip_requeue = false
+      if another_watchdog_queued?
+        logger.info "[Postburner::Scheduler] Another watchdog already queued, exiting to deduplicate"
+        @skip_requeue = true
+        return
+      end
+
       logger.info "[Postburner::Scheduler] Starting scheduler run"
 
       ActiveSupport::Notifications.instrument('perform_start.scheduler.postburner', {
@@ -161,15 +171,24 @@ module Postburner
 
       # Use advisory lock to coordinate multiple workers
       ActiveSupport::Notifications.instrument('perform.scheduler.postburner', payload) do
-        lock_acquired = Postburner::AdvisoryLock.with_lock(AdvisoryLock::SCHEDULER_LOCK_KEY, blocking: false) do
-          process_all_schedules
-          true
-        end
-
-        if lock_acquired
-          logger.info "[Postburner::Scheduler] Scheduler run complete"
-        else
-          logger.info "[Postburner::Scheduler] Could not acquire lock, skipping"
+        begin
+          lock_acquired = Postburner::AdvisoryLock.with_lock(AdvisoryLock::SCHEDULER_LOCK_KEY, blocking: false) do
+            process_all_schedules
+            true
+          end
+
+          if lock_acquired
+            logger.info "[Postburner::Scheduler] Scheduler run complete"
+          else
+            logger.info "[Postburner::Scheduler] Could not acquire lock, skipping"
+          end
+        rescue ActiveRecord::ConnectionTimeoutError => e
+          # This can happen if the connection pool is exhausted
+          # Log cleanly and let the watchdog retry on next interval
+          logger.warn "[Postburner::Scheduler] Database connection pool exhausted. Skipping scheduler run (need advisory lock), will retry on next interval"
+          logger.debug "[Postburner::Scheduler] Check database.yml pool and max_connections i.e. pool >= needed connection count from web/job workers"
+          logger.debug "[Postburner::Scheduler] ActiveRecord Connection timeout: #{e.message}"
+          lock_acquired = false
         end
 
         # Update payload with final stats (mutates the hash subscribers receive)
@@ -180,8 +199,8 @@ module Postburner
         payload[:orphans_enqueued] = @orphans_enqueued
       end
     ensure
-      # Always re-queue watchdog for next run
-      requeue_watchdog
+      # Re-queue watchdog for next run (unless we're deduplicating)
+      requeue_watchdog unless @skip_requeue
     end
 
     # Class method to enqueue watchdog to Beanstalkd
@@ -321,6 +340,32 @@ module Postburner
 
     private
 
+    # Check if another watchdog is already queued in Beanstalkd.
+    #
+    # Used for self-deduplication: if this watchdog sees another one queued,
+    # it exits early and lets that one handle scheduling. This resolves
+    # duplicate watchdogs that can occur from race conditions.
+    #
+    # @return [Boolean] true if another watchdog exists (ready or delayed)
+    #
+    # @api private
+    #
+    def another_watchdog_queued?
+      Postburner.connected do |conn|
+        tube_name = Postburner.scheduler_tube_name
+        tube = conn.beanstalk.tubes[tube_name]
+        stats = tube.stats
+
+        # Check for ready or delayed jobs (not counting reserved, which is us)
+        # Note: beaneater transforms hyphenated beanstalkd stats to underscores
+        queued_count = stats.current_jobs_ready.to_i + stats.current_jobs_delayed.to_i
+        queued_count > 0
+      end
+    rescue => e
+      logger.debug "[Postburner::Scheduler] Could not check for duplicate watchdogs: #{e.message}"
+      false # Assume no duplicate if we can't check
+    end
+
     # Process all enabled schedules.
     #
     # Iterates through all enabled schedules and calls process_schedule for each.
@@ -472,6 +517,10 @@ module Postburner
 
         self.class.enqueue_watchdog
       end
+    rescue ActiveRecord::ConnectionTimeoutError => e
+      # Connection pool exhausted - workers will recreate watchdog on next timeout
+      logger.warn "[Postburner::Scheduler] Could not re-queue watchdog (connection pool exhausted), workers will recreate on timeout"
+      logger.debug "[Postburner::Scheduler] Connection timeout details: #{e.message}"
     rescue => e
       logger.error "[Postburner::Scheduler] Failed to re-queue watchdog: #{e.class} - #{e.message}"
       # This is critical - if watchdog isn't re-queued, scheduling stops

data/lib/postburner/tube.rb

@@ -31,10 +31,9 @@ module Postburner
     # Just pass the last known id to after for the next batch.
     #
     def jobs(count=20, limit: 1000, after: nil)
-      # Access raw hash to avoid beaneater FastStruct method definition issues in Ruby 3.4
+      # Note: beaneater transforms hyphenated beanstalkd stats to underscores
       stats = @tube.stats
-      stats_hash = stats.instance_variable_get(:@hash) || {}
-      tube_name = stats_hash['name']
+      tube_name = stats.name
 
       jobs = Array.new
 
@@ -48,8 +47,7 @@ module Postburner
         job = @tube.client.jobs.find(i)
         if job
           job_stats = job.stats
-          job_stats_hash = job_stats.instance_variable_get(:@hash) || {}
-          jobs << job if job_stats_hash['tube'] == tube_name
+          jobs << job if job_stats.tube == tube_name
         end
         break if jobs.length >= count
       end

data/lib/postburner/version.rb

@@ -1,3 +1,3 @@
 module Postburner
-  VERSION = '1.0.0.pre.15'
+  VERSION = '1.0.0.pre.17'
 end

data/lib/postburner/worker.rb

@@ -136,6 +136,28 @@ module Postburner
       @shutdown
     end
 
+    # Checks if this process has been orphaned (parent died).
+    #
+    # When the parent process dies, the kernel re-parents children to init (PID 1).
+    # Detecting this allows forked children to exit gracefully instead of running
+    # indefinitely as orphans.
+    #
+    # @return [Boolean] true if parent PID is 1, false otherwise
+    def orphaned?
+      Process.ppid == 1
+    end
+
+    # Calculates exponential backoff sleep duration for reconnection attempts.
+    #
+    # Uses exponential backoff starting at 1 second and doubling each attempt,
+    # capped at 32 seconds to prevent excessively long waits.
+    #
+    # @param attempts [Integer] Number of consecutive failed attempts (0-based)
+    # @return [Integer] Sleep duration in seconds (1, 2, 4, 8, 16, or 32)
+    def reconnect_backoff(attempts)
+      [2**attempts, 32].min
+    end
+
     private
 
     # Returns the worker configuration hash.
@@ -148,13 +170,17 @@ module Postburner
 
     # Sets up signal handlers for graceful shutdown.
     #
-    # Traps TERM and INT signals to initiate graceful shutdown.
+    # Trapped signals:
+    # - TERM: Graceful termination request (systemd, kill, process managers)
+    # - INT: Interrupt from keyboard (Ctrl+C)
+    # - HUP: Hangup signal when controlling terminal dies (prevents orphaned children)
     #
     # @return [void]
     # @api private
     def setup_signal_handlers
       Signal.trap('TERM') { shutdown }
       Signal.trap('INT') { shutdown }
+      Signal.trap('HUP') { shutdown }
     end
 
     # Expands queue name to full tube name with environment prefix.
@@ -245,17 +271,19 @@ module Postburner
     def process_jobs
       connection = Postburner::Connection.new
       timeout = worker_config[:timeout]
+      reconnect_attempts = 0
 
       watch_queues(connection, config.queue_names)
 
       until shutdown? || (@gc_limit && @jobs_processed.value >= @gc_limit)
         begin
-          job = connection.beanstalk.tubes.reserve(timeout: timeout)
+          job = connection.beanstalk.tubes.reserve(timeout)
 
           if job
             logger.debug "[Postburner::Worker] Thread #{Thread.current.object_id} reserved job #{job.id}"
             execute_job(job)
             @jobs_processed.increment
+            reconnect_attempts = 0  # Reset backoff on successful job execution
           else
             ensure_scheduler_watchdog!(connection)
           end
@@ -263,8 +291,10 @@ module Postburner
           ensure_scheduler_watchdog!(connection)
           next
         rescue Beaneater::NotConnected => e
-          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}"
-          sleep 1
+          backoff = reconnect_backoff(reconnect_attempts)
+          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}, reconnecting in #{backoff}s (attempt #{reconnect_attempts + 1})"
+          sleep backoff
+          reconnect_attempts += 1
           connection.reconnect!
           watch_queues(connection, config.queue_names)
         rescue => e
@@ -359,6 +389,11 @@ module Postburner
       end
 
       until shutdown? || (gc_limit && jobs_processed.value >= gc_limit)
+        if orphaned?
+          logger.error "[Postburner::Worker] Fork #{fork_num} detected parent died (orphaned), initiating shutdown"
+          shutdown
+          break
+        end
         sleep 0.5
       end
 
@@ -391,17 +426,19 @@ module Postburner
     def process_jobs_in_fork(fork_num, jobs_processed, gc_limit)
       connection = Postburner::Connection.new
       timeout = worker_config[:timeout]
+      reconnect_attempts = 0
 
       watch_queues(connection, config.queue_names)
 
       until shutdown? || (gc_limit && jobs_processed.value >= gc_limit)
         begin
-          job = connection.beanstalk.tubes.reserve(timeout: timeout)
+          job = connection.beanstalk.tubes.reserve(timeout)
 
           if job
             logger.debug "[Postburner::Worker] Fork #{fork_num} thread #{Thread.current.object_id} reserved job #{job.id}"
             execute_job(job)
             jobs_processed.increment
+            reconnect_attempts = 0  # Reset backoff on successful job execution
           else
             ensure_scheduler_watchdog!(connection)
           end
@@ -409,8 +446,10 @@ module Postburner
           ensure_scheduler_watchdog!(connection)
           next
         rescue Beaneater::NotConnected => e
-          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}"
-          sleep 1
+          backoff = reconnect_backoff(reconnect_attempts)
+          logger.error "[Postburner::Worker] Thread disconnected: #{e.message}, reconnecting in #{backoff}s (attempt #{reconnect_attempts + 1})"
+          sleep backoff
+          reconnect_attempts += 1
           connection.reconnect!
           watch_queues(connection, config.queue_names)
         rescue => e
@@ -549,12 +588,16 @@ module Postburner
 
     # Handles job execution errors with retry logic.
     #
-    # For tracked jobs and legacy Postburner::Job: Buries the job for inspection
-    # and emits retry_stopped.job.postburner event.
-    # For default ActiveJob: Applies exponential backoff retry with max 5 attempts.
+    # For Postburner::Job: The job handles its own retries in perform!. If an
+    # exception bubbles up here, it means no retries configured or max exceeded.
+    # Buries the job for inspection, reports to Rails.error, and emits
+    # retry_stopped.job.postburner event.
+    #
+    # For default ActiveJob: Applies exponential backoff retry with max 5 attempts,
+    # reporting to Rails.error only on final discard.
     #
     # Instruments with ActiveSupport::Notifications:
-    # - retry_stopped.job.postburner: When tracked job is buried after failure
+    # - retry_stopped.job.postburner: When Postburner::Job is buried after failure
     #
     # @param beanstalk_job [Beaneater::Job] Failed job
     # @param error [Exception] The error that caused the failure
@@ -562,24 +605,14 @@ module Postburner
     # @api private
     def handle_error(beanstalk_job, error)
       logger.error "[Postburner] Job failed: #{error.class} - #{error.message}"
-      logger.error error.backtrace.join("\n")
 
       begin
         payload = JSON.parse(beanstalk_job.body)
 
         if payload['tracked'] || Postburner::ActiveJob::Payload.legacy_format?(payload)
-          logger.info "[Postburner] Burying tracked/legacy job for inspection"
-
-          job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
-          ActiveSupport::Notifications.instrument('retry_stopped.job.postburner', {
-            job: job_payload,
-            beanstalk_job_id: beanstalk_job.id,
-            error: error
-          })
-
-          beanstalk_job.bury
+          handle_postburner_job_error(beanstalk_job, payload, error)
         else
-          handle_default_retry(beanstalk_job, payload, error)
+          handle_default_job_error(beanstalk_job, payload, error)
         end
       rescue => retry_error
         logger.error "[Postburner] Error handling failure: #{retry_error.message}"
@@ -587,62 +620,85 @@ module Postburner
       end
     end
 
-    # Handles retry logic for default jobs.
+    # Handles errors for Postburner::Job (including tracked ActiveJob).
     #
-    # Applies exponential backoff (2^retry_count seconds, max 1 hour).
-    # After 5 failed attempts, discards the job permanently.
+    # The job handles its own retries in perform!. If the exception bubbles up
+    # here, it means either no retries are configured or max_retries was exceeded.
+    # Buries the job for inspection.
     #
     # Instruments with ActiveSupport::Notifications:
-    # - retry.job.postburner: When job is retried
-    # - discard.job.postburner: When job is discarded after max retries
+    # - retry_stopped.job.postburner: When job is buried
     #
-    # @param beanstalk_job [Beaneater::Job] Failed job to retry
-    # @param payload [Hash] Parsed job body (modified with retry_count)
+    # @param beanstalk_job [Beaneater::Job] Failed job
+    # @param payload [Hash] Parsed job body
     # @param error [Exception] The error that caused the failure
     # @return [void]
     # @api private
-    def handle_default_retry(beanstalk_job, payload, error)
-      retry_count = payload['retry_count'] || 0
-      max_retries = 5
-      job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
+    def handle_postburner_job_error(beanstalk_job, payload, error)
+      job_id = if Postburner::ActiveJob::Payload.legacy_format?(payload)
+                 payload['args']&.first
+               else
+                 payload['postburner_job_id']
+               end
 
-      if retry_count < max_retries
-        payload['retry_count'] = retry_count + 1
-        payload['executions'] = (payload['executions'] || 0) + 1
+      job_class_name = payload['job_class'] || payload['class']
 
-        delay = [2 ** retry_count, 3600].min
+      # Log the error with backtrace (Postburner::Job doesn't use ActiveJob's logging)
+      logger.error "[Postburner] #{job_class_name}##{job_id} failed: #{error.class} - #{error.message}"
+      logger.error error.backtrace.join("\n")
 
-        beanstalk_job.delete
+      logger.info "[Postburner] Burying #{job_class_name}##{job_id} for inspection (bkid: #{beanstalk_job.id})"
 
-        Postburner.connected do |conn|
-          tube_name = expand_tube_name(payload['queue_name'])
-          conn.tubes[tube_name].put(
-            JSON.generate(payload),
-            pri: payload['priority'] || config.default_priority,
-            delay: delay,
-            ttr: payload['ttr'] || config.default_ttr
-          )
-        end
+      # Report to Rails error reporter for integration with error tracking services
+      Rails.error.report(error, handled: false, context: {
+        job_class: job_class_name,
+        job_id: job_id,
+        beanstalk_job_id: beanstalk_job.id,
+        queue_name: payload['queue_name']
+      })
 
-        ActiveSupport::Notifications.instrument('retry.job.postburner', {
-          job: job_payload,
-          beanstalk_job_id: beanstalk_job.id,
-          error: error,
-          wait: delay,
-          attempt: retry_count + 1
-        })
+      job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
+      ActiveSupport::Notifications.instrument('retry_stopped.job.postburner', {
+        job: job_payload,
+        beanstalk_job_id: beanstalk_job.id,
+        error: error
+      })
 
-        logger.info "[Postburner] Retrying default job #{payload['job_id']}, attempt #{retry_count + 1} in #{delay}s"
-      else
-        ActiveSupport::Notifications.instrument('discard.job.postburner', {
-          job: job_payload,
-          beanstalk_job_id: beanstalk_job.id,
-          error: error
-        })
-
-        logger.error "[Postburner] Discarding default job #{payload['job_id']} after #{retry_count} retries"
-        beanstalk_job.delete
-      end
+      beanstalk_job.bury
+    end
+
+    # Handles errors for default ActiveJob jobs.
+    #
+    # Discards the job and reports to Rails.error. No automatic retries -
+    # use ActiveJob's retry_on/discard_on for retry behavior.
+    #
+    # Instruments with ActiveSupport::Notifications:
+    # - discard.job.postburner: When job is discarded
+    #
+    # @param beanstalk_job [Beaneater::Job] Failed job
+    # @param payload [Hash] Parsed job body
+    # @param error [Exception] The error that caused the failure
+    # @return [void]
+    # @api private
+    def handle_default_job_error(beanstalk_job, payload, error)
+      job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
+
+      # Report to Rails error reporter for integration with error tracking services
+      Rails.error.report(error, handled: false, context: {
+        job_class: payload['job_class'],
+        job_id: payload['job_id'],
+        beanstalk_job_id: beanstalk_job.id,
+        queue_name: payload['queue_name']
+      })
+
+      ActiveSupport::Notifications.instrument('discard.job.postburner', {
+        job: job_payload,
+        beanstalk_job_id: beanstalk_job.id,
+        error: error
+      })
+
+      logger.error "[Postburner] Discarding #{payload['job_class']} (#{payload['job_id']})"
+      beanstalk_job.delete
     end
 
     # Watches all configured queues in Beanstalkd.

data/lib/postburner.rb

@@ -559,19 +559,18 @@ module Postburner
       tubes_to_inspect.each do |tube|
         begin
           stats = tube.stats
-          # Beaneater returns a StatStruct; access the underlying hash
-          stats_hash = stats.instance_variable_get(:@hash) || {}
+          # Note: beaneater transforms hyphenated beanstalkd stats to underscores
 
           tube_data = {
             name: tube.name,
-            ready: stats_hash['current_jobs_ready'] || 0,
-            delayed: stats_hash['current_jobs_delayed'] || 0,
-            buried: stats_hash['current_jobs_buried'] || 0,
-            reserved: stats_hash['current_jobs_reserved'] || 0,
-            total: (stats_hash['current_jobs_ready'] || 0) +
-                   (stats_hash['current_jobs_delayed'] || 0) +
-                   (stats_hash['current_jobs_buried'] || 0) +
-                   (stats_hash['current_jobs_reserved'] || 0)
+            ready: stats.current_jobs_ready || 0,
+            delayed: stats.current_jobs_delayed || 0,
+            buried: stats.current_jobs_buried || 0,
+            reserved: stats.current_jobs_reserved || 0,
+            total: (stats.current_jobs_ready || 0) +
+                   (stats.current_jobs_delayed || 0) +
+                   (stats.current_jobs_buried || 0) +
+                   (stats.current_jobs_reserved || 0)
           }
         rescue Beaneater::NotFoundError
           # Tube doesn't exist yet, skip it

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: postburner
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre.15
+  version: 1.0.0.pre.17
 platform: ruby
 authors:
 - Matt Smith