postburner 1.0.0.pre.15 → 1.0.0.pre.16

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 971531132068ea4bddfdb5b5a8aaae4c788ea576aae268010fb6cd4dbd2b82f1
-  data.tar.gz: 44053985f2bebd385bc2a17eb1eb655d89de17d77a8c25b4a1210177cefead66
+  metadata.gz: 8e032f7d37f1680fe34a1402f8eb9f0b3ca61aa9231f6d540c35c3a919e3f58d
+  data.tar.gz: bd35338dd14bc499c4c7d9f57d30eff3902615cd72876f766f1e379f12baba77
 SHA512:
-  metadata.gz: 19c023063d9eb06913fcb59eaaf581a537b8a944b8ab8713b95210475517223a42887f239bd8788cd313919ffe9bfb89c6c9f80c8b24869239b943bfd2c300ad
-  data.tar.gz: 0e0ed766a67ddd9ef02c0b97eff3a25c1e923c79068608a4d31db779ae23187eb85a5b075e8100644faf1c196f0bafaf1a7711e09db8a948b0ab09e9138f9b49
+  metadata.gz: f207c1d4dd840bee6b00ad71de9cc1b774d9b2376f7f34125fcc3a164220a340badc6b1ef17931d3dc09a0bb78a09c66b1f1f9ffdf1ead3e9737c44a39fe9a29
+  data.tar.gz: e0212499c62223a87720462226660045585ac4bde21258eb54aeed9d85683c0c9652477a6cf5824bdffa7e30314efb1e1cbb9d146eae8ef3ade2202af73f2abe

data/README.md

@@ -97,6 +97,7 @@ bundle exec rake postburner:work WORKER=default
   - [Postburner::Job](#postburnerjob)
   - [Scheduler](#scheduler)
   - [Job Management](#job-management)
+- [Writing Jobs](#writing-jobs)
 - [Queue Strategies](#queue-strategies)
 - [Testing](#testing)
 - [Workers](#workers)
@@ -177,16 +178,45 @@ config.active_job.queue_adapter = :postburner
 #config.active_job.queue_name_prefix = Postburner.tube_prefix(Rails.env) # i.e. "postburner.#{Rails.env}"
 config.action_mailer.deliver_later_queue_name = 'mailers' # gets prefixed by config.active_job.queue_name_prefix
 
-
 bundle exec postburner           # start with bin/postburner
 bundle exec rake postburner:work # or with rake task
 ```
 
-## Usage
+### Enqueueing Jobs
 
-**Job Idempotency:** Jobs should be designed to be idempotent and safely re-runnable. Like all job queues, Postburner provides at-least-once delivery—in rare errant cases outside of Postburner's control, a job may be executed more than once, i.e. network issues, etc.
+```ruby
+# ActiveJob (standard Rails API)
+SendEmailJob.perform_later(user_id)                           # Enqueue immediately
+SendEmailJob.set(wait: 1.hour).perform_later(user_id)         # Delay by duration
+SendEmailJob.set(wait_until: Date.tomorrow.noon).perform_later(user_id)  # Run at specific time
+SendEmailJob.set(queue: 'critical').perform_later(user_id)    # Override queue
+SendEmailJob.set(priority: 0).perform_later(user_id)          # Override priority
+
+# Postburner::Job (always tracked, full API)
+job = ProcessPayment.create!(args: { 'payment_id' => 123 })
+job.queue!                                    # Enqueue immediately
+job.queue!(delay: 1.hour)                     # Delay by duration
+job.queue!(at: Date.tomorrow.noon)            # Run at specific time
+job.queue!(queue: 'critical')                 # Override queue
+job.queue!(priority: 0, ttr: 600)             # Set priority and TTR
+```
 
-**TTR (Time-to-Run):** If a job exceeds its TTR without completion, Beanstalkd releases it back to the queue while still running—causing duplicate execution. For long-running jobs, call `extend!` periodically to reset the TTR, or set a sufficiently high TTR value. You must include the `Postburner::Beanstalkd` or `Postburner::Tracked` module with `ActiveJob` to use `extend!`.
+### ActiveJob vs Postburner::Job TL;DR
+
+`Postburner::Job` as simple subclass of `ActiveRecord`, so the normal
+`ActiveRecord` API applies! Thus the workflow is to create an instance,
+then queue it!
+
+| Operation | ActiveJob | Postburner::Job |
+|-----------|-----------|-----------------|
+| **Enqueue immediately** | `MyJob.perform_later(args)` | `MyJob.create!(args: {}).queue!` |
+| **Delay** | `.set(wait: 1.hour)` | `job.queue!(delay: 1.hour)` |
+| **Run at** | `.set(wait_until: time)` | `job.queue!(at: time)` |
+| **Set queue** | `.set(queue: 'critical')` | `job.queue!(queue: 'critical')` |
+| **Set priority** | `.set(priority: 0)` | `job.queue!(priority: 0)` |
+| **Set TTR** | `.set(ttr: 300)` | `job.queue!(ttr: 300)` |
+
+## Usage
 
 ### Default Jobs
 
@@ -335,7 +365,7 @@ job.duration  # Execution time in milliseconds
 job.lag       # Queue lag in milliseconds
 ```
 
-### Direct Postburner::Job Usage
+### Postburner::Job Usage
 
 Direct `Postburner::Job` subclasses are **always tracked**:
 
@@ -363,6 +393,15 @@ job.queue!(delay: 1.hour)
 job.queue!(at: 2.days.from_now)
 ```
 
+> **Note:** The `args` parameter in `perform(args)` is optional. It's a convenience accessor to `self.args`, which is stored in a JSONB column on the job record. You can omit the parameter and access args directly:
+>
+> ```ruby
+> def perform
+>   payment = Payment.find(self.args['payment_id'])
+>   # ...
+> end
+> ```
+
 #### Instance-Level Queue Configuration
 
 Override queue priority and TTR per job instance for dynamic behavior:
@@ -775,6 +814,14 @@ job.errata  # Array of exceptions with backtraces
 job.attempts # Array of attempt timestamps
 ```
 
+## Writing Jobs
+
+Pay attention to the following when writing jobs:
+
+**Job Idempotency:** Jobs should be designed to be idempotent and safely re-runnable. Like all job queues, Postburner provides at-least-once delivery—in rare errant cases outside of Postburner's control, a job may be executed more than once, i.e. network issues, etc.
+
+**TTR (Time-to-Run):** If a job exceeds its TTR without completion, Beanstalkd releases it back to the queue while still running—causing duplicate execution. For long-running jobs, call `extend!` periodically to reset the TTR, or set a sufficiently high TTR value. You must include the `Postburner::Beanstalkd` or `Postburner::Tracked` module with `ActiveJob` to use `extend!`.
+
 ## Queue Strategies
 
 Postburner uses different strategies to control job execution. These affect `Postburner::Job` subclasses (not ActiveJob classes).

data/app/concerns/postburner/execution.rb

@@ -134,7 +134,7 @@ module Postburner
 
           run_callbacks :processing do
             begin
-              self.perform(args)
+              method(:perform).arity == 0 ? self.perform : self.perform(args)
             rescue Exception => exception
               self.persist_metadata!
               self.log! '[Postburner] Exception raised during perform prevented completion.'

data/app/concerns/postburner/insertion.rb

@@ -37,10 +37,12 @@ module Postburner
     # @param options [Hash] Queue options
     # @option options [Time, ActiveSupport::Duration] :at Absolute time to run the job
     # @option options [Integer, ActiveSupport::Duration] :delay Seconds to delay execution
-    # @option options [Integer] :pri Beanstalkd priority (lower = higher priority)
-    # @option options [Integer] :ttr Time-to-run in seconds before job times out
+    # @option options [Integer] :priority Priority (0-4294967295, 0 = HIGHEST), sets instance attribute
+    # @option options [Integer] :pri Beanstalkd priority (pass-through, for backwards compatibility)
+    # @option options [Integer] :ttr Time-to-run in seconds (1-4294967295, 0 is silently changed to 1)
+    # @option options [String] :queue Queue name override
     #
-    # @return [void]
+    # @return [true] on success (including if already queued)
     #
     # @raise [ActiveRecord::RecordInvalid] if job is not valid
     # @raise [AlreadyProcessed] if job was already processed
@@ -57,17 +59,25 @@ module Postburner
     #   job.queue!(at: '2025-01-15 09:00:00'.in_time_zone)
     #   job.queue!(at: Time.parse('2025-01-15 09:00:00 EST'))
     #
-    # @example Queue with priority
-    #   job.queue!(pri: 0, delay: 30.minutes)
+    # @example Queue with priority and TTR
+    #   job.queue!(priority: 0, ttr: 600)
+    #
+    # @example Queue to specific queue
+    #   job.queue!(queue: 'critical', delay: 30.minutes)
     #
     # @see #requeue!
     # @see Postburner.queue_strategy
     #
     def queue!(options={})
-      return if self.queued_at.present? && self.bkid.present?
+      return true if self.queued_at.present? && self.bkid.present?
       raise ActiveRecord::RecordInvalid, "Can't queue unless valid." unless self.valid?
       raise AlreadyProcessed, "Processed at #{self.processed_at}" if self.processed_at
 
+      # Extract and set instance-level overrides
+      self.priority = options.delete(:priority) if options.key?(:priority)
+      self.ttr = options.delete(:ttr) if options.key?(:ttr)
+      self.queue_name = options.delete(:queue) if options.key?(:queue)
+
       at = options.delete(:at)
       now = Time.current
 
@@ -86,6 +96,8 @@ module Postburner
       run_callbacks :enqueue do
         self.save!
       end
+
+      true
     end
 
     # Re-queues an existing job by removing it from Beanstalkd and queueing again.
@@ -96,10 +108,11 @@ module Postburner
     # @param options [Hash] Queue options (same as {#queue!})
     # @option options [Time, ActiveSupport::Duration] :at Absolute time to run the job
     # @option options [Integer, ActiveSupport::Duration] :delay Seconds to delay execution
-    # @option options [Integer] :pri Beanstalkd priority
-    # @option options [Integer] :ttr Time-to-run in seconds
+    # @option options [Integer] :priority Priority (0-4294967295, lower = higher priority)
+    # @option options [Integer] :ttr Time-to-run in seconds (1-4294967295)
+    # @option options [String] :queue Queue name override
     #
-    # @return [void]
+    # @return [true] on success
     #
     # @raise [ActiveRecord::RecordInvalid] if job is not valid
     # @raise [Beaneater::NotConnected] if Beanstalkd connection fails

data/app/concerns/postburner/properties.rb

@@ -23,7 +23,7 @@ module Postburner
 
     included do
       # Instance-level queue configuration (overrides class-level defaults)
-      attr_writer :priority, :ttr
+      attr_writer :priority, :ttr, :queue_name
 
       class_attribute :postburner_queue_name, default: 'default'
       class_attribute :postburner_priority, default: nil
@@ -166,8 +166,13 @@ module Postburner
     #   job = MyJob.create!(args: {})
     #   job.queue_name  # => 'critical'
     #
+    # @example Instance-level override
+    #   job = MyJob.create!(args: {})
+    #   job.queue_name = 'urgent'
+    #   job.queue_name  # => 'urgent'
+    #
     def queue_name
-      self.class.queue
+      @queue_name || self.class.queue
     end
 
     # Returns the full tube name with environment prefix.

data/app/models/postburner/job.rb

@@ -96,13 +96,13 @@ module Postburner
     #
     # @abstract Subclasses must implement this method
     #
-    # @param args [Hash] Job arguments from the args JSONB column
+    # @param args [Hash] Job arguments from the args JSONB column (optional)
     #
     # @return [void]
     #
     # @raise [NotImplementedError] if subclass does not implement this method
     #
-    # @example
+    # @example With args parameter
     #   class ProcessPayment < Postburner::Job
     #     def perform(args)
     #       payment = Payment.find(args['payment_id'])
@@ -111,14 +111,23 @@ module Postburner
     #     end
     #   end
     #
+    # @example Without args parameter (access via self.args)
+    #   class CleanupJob < Postburner::Job
+    #     def perform
+    #       log "Cleaning up #{self.args['table']}"
+    #       # self.args is always available
+    #     end
+    #   end
+    #
     # @note Use {#log} or {#log!} within perform to add entries to the job's audit trail
     # @note Exceptions will be caught, logged to errata, and re-raised
+    # @note Args are always accessible via self.args regardless of method signature
     #
     # @see #perform!
     # @see #log
     # @see #log_exception
     #
-    def perform(args)
+    def perform(args=nil)
       raise NotImplementedError, "Subclasses must implement the perform method"
     end
 

data/lib/postburner/scheduler.rb

@@ -136,6 +136,16 @@ module Postburner
     #   scheduler.perform
     #
     def perform
+      # Self-deduplicate: if another watchdog exists in the queue, exit early
+      # and let that one handle scheduling. This naturally resolves duplicate
+      # watchdogs that can occur from race conditions.
+      @skip_requeue = false
+      if another_watchdog_queued?
+        logger.info "[Postburner::Scheduler] Another watchdog already queued, exiting to deduplicate"
+        @skip_requeue = true
+        return
+      end
+
       logger.info "[Postburner::Scheduler] Starting scheduler run"
 
       ActiveSupport::Notifications.instrument('perform_start.scheduler.postburner', {
@@ -161,15 +171,24 @@ module Postburner
 
       # Use advisory lock to coordinate multiple workers
       ActiveSupport::Notifications.instrument('perform.scheduler.postburner', payload) do
-        lock_acquired = Postburner::AdvisoryLock.with_lock(AdvisoryLock::SCHEDULER_LOCK_KEY, blocking: false) do
-          process_all_schedules
-          true
-        end
-
-        if lock_acquired
-          logger.info "[Postburner::Scheduler] Scheduler run complete"
-        else
-          logger.info "[Postburner::Scheduler] Could not acquire lock, skipping"
+        begin
+          lock_acquired = Postburner::AdvisoryLock.with_lock(AdvisoryLock::SCHEDULER_LOCK_KEY, blocking: false) do
+            process_all_schedules
+            true
+          end
+
+          if lock_acquired
+            logger.info "[Postburner::Scheduler] Scheduler run complete"
+          else
+            logger.info "[Postburner::Scheduler] Could not acquire lock, skipping"
+          end
+        rescue ActiveRecord::ConnectionTimeoutError => e
+          # This can happen if the connection pool is exhausted
+          # Log cleanly and let the watchdog retry on next interval
+          logger.warn "[Postburner::Scheduler] Database connection pool exhausted. Skipping scheduler run (need advisory lock), will retry on next interval"
+          logger.debug "[Postburner::Scheduler] Check database.yml pool and max_connections i.e. pool >= needed connection count from web/job workers"
+          logger.debug "[Postburner::Scheduler] ActiveRecord Connection timeout: #{e.message}"
+          lock_acquired = false
         end
 
         # Update payload with final stats (mutates the hash subscribers receive)
@@ -180,8 +199,8 @@ module Postburner
         payload[:orphans_enqueued] = @orphans_enqueued
       end
     ensure
-      # Always re-queue watchdog for next run
-      requeue_watchdog
+      # Re-queue watchdog for next run (unless we're deduplicating)
+      requeue_watchdog unless @skip_requeue
     end
 
     # Class method to enqueue watchdog to Beanstalkd
@@ -321,6 +340,32 @@ module Postburner
 
     private
 
+    # Check if another watchdog is already queued in Beanstalkd.
+    #
+    # Used for self-deduplication: if this watchdog sees another one queued,
+    # it exits early and lets that one handle scheduling. This resolves
+    # duplicate watchdogs that can occur from race conditions.
+    #
+    # @return [Boolean] true if another watchdog exists (ready or delayed)
+    #
+    # @api private
+    #
+    def another_watchdog_queued?
+      Postburner.connected do |conn|
+        tube_name = Postburner.scheduler_tube_name
+        tube = conn.beanstalk.tubes[tube_name]
+        stats = tube.stats
+
+        # Check for ready or delayed jobs (not counting reserved, which is us)
+        # Note: beaneater transforms hyphenated beanstalkd stats to underscores
+        queued_count = stats.current_jobs_ready.to_i + stats.current_jobs_delayed.to_i
+        queued_count > 0
+      end
+    rescue => e
+      logger.debug "[Postburner::Scheduler] Could not check for duplicate watchdogs: #{e.message}"
+      false # Assume no duplicate if we can't check
+    end
+
     # Process all enabled schedules.
     #
     # Iterates through all enabled schedules and calls process_schedule for each.
@@ -472,6 +517,10 @@ module Postburner
 
         self.class.enqueue_watchdog
       end
+    rescue ActiveRecord::ConnectionTimeoutError => e
+      # Connection pool exhausted - workers will recreate watchdog on next timeout
+      logger.warn "[Postburner::Scheduler] Could not re-queue watchdog (connection pool exhausted), workers will recreate on timeout"
+      logger.debug "[Postburner::Scheduler] Connection timeout details: #{e.message}"
     rescue => e
       logger.error "[Postburner::Scheduler] Failed to re-queue watchdog: #{e.class} - #{e.message}"
       # This is critical - if watchdog isn't re-queued, scheduling stops

data/lib/postburner/tube.rb

@@ -31,10 +31,9 @@ module Postburner
     # Just pass the last known id to after for the next batch.
     #
     def jobs(count=20, limit: 1000, after: nil)
-      # Access raw hash to avoid beaneater FastStruct method definition issues in Ruby 3.4
+      # Note: beaneater transforms hyphenated beanstalkd stats to underscores
       stats = @tube.stats
-      stats_hash = stats.instance_variable_get(:@hash) || {}
-      tube_name = stats_hash['name']
+      tube_name = stats.name
 
       jobs = Array.new
 
@@ -48,8 +47,7 @@ module Postburner
         job = @tube.client.jobs.find(i)
         if job
           job_stats = job.stats
-          job_stats_hash = job_stats.instance_variable_get(:@hash) || {}
-          jobs << job if job_stats_hash['tube'] == tube_name
+          jobs << job if job_stats.tube == tube_name
         end
         break if jobs.length >= count
       end

data/lib/postburner/version.rb

@@ -1,3 +1,3 @@
 module Postburner
-  VERSION = '1.0.0.pre.15'
+  VERSION = '1.0.0.pre.16'
 end

data/lib/postburner/worker.rb

@@ -549,9 +549,10 @@ module Postburner
 
     # Handles job execution errors with retry logic.
     #
-    # For tracked jobs and legacy Postburner::Job: Buries the job for inspection
-    # and emits retry_stopped.job.postburner event.
-    # For default ActiveJob: Applies exponential backoff retry with max 5 attempts.
+    # For tracked jobs and legacy Postburner::Job: Buries the job for inspection,
+    # reports to Rails.error, and emits retry_stopped.job.postburner event.
+    # For default ActiveJob: Applies exponential backoff retry with max 5 attempts,
+    # reporting to Rails.error only on final discard.
     #
     # Instruments with ActiveSupport::Notifications:
     # - retry_stopped.job.postburner: When tracked job is buried after failure
@@ -568,7 +569,22 @@ module Postburner
         payload = JSON.parse(beanstalk_job.body)
 
         if payload['tracked'] || Postburner::ActiveJob::Payload.legacy_format?(payload)
-          logger.info "[Postburner] Burying tracked/legacy job for inspection"
+          # Extract job ID from payload
+          job_id = if Postburner::ActiveJob::Payload.legacy_format?(payload)
+                     payload['args']&.first
+                   else
+                     payload['postburner_job_id']
+                   end
+
+          logger.info "[Postburner] Burying tracked/legacy job for inspection (bkid: #{beanstalk_job.id}, job_id: #{job_id})"
+
+          # Report to Rails error reporter for integration with error tracking services
+          Rails.error.report(error, handled: false, context: {
+            job_class: payload['job_class'] || payload['class'],
+            job_id: job_id,
+            beanstalk_job_id: beanstalk_job.id,
+            queue_name: payload['queue_name']
+          })
 
           job_payload = Postburner::Instrumentation.job_payload_from_hash(payload, beanstalk_job_id: beanstalk_job.id)
           ActiveSupport::Notifications.instrument('retry_stopped.job.postburner', {
@@ -590,7 +606,8 @@ module Postburner
     # Handles retry logic for default jobs.
     #
     # Applies exponential backoff (2^retry_count seconds, max 1 hour).
-    # After 5 failed attempts, discards the job permanently.
+    # After 5 failed attempts, discards the job permanently and reports
+    # to Rails.error for integration with error tracking services.
     #
     # Instruments with ActiveSupport::Notifications:
     # - retry.job.postburner: When job is retried
@@ -634,6 +651,15 @@ module Postburner
 
         logger.info "[Postburner] Retrying default job #{payload['job_id']}, attempt #{retry_count + 1} in #{delay}s"
       else
+        # Report to Rails error reporter for integration with error tracking services
+        Rails.error.report(error, handled: false, context: {
+          job_class: payload['job_class'],
+          job_id: payload['job_id'],
+          beanstalk_job_id: beanstalk_job.id,
+          queue_name: payload['queue_name'],
+          retry_count: retry_count
+        })
+
         ActiveSupport::Notifications.instrument('discard.job.postburner', {
           job: job_payload,
           beanstalk_job_id: beanstalk_job.id,

data/lib/postburner.rb

@@ -559,19 +559,18 @@ module Postburner
       tubes_to_inspect.each do |tube|
         begin
           stats = tube.stats
-          # Beaneater returns a StatStruct; access the underlying hash
-          stats_hash = stats.instance_variable_get(:@hash) || {}
+          # Note: beaneater transforms hyphenated beanstalkd stats to underscores
 
           tube_data = {
             name: tube.name,
-            ready: stats_hash['current_jobs_ready'] || 0,
-            delayed: stats_hash['current_jobs_delayed'] || 0,
-            buried: stats_hash['current_jobs_buried'] || 0,
-            reserved: stats_hash['current_jobs_reserved'] || 0,
-            total: (stats_hash['current_jobs_ready'] || 0) +
-                   (stats_hash['current_jobs_delayed'] || 0) +
-                   (stats_hash['current_jobs_buried'] || 0) +
-                   (stats_hash['current_jobs_reserved'] || 0)
+            ready: stats.current_jobs_ready || 0,
+            delayed: stats.current_jobs_delayed || 0,
+            buried: stats.current_jobs_buried || 0,
+            reserved: stats.current_jobs_reserved || 0,
+            total: (stats.current_jobs_ready || 0) +
+                   (stats.current_jobs_delayed || 0) +
+                   (stats.current_jobs_buried || 0) +
+                   (stats.current_jobs_reserved || 0)
           }
         rescue Beaneater::NotFoundError
           # Tube doesn't exist yet, skip it

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: postburner
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre.15
+  version: 1.0.0.pre.16
 platform: ruby
 authors:
 - Matt Smith