postburner 1.0.0.pre.3 → 1.0.0.pre.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2f7e4e766eb885d932fe5949dc935b1efcb927894117cc427b7582e6807942f6
-  data.tar.gz: d9b2ed2f5fe0aa61cf0a28d1560f861619c3049aa0bb44fb43b6a6d662914ce8
+  metadata.gz: 38e2982c4478f99aa0d19c58ac7455cdf82ff1c3707a154d1dbd0c3a1546ceb8
+  data.tar.gz: 0d17bb9da882d8e1cf2cce7c1cddd508ddbc7531c037dd56b7e0c6d34bd40f5b
 SHA512:
-  metadata.gz: 4cb2922ca8f02625680f73232e4cc15590deecd1e6e7f792fda4b547cbb3b300de44107a055247a72ab5b56d0706631323149151708780dd9d719140255973f4
-  data.tar.gz: c749613be57e8ca5bf14671a2f8c4743e34e9839f4432305c12bfb35e34e4718b1319096ed7336fe5d083994b5da7047b706c55f3a1b735439be4bb6a28c4a55
+  metadata.gz: 28421eb19f5e288e43e9d455364906dabe0ee3ab024a230efb9aca19842ac3677c4b602788d8f0f7c67253f64e1b382d12a4f23756ff01ef2e43a5b51d478d55
+  data.tar.gz: f13d5452e5394c3e6d8b9468d8863344f35049972fbe5eac6c20cbede12ddc7e9eb2a17538bccb626a7a00a744c567d33e59558f5e7309ab3ecf775bea3db5b9

data/README.md

@@ -3,13 +3,58 @@
 Fast Beanstalkd-backed job queue with **optional PostgreSQL records** for ActiveJob.
 
 Postburner provides dual-mode job execution:
-- **Default jobs**: Fast execution via Beanstalkd only (no PostgreSQL overhead)
-- **Tracked jobs**: Full audit trail with logs, timing, errors, and statistics
+- **Fast jobs**: Fast execution via Beanstalkd
+- **Tracked jobs**: Audited jobs logs, timing, errors, and statistics
 
 Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
 
 Depends on Beanstalkd, ActiveRecord, ActiveJob, Postgres.
 
+```ruby
+# Default job (fast, no PostgreSQL overhead)
+class SendSms < ApplicationJob
+  def perform(user_id)
+    user = User.find(user_id)
+    TextMessage.welcome(
+      to: user.phone_number,
+      body: "Welcome to our app!"
+    ).deliver_now
+  end
+end
+
+# Default job with Beanstalkd configuration
+class DoSomething < ApplicationJob
+  include Postburner::Beanstalkd # optional, allow access to beanstalkd
+
+  priority 5000 # 0=highest, 65536=default, can set per job
+  ttr 30        # 30 second timeout
+
+  def perform(user_id)
+    # Do something
+  end
+end
+
+# Tracked job (full audit trail, includes Beanstalkd automatically)
+class ProcessPayment < ApplicationJob
+  include Postburner::Tracked  # ← Opt-in to tracking (includes Beanstalkd)
+
+  priority 0  # Highest priority
+  ttr 600     # 10 minute timeout
+
+  def perform(payment_id)
+    log "Processing payment #{payment_id}"
+    Payment.find(payment_id).process!
+    log! "Payment processed successfully"
+  end
+end
+
+# Run worker (bin/postburner)
+bundle exec postburner --worker default
+
+# Or with rake task
+bundle exec rake postburner:work WORKER=default
+```
+
 ## Why
 
 Postburner supports Async, Queues, Delayed, Priorities, Timeouts, and Retries from the [Backend Matrix](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html). But uniquely, priorities are per job, in addition to the class level. Timeouts are per job and class level as well, and can be extended dynamically.
@@ -23,6 +68,7 @@ Thus old-school [beanstalkd](https://beanstalkd.github.io/) is used with Postgre
 - Store the jobs outside of the database, but also persist them to disk for disaster recovery (beanstalkd binlogs)
 - Introspect the jobs either with ActiveRecord or Beanstalkd.
 - Only one worker type, that can be single/multi-process, with optional threading, and optional GC (Garbage Collection) limits (kill fork after processing N jobs).
+- Easy testing.
 
 ## Features
 
@@ -36,35 +82,17 @@ Thus old-school [beanstalkd](https://beanstalkd.github.io/) is used with Postgre
 
 ## Quick Start
 
-```bash
-sudo apt-get install beanstalkd # OR brew install beanstalkd
-
-# Start beanstalkd (in-memory only)
-beanstalkd -l 127.0.0.1 -p 11300
-
-# OR with persistence (recommended for production)
-mkdir -p /var/lib/beanstalkd
-beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
-```
-
 ```ruby
 # Gemfile
-gem 'postburner', '~> 1.0.0.pre.2'
-```
+gem 'postburner', '~> 1.0.0.pre.3'
 
-```bash
-rails generate postburner:install
-rails db:migrate
-```
-
-```ruby
 # config/application.rb
 config.active_job.queue_adapter = :postburner
 ```
 
 ```yaml
 # config/postburner.yml
-production:  # <- environment config, i.e. defaults
+development:  # <- environment config, i.e. defaults
   beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
   default_forks: 2
   default_threads: 10
@@ -91,48 +119,16 @@ production:  # <- environment config, i.e. defaults
         - video
 ```
 
-**Usage:**
-
-```ruby
-# Default job (fast, no PostgreSQL overhead)
-class SendSms < ApplicationJob
-  def perform(user_id)
-    user = User.find(user_id)
-    TextMessage.welcome(
-      to: user.phone_number,
-      body: "Welcome to our app!"
-    ).deliver_now
-  end
-end
-
-# Default job with Beanstalkd configuration
-class DoSomething < ApplicationJob
-  include Postburner::Beanstalkd # optional, allow access to beanstalkd
-
-  priority 5000 # 0=highest, 65536=default, can set per job
-  ttr 30        # 30 second timeout
-
-  def perform(user_id)
-    # Do something
-  end
-end
+```bash
+sudo apt-get install beanstalkd # OR brew install beanstalkd
 
-# Tracked job (full audit trail, includes Beanstalkd automatically)
-class ProcessPayment < ApplicationJob
-  include Postburner::Tracked  # ← Opt-in to tracking (includes Beanstalkd)
+beanstalkd -l 127.0.0.1 -p 11300 # Start beanstalkd
 
-  priority 0  # Highest priority
-  ttr 600     # 10 minute timeout
+bundle exec rails generate postburner:install
+bundle exec rails db:migrate
 
-  def perform(payment_id)
-    log "Processing payment #{payment_id}"
-    Payment.find(payment_id).process!
-    log! "Payment processed successfully"
-  end
-end
-
-# Run worker
-bundle exec postburner --worker default
+bundle exec postburner           # start with bin/postburner
+bundle exec rake postburner:work # or with rake task
 ```
 
 ## Table of Contents
@@ -159,11 +155,14 @@ The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol
 
 Here is a picture of the typical job lifecycle:
 
+```
    put            reserve               delete
-  -----> [READY] ---------> [RESERVED] --------> *poof*
+  -----> [READY] ---------> [RESERVED] --------> *poof*`
+```
 
 Here is a picture with more possibilities:
 
+```
    put with delay               release with delay
   ----------------> [DELAYED] <------------.
                         |                   |
@@ -182,13 +181,12 @@ Here is a picture with more possibilities:
                        |
                        |  delete
                         `--------> *poof*
+```
 
 ### Binlogs
 
 Beanstalkd lets you persist jobs to disk in case of a crash or restart. Just restart beanstalkd and the jobs will be back in the queue.
 
-**Setup:**
-
 ```bash
 # Create binlog directory
 sudo mkdir -p /var/lib/beanstalkd
@@ -198,7 +196,7 @@ sudo chown beanstalkd:beanstalkd /var/lib/beanstalkd  # If running as service
 beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
 ```
 
-**Configuration options:**
+**Other options:**
 
 ```bash
 # Basic persistence
@@ -214,7 +212,7 @@ beanstalkd -b /var/lib/beanstalkd -f 200  # fsync at most every 200ms (default:
 beanstalkd -b /var/lib/beanstalkd -F
 ```
 
-**Options:**
+**Beanstalkd switches:**
 - `-b <dir>` - Enable binlog persistence in specified directory
 - `-s <bytes>` - Maximum size of each binlog file (requires `-b`)
 - `-f <ms>` - Call fsync at most once every N milliseconds (requires `-b`, default: 50ms)
@@ -855,6 +853,52 @@ bin/postburner --worker general    # Run the 'general' worker
 bin/postburner --worker general --queues default,mailers  # Only process specific queues
 ```
 
+**Rake task:**
+```bash
+bundle exec rake postburner:work                          # Auto-select worker
+bundle exec rake postburner:work WORKER=general           # Specific worker
+bundle exec rake postburner:work WORKER=general QUEUES=default,mailers  # Filter queues
+bundle exec rake postburner:work CONFIG=config/custom.yml # Custom config
+```
+
+### Programmatic Worker Control
+
+Use `Postburner::Runner` to start workers programmatically from Ruby code:
+
+```ruby
+# Basic usage
+runner = Postburner::Runner.new(
+  config: 'config/postburner.yml',
+  env: 'production',
+  worker: 'default'
+)
+runner.run
+
+# With queue filtering
+runner = Postburner::Runner.new(
+  worker: 'general',
+  queues: ['default', 'mailers']
+)
+runner.run
+
+# From environment variables (Rake task pattern)
+runner = Postburner::Runner.new(
+  config: ENV['CONFIG'] || 'config/postburner.yml',
+  env: Rails.env,
+  worker: ENV['WORKER'],
+  queues: ENV['QUEUES']&.split(',')
+)
+runner.run
+```
+
+**Options:**
+- `:config` - Path to YAML configuration file (default: `'config/postburner.yml'`)
+- `:env` - Environment name (default: `RAILS_ENV` or `'development'`)
+- `:worker` - Worker name from config (required if multiple workers defined)
+- `:queues` - Array of queue names to filter (overrides config queues)
+
+This provides a unified interface used by both `bin/postburner` and `rake postburner:work`, making it easy to integrate Postburner workers into custom deployment scripts or process managers.
+
 ### Running Workers in Separate Processes
 
 For production deployments, run different workers in separate OS processes for isolation and resource allocation:
@@ -1296,7 +1340,7 @@ Direct access to Beanstalkd for advanced operations:
 job.bkid  # => 12345
 
 # Access Beaneater job object
-job.beanstalk_job.stats
+job.bk.stats
 # => {"id"=>12345, "tube"=>"critical", "state"=>"ready", ...}
 
 # Connection management

data/app/concerns/postburner/commands.rb

@@ -0,0 +1,143 @@
+# frozen_string_literal: true
+
+module Postburner
+  # Concern providing Beanstalkd command methods for Postburner jobs.
+  #
+  # Provides methods for interacting with Beanstalkd queue operations such as
+  # deleting, kicking, and extending TTR. All methods handle connection retries
+  # and gracefully handle missing bkid (e.g., in test mode).
+  #
+  # @example Basic Beanstalkd operations
+  #   class MyJob < Postburner::Job
+  #     def perform(args)
+  #       # ... work ...
+  #       extend!  # Extend TTR during long operation
+  #     end
+  #   end
+  #
+  #   job.delete!  # Remove from Beanstalkd
+  #   job.kick!    # Retry buried job
+  #
+  module Commands
+    extend ActiveSupport::Concern
+
+    # Kicks a buried job back into the ready queue in Beanstalkd.
+    #
+    # This is a Beanstalkd operation used to retry jobs that were buried due to
+    # repeated failures or explicit burial. Does not modify the ActiveRecord model.
+    #
+    # Automatically retries with fresh connection if Beanstalkd connection is stale.
+    #
+    # @return [void]
+    #
+    # @raise [Beaneater::NotConnected] if Beanstalkd connection fails after retry
+    #
+    # @note Does nothing if job has no bkid (e.g., in test mode)
+    # @note Only works on buried jobs - see Beanstalkd documentation
+    #
+    # @example
+    #   job.kick!  # Moves buried job back to ready queue
+    #
+    # @see #delete!
+    # @see #bk
+    #
+    def kick!
+      return unless self.bk
+      begin
+        self.bk.kick
+      rescue Beaneater::NotConnected => e
+        self.bk!.kick
+      end
+    end
+
+    # Extends the job's time-to-run (TTR) in Beanstalkd.
+    #
+    # Calls touch on the Beanstalkd job, extending the TTR by the original
+    # TTR value. Use this during long-running operations to prevent the job
+    # from timing out.
+    #
+    # @return [void]
+    #
+    # @note Does nothing if job has no bkid (e.g., in test mode)
+    #
+    # @example Process large file line by line
+    #   def perform(args)
+    #     file = File.find(args['file_id'])
+    #     file.each_line do |line|
+    #       # ... process line ...
+    #       extend!  # Extend TTR to prevent timeout
+    #     end
+    #   end
+    #
+    # @see #bk
+    #
+    def extend!
+      return unless self.bk
+      begin
+        self.bk.touch
+      rescue Beaneater::NotConnected => e
+        self.bk!.touch
+      end
+    end
+
+    # Deletes the job from the Beanstalkd queue.
+    #
+    # This is a Beanstalkd operation that removes the job from the queue but
+    # does NOT destroy the ActiveRecord model. Use {#destroy} or {#remove!} to
+    # also update the database record.
+    #
+    # Automatically retries with fresh connection if Beanstalkd connection is stale.
+    # Called automatically by before_destroy callback when using {#destroy}.
+    #
+    # @return [void]
+    #
+    # @raise [Beaneater::NotConnected] if Beanstalkd connection fails after retry
+    #
+    # @note Does nothing if job has no bkid (e.g., in test mode)
+    # @note Does not modify ActiveRecord model - only affects Beanstalkd
+    #
+    # @example
+    #   job.delete!  # Removes from Beanstalkd queue only
+    #   job.reload   # Job still exists in database
+    #
+    # @see #remove!
+    # @see #destroy
+    # @see #bk
+    #
+    def delete!
+      return unless self.bk
+      begin
+        self.bk.delete
+      rescue Beaneater::NotConnected => e
+        self.bk!.delete
+      end
+    end
+
+    # Soft-deletes the job by removing from Beanstalkd and setting removed_at timestamp.
+    #
+    # Unlike {#destroy}, this preserves the job record in the database for audit trails
+    # while removing it from the Beanstalkd queue and marking it as removed.
+    #
+    # @return [void]
+    #
+    # @raise [Beaneater::NotConnected] if Beanstalkd connection fails
+    #
+    # @note Idempotent - does nothing if already removed
+    # @note Does not destroy ActiveRecord model - only soft deletes
+    #
+    # @example
+    #   job.remove!
+    #   job.removed_at  # => 2025-10-31 12:34:56 UTC
+    #   job.persisted?  # => true (still in database)
+    #
+    # @see #delete!
+    # @see #destroy
+    #
+    def remove!
+      return if self.removed_at
+      self.delete!
+      self.update_column(:removed_at, Time.zone.now)
+    end
+
+  end
+end

data/app/concerns/postburner/execution.rb

@@ -0,0 +1,190 @@
+# frozen_string_literal: true
+
+module Postburner
+  # Concern providing job execution methods for Postburner jobs.
+  #
+  # Handles the full job execution lifecycle including validation, callbacks,
+  # timing tracking, error handling, and state management. Provides both the
+  # class-level entry point for workers and the instance-level execution logic.
+  #
+  # @example Direct execution
+  #   Postburner::Job.perform(job.id)
+  #
+  # @example Instance execution
+  #   job.perform!(job.args)
+  #
+  module Execution
+    extend ActiveSupport::Concern
+
+    class_methods do
+      # Executes a job by ID, delegating to the current queue strategy.
+      #
+      # Loads the job from database by ID and delegates execution to the current
+      # queue strategy's {handle_perform!} method. This provides a unified API
+      # for job execution regardless of strategy (async, test, or null).
+      #
+      # Called automatically by Backburner workers in production. Can also be
+      # called manually for test/null strategies to trigger execution.
+      #
+      # @param id [Integer] Job ID to execute
+      # @param _ [Hash] Unused Backburner metadata parameter
+      #
+      # @return [void]
+      #
+      # @example Backburner automatic execution (production)
+      #   # Jobs execute in tube: backburner.worker.queue.backburner-jobs
+      #   # Backburner calls: Postburner::Job.perform(job_id)
+      #
+      # @example Manual execution with NullQueue
+      #   Postburner.null_strategy!
+      #   job = MyJob.create!(args: {})
+      #   job.queue!(delay: 1.hour)
+      #   Postburner::Job.perform(job.id)  # Time travels and executes
+      #
+      # @note Strategy-aware: delegates to Postburner.queue_strategy.handle_perform!
+      # @note For NullQueue, automatically handles time travel for scheduled jobs
+      #
+      # @see #perform!
+      # @see Queue.handle_perform!
+      # @see NullQueue.handle_perform!
+      #
+      def perform(id, _={})
+        job = nil
+        begin
+          job = self.find(id)
+        rescue ActiveRecord::RecordNotFound => e
+          Rails.logger.warn <<-MSG
+[Postburner::Job] [#{id}] Not Found.
+          MSG
+        end
+        #job.perform!(job.args)
+        Postburner.queue_strategy.handle_perform!(job)
+      end
+    end
+
+    # Executes the job with full lifecycle management and error handling.
+    #
+    # This is the main execution method called by Backburner workers or test strategies.
+    # Performs validation checks, executes callbacks, calls the subclass {#perform} method,
+    # tracks timing and statistics, and handles errors.
+    #
+    # Execution flow:
+    # 1. Runs attempt callbacks (fires on every retry)
+    # 2. Updates attempting metadata (attempting_at, attempts, lag)
+    # 3. Validates job state (queued, not processed, not removed, not premature)
+    # 4. Runs processing callbacks
+    # 5. Calls subclass {#perform} method
+    # 6. Runs processed callbacks (only on success)
+    # 7. Updates completion metadata (processed_at, duration)
+    # 8. Logs and tracks any exceptions
+    #
+    # @param args [Hash] Arguments to pass to {#perform} method
+    #
+    # @return [void]
+    #
+    # @raise [Exception] Any exception raised by {#perform} is logged and re-raised
+    #
+    # @note Does not execute if queued_at is nil, in the future, already processed, or removed
+    # @note Premature execution (before run_at) is delegated to queue strategy
+    #
+    # @see #perform
+    # @see Postburner.queue_strategy
+    # @see Callbacks
+    #
+    def perform!(args={})
+      run_callbacks :attempt do
+        self.attempting
+
+        self.update_columns(
+          attempting_at:  self.attempting_at,
+          attempts:       self.attempts,
+          attempt_count:  self.attempts.length,
+          lag:            self.lag,
+          processing_at:  Time.zone.now,
+        )
+
+        begin
+          if self.queued_at.nil?
+            self.log! "Not Queued", level: :error
+            return
+          end
+
+          if self.queued_at > Time.zone.now
+            self.log! "Future Queued", level: :error
+            return
+          end
+
+          if self.processed_at.present?
+            self.log! "Already Processed", level: :error
+            self.delete!
+            return
+          end
+
+          if self.removed_at.present?
+            self.log! "Removed", level: :error
+            return
+          end
+
+          if self.run_at && self.run_at.to_i > Time.zone.now.to_i
+            Postburner.queue_strategy.handle_premature_perform(self)
+            return
+          end
+
+          self.log!("START (bkid #{self.bkid})")
+
+          run_callbacks :processing do
+            begin
+              self.perform(args)
+            rescue Exception => exception
+              self.persist_metadata!
+              self.log! '[Postburner] Exception raised during perform prevented completion.'
+              raise exception
+            end
+          end
+
+          self.log!("DONE (bkid #{self.bkid})")
+
+          begin
+            now = Time.zone.now
+            _duration =  (now - self.processing_at) * 1000 rescue nil
+
+            run_callbacks :processed do
+              persist_metadata!(
+                processed_at: now,
+                duration:     _duration,
+              )
+            end
+          rescue Exception => e
+            self.log_exception!(e)
+            self.log! '[Postburner] Could not set data after processing.'
+            # TODO README doesn't retry if Postburner is to blame
+          end
+
+        rescue Exception => exception
+          self.log_exception!(exception)
+          raise exception
+        end
+      end # run_callbacks :attempt
+
+    end
+
+    private
+
+    # Records an attempt and calculates execution lag.
+    #
+    # Appends current time to attempts array, sets attempting_at on first attempt,
+    # and calculates lag (delay between intended and actual execution time).
+    #
+    # @return [Time] Current timestamp
+    #
+    # @api private
+    #
+    def attempting
+      now = Time.zone.now
+      self.attempts << now
+      self.attempting_at ||= now
+      self.lag ||= (self.attempting_at - self.intended_at) * 1000 rescue nil
+      now
+    end
+  end
+end