postburner 1.0.0.pre.2 → 1.0.0.pre.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5195def95321261f581896c066a199022239c6a2c9eb86504d0beb8ce176b111
-  data.tar.gz: cf51e2b5f1fd16938d61cc2daa626e08b3340cbe580cfcae1b42d560626fb643
+  metadata.gz: 2f7e4e766eb885d932fe5949dc935b1efcb927894117cc427b7582e6807942f6
+  data.tar.gz: d9b2ed2f5fe0aa61cf0a28d1560f861619c3049aa0bb44fb43b6a6d662914ce8
 SHA512:
-  metadata.gz: e8b2e7bbba0c0dc19ddcd5b6747a20aa0e59509fa05211e616b4eacc177bc9881fe2a873eeb407f29a3697cad280525f4dd71b2cfd412b484a876b23689c432c
-  data.tar.gz: 7f954cd5c0241e4b9af5a123ac6de7f3482fd4426a7e9f924975fc786b11b1fd48ba2e7d6942f575998ab9fb147e27e0efe0aeb4468e5fc09caa9555a009c527
+  metadata.gz: 4cb2922ca8f02625680f73232e4cc15590deecd1e6e7f792fda4b547cbb3b300de44107a055247a72ab5b56d0706631323149151708780dd9d719140255973f4
+  data.tar.gz: c749613be57e8ca5bf14671a2f8c4743e34e9839f4432305c12bfb35e34e4718b1319096ed7336fe5d083994b5da7047b706c55f3a1b735439be4bb6a28c4a55

data/README.md

@@ -1,6 +1,6 @@
 # Postburner
 
-Fast Beanstalkd-backed job queue with **optional PostgreSQL audit trail** for ActiveJob.
+Fast Beanstalkd-backed job queue with **optional PostgreSQL records** for ActiveJob.
 
 Postburner provides dual-mode job execution:
 - **Default jobs**: Fast execution via Beanstalkd only (no PostgreSQL overhead)
@@ -8,6 +8,22 @@ Postburner provides dual-mode job execution:
 
 Built for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
 
+Depends on Beanstalkd, ActiveRecord, ActiveJob, Postgres.
+
+## Why
+
+Postburner supports Async, Queues, Delayed, Priorities, Timeouts, and Retries from the [Backend Matrix](https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters.html). But uniquely, priorities are per job, in addition to the class level. Timeouts are per job and class level as well, and can be extended dynamically.
+
+Postburner is inspired by the [Backburner](https://github.com/nesquena/backburner) gem i.e. "burner", Postgres i.e. "Post", and the database backends (SolidQueue, Que, etc), and the in memory/redis backends (Sidekiq, Resque, etc). And puma's concurrency model.
+
+Thus old-school [beanstalkd](https://beanstalkd.github.io/) is used with PostgreSQL to cover all the cases we care about.
+- Fast when you want it (light, ephemeral jobs like delayed turbo_stream rendering, communications)
+- Tracked when you need it (critical operations like payments, and processes.
+- Able to record jobs before and after execution in PostgreSQL, with foreign keys and constraints.
+- Store the jobs outside of the database, but also persist them to disk for disaster recovery (beanstalkd binlogs)
+- Introspect the jobs either with ActiveRecord or Beanstalkd.
+- Only one worker type, that can be single/multi-process, with optional threading, and optional GC (Garbage Collection) limits (kill fork after processing N jobs).
+
 ## Features
 
 - **ActiveJob native** - Works seamlessly with Rails, ActionMailer, ActiveStorage
@@ -20,8 +36,6 @@ Built for production environments where you want fast background processing for
 
 ## Quick Start
 
-**Installation:**
-
 ```bash
 sudo apt-get install beanstalkd # OR brew install beanstalkd
 
@@ -35,17 +49,14 @@ beanstalkd -l 127.0.0.1 -p 11300 -b /var/lib/beanstalkd
 
 ```ruby
 # Gemfile
-gem 'postburner', '~> 1.0.0.pre.1'
+gem 'postburner', '~> 1.0.0.pre.2'
 ```
 
 ```bash
-bundle install
 rails generate postburner:install
 rails db:migrate
 ```
 
-**Configuration:**
-
 ```ruby
 # config/application.rb
 config.active_job.queue_adapter = :postburner
@@ -53,34 +64,56 @@ config.active_job.queue_adapter = :postburner
 
 ```yaml
 # config/postburner.yml
-production:
+production:  # <- environment config, i.e. defaults
   beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
-  worker_type: simple
-  queues:
-    default: {}
-    critical: {}
-    mailers: {}
+  default_forks: 2
+  default_threads: 10
+  default_gc_limit: 500
+
+  workers:  # <- worker config, i.e. overrides
+    default:
+      threads: 16        # Overrides default_threads
+      queues:
+        - default
+        - mailers
+    critical:
+      forks: 4           # Overrides default_forks
+      threads: 8         # Overrides default_threads
+      gc_limit: 24       # Overrides default_gc_limit
+      queues:
+        - payments
+    slow:
+      forks: 4           # Overrides default_forks
+      threads: 1         # Overrides default_threads
+      gc_limit: 1        # Overrides default_gc_limit
+      queues:
+        - imports
+        - video
 ```
 
 **Usage:**
 
 ```ruby
 # Default job (fast, no PostgreSQL overhead)
-class SendEmail < ApplicationJob
+class SendSms < ApplicationJob
   def perform(user_id)
-    UserMailer.welcome(user_id).deliver_now
+    user = User.find(user_id)
+    TextMessage.welcome(
+      to: user.phone_number,
+      body: "Welcome to our app!"
+    ).deliver_now
   end
 end
 
 # Default job with Beanstalkd configuration
-class SendEmail < ApplicationJob
-  include Postburner::Beanstalkd
+class DoSomething < ApplicationJob
+  include Postburner::Beanstalkd # optional, allow access to beanstalkd
 
-  queue_priority 100  # Custom priority
-  queue_ttr 300       # 5 minute timeout
+  priority 5000 # 0=highest, 65536=default, can set per job
+  ttr 30        # 30 second timeout
 
   def perform(user_id)
-    UserMailer.welcome(user_id).deliver_now
+    # Do something
   end
 end
 
@@ -88,8 +121,8 @@ end
 class ProcessPayment < ApplicationJob
   include Postburner::Tracked  # ← Opt-in to tracking (includes Beanstalkd)
 
-  queue_priority 0  # Highest priority
-  queue_ttr 600     # 10 minute timeout
+  priority 0  # Highest priority
+  ttr 600     # 10 minute timeout
 
   def perform(payment_id)
     log "Processing payment #{payment_id}"
@@ -99,7 +132,7 @@ class ProcessPayment < ApplicationJob
 end
 
 # Run worker
-bin/postburner --config config/postburner.yml --env production
+bundle exec postburner --worker default
 ```
 
 ## Table of Contents
@@ -122,7 +155,33 @@ bin/postburner --config config/postburner.yml --env production
 
 Beanstalkd is a simple, fast, and reliable queue system. It is a good choice for production environments where you want fast background processing for most jobs, but comprehensive auditing for critical operations.
 
-The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt) reads more like a README than a protocol. Check it out and you will instantly understand how it works.
+The [protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt) reads more like a README than a protocol. Check it out and you will instantly understand how it works. Here is a help
+
+Here is a picture of the typical job lifecycle:
+
+   put            reserve               delete
+  -----> [READY] ---------> [RESERVED] --------> *poof*
+
+Here is a picture with more possibilities:
+
+   put with delay               release with delay
+  ----------------> [DELAYED] <------------.
+                        |                   |
+                        | (time passes)     |
+                        |                   |
+   put                  v     reserve       |       delete
+  -----------------> [READY] ---------> [RESERVED] --------> *poof*
+                       ^  ^                |  |
+                       |   \  release      |  |
+                       |    `-------------'   |
+                       |                      |
+                       | kick                 |
+                       |                      |
+                       |       bury           |
+                    [BURIED] <---------------'
+                       |
+                       |  delete
+                        `--------> *poof*
 
 ### Binlogs
 
@@ -181,14 +240,14 @@ class ProcessPayment < ApplicationJob
   include Postburner::Beanstalkd
 
   queue_as :critical
-  queue_priority 0  # Highest priority - processed immediately
+  priority 0  # Highest priority - processed immediately
 end
 
 class SendEmail < ApplicationJob
   include Postburner::Beanstalkd
 
   queue_as :mailers
-  queue_priority 1000  # Lower priority - processed after critical jobs
+  priority 1000  # Lower priority - processed after critical jobs
 end
 ```
 
@@ -197,7 +256,7 @@ end
 ```ruby
 class CriticalJob < Postburner::Job
   queue 'critical'
-  queue_priority 0  # Highest priority
+  priority 0  # Highest priority
 
   def perform(args)
     # Critical business logic
@@ -206,7 +265,7 @@ end
 
 class BackgroundTask < Postburner::Job
   queue 'default'
-  queue_priority 5000  # Lower priority
+  priority 5000  # Lower priority
 
   def perform(args)
     # Non-urgent background work
@@ -239,7 +298,7 @@ production:
 ```ruby
 class ProcessOrder < Postburner::Job
   queue 'orders'
-  queue_priority 100  # Default
+  priority 100  # Default
 
   before_enqueue :adjust_priority
 
@@ -252,7 +311,7 @@ class ProcessOrder < Postburner::Job
 
   def adjust_priority
     if args['urgent']
-      self.queue_priority = 0  # Override to highest priority
+      self.priority = 0  # Override to highest priority
     end
   end
 end
@@ -284,22 +343,22 @@ class ProcessPayment < ApplicationJob
   include Postburner::Beanstalkd
 
   queue_as :critical
-  queue_priority 0
-  queue_ttr 300  # 5 minutes to complete
+  priority 0
+  ttr 300  # 5 minutes to complete
 end
 
 class QuickEmail < ApplicationJob
   include Postburner::Beanstalkd
 
   queue_as :mailers
-  queue_ttr 60  # 1 minute for fast email jobs
+  ttr 60  # 1 minute for fast email jobs
 end
 
 class LongRunningReport < ApplicationJob
   include Postburner::Beanstalkd
 
   queue_as :reports
-  queue_ttr 3600  # 1 hour for complex reports
+  ttr 3600  # 1 hour for complex reports
 end
 ```
 
@@ -308,7 +367,7 @@ end
 ```ruby
 class DataImport < Postburner::Job
   queue 'imports'
-  queue_ttr 1800  # 30 minutes (equivalent to queue_ttr)
+  ttr 1800  # 30 minutes (equivalent to queue_ttr)
 
   def perform(args)
     # Long-running import logic
@@ -329,7 +388,7 @@ class ProcessImport < ApplicationJob
   include Postburner::Tracked  # Includes Beanstalkd and enables extend!
 
   queue_as :imports
-  queue_ttr 300  # 5 minutes initial TTR
+  ttr 300  # 5 minutes initial TTR
 
   def perform(file_id)
     file = ImportFile.find(file_id)
@@ -351,7 +410,7 @@ end
 ```ruby
 class LargeDataProcessor < Postburner::Job
   queue 'processing'
-  queue_ttr 600  # 10 minutes
+  ttr 600  # 10 minutes
 
   def perform(args)
     dataset = Dataset.find(args['dataset_id'])
@@ -370,7 +429,7 @@ end
 
 - Calls Beanstalkd's `touch` command on the reserved job
 - Resets the TTR countdown to the original TTR value from the current moment
-- Example: Job has `queue_ttr 300`. After 200 seconds, calling `extend!` gives you another 300 seconds (not just 100)
+- Example: Job has `ttr 300`. After 200 seconds, calling `extend!` gives you another 300 seconds (not just 100)
 - Can be called multiple times during job execution
 - Useful for iterative processing where each iteration is quick but total time is unpredictable
 
@@ -512,8 +571,8 @@ class SendWelcomeEmail < ApplicationJob
   include Postburner::Beanstalkd
 
   queue_as :mailers
-  queue_priority 100    # Lower = higher priority (0 is highest)
-  queue_ttr 300         # Time-to-run in seconds (5 minutes)
+  priority 100    # Lower = higher priority (0 is highest)
+  ttr 300         # Time-to-run in seconds (5 minutes)
 
   def perform(user_id)
     UserMailer.welcome(user_id).deliver_now
@@ -540,8 +599,8 @@ class ProcessPayment < ApplicationJob
   include Postburner::Tracked  # ← Opt-in to PostgreSQL tracking (includes Beanstalkd)
 
   queue_as :critical
-  queue_priority 0              # Highest priority (Beanstalkd included automatically)
-  queue_ttr 600                 # 10 minute timeout
+  priority 0              # Highest priority (Beanstalkd included automatically)
+  ttr 600                 # 10 minute timeout
   retry_on StandardError, wait: :exponentially_longer, attempts: 5
 
   def perform(payment_id)
@@ -603,8 +662,8 @@ Direct `Postburner::Job` subclasses are **always tracked**:
 ```ruby
 class RunDonation < Postburner::Job
   queue 'critical'
-  queue_priority 0
-  queue_max_job_retries 0
+  priority 0
+  max_retries 0
 
   def perform(args)
     donation = Donation.find(args['donation_id'])
@@ -639,15 +698,15 @@ job.queue!
 
 # Set dynamically after creation
 job = RunDonation.create!(args: { 'donation_id' => 456 })
-job.queue_priority = 2000
-job.queue_ttr = 300
+job.priority = 2000
+job.ttr = 300
 job.queue!
 
 # Use in before_enqueue callback for conditional behavior
 class ProcessOrder < Postburner::Job
   queue 'orders'
-  queue_priority 100   # Default priority
-  queue_ttr 120  # Default TTR
+  priority 100   # Default priority
+  ttr 120  # Default TTR
 
   before_enqueue :set_priority_based_on_urgency
 
@@ -660,11 +719,11 @@ class ProcessOrder < Postburner::Job
 
   def set_priority_based_on_urgency
     if args['express_shipping']
-      self.queue_priority = 0    # High priority for express orders
-      self.queue_ttr = 600       # Allow 10 minutes to complete
+      self.priority = 0    # High priority for express orders
+      self.ttr = 600       # Allow 10 minutes to complete
     else
-      self.queue_priority = 1000 # Low priority for standard orders
-      self.queue_ttr = 120       # Standard 2 minute timeout
+      self.priority = 1000 # Low priority for standard orders
+      self.ttr = 120       # Standard 2 minute timeout
     end
   end
 end
@@ -686,24 +745,25 @@ Postburner uses named worker configurations to support different deployment patt
 Configure multiple named workers with different concurrency profiles:
 
 ```yaml
-production:
+production:  # <- environment config, i.e. defaults
   beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
+  default_forks: 2
+  default_threads: 10
+  default_gc_limit: 5000
 
-  workers:
+  workers:  # <- worker config, i.e. overrides
     # Heavy, memory-intensive jobs - more processes, fewer threads
     imports:
-      default_forks: 4
-      default_threads: 1
-      default_gc_limit: 500
+      forks: 4           # Overrides default_forks
+      threads: 1         # Overrides default_threads
+      gc_limit: 500      # Overrides default_gc_limit
       queues:
         - imports
         - data_processing
 
     # General jobs - fewer processes, many threads
     general:
-      default_forks: 2
-      default_threads: 100
-      default_gc_limit: 5000
+      threads: 100       # Overrides default_threads (forks uses default_forks=2)
       queues:
         - default
         - mailers
@@ -722,12 +782,12 @@ This gives you a natural progression from simple single-threaded development to
 
 **Development (single worker, single-threaded):**
 ```yaml
-development:
+development:  # <- environment config
   beanstalk_url: beanstalk://localhost:11300
 
-  workers:
+  workers:  # <- worker config
     default:
-      # Defaults: forks=0, threads=1, gc_limit=nil
+      # Uses env defaults: forks=0, threads=1, gc_limit=nil
       queues:
         - default
         - mailers
@@ -735,13 +795,14 @@ development:
 
 **Staging (single worker, multi-threaded):**
 ```yaml
-staging:
+staging:  # <- environment config
   beanstalk_url: beanstalk://localhost:11300
+  default_threads: 10
+  default_gc_limit: 5000
 
-  workers:
+  workers:  # <- worker config
     default:
-      default_threads: 10
-      default_gc_limit: 5000
+      # Uses env defaults: default_threads=10, default_gc_limit=5000
       queues:
         - critical
         - default
@@ -750,22 +811,25 @@ staging:
 
 **Production (multiple workers with different profiles):**
 ```yaml
-production:
+production:  # <- environment config, i.e. defaults
   beanstalk_url: <%= ENV['BEANSTALK_URL'] %>
+  default_forks: 2
+  default_threads: 10
+  default_gc_limit: 5000
 
-  workers:
+  workers:  # <- worker config, i.e. overrides
     imports:
-      default_forks: 4      # 4 processes
-      default_threads: 1    # 1 thread per process = 4 concurrent jobs
-      default_gc_limit: 500
+      forks: 4           # Overrides default_forks (4 processes)
+      threads: 1         # Overrides default_threads (1 thread per process = 4 concurrent jobs)
+      gc_limit: 500      # Overrides default_gc_limit
       queues:
         - imports
         - data_processing
 
     general:
-      default_forks: 2      # 2 processes
-      default_threads: 100  # 100 threads per process = 200 concurrent jobs
-      default_gc_limit: 5000
+      # forks uses default_forks=2 (2 processes)
+      threads: 100       # Overrides default_threads (100 threads per process = 200 concurrent jobs)
+      # gc_limit uses default_gc_limit=5000
       queues:
         - default
         - mailers
@@ -872,7 +936,7 @@ Parent Process
 
 ### GC Limits
 
-Set `default_gc_limit` per worker to automatically restart after processing N jobs.
+Set `default_gc_limit` at environment level or `gc_limit` per worker to automatically restart after processing N jobs.
 
 - Worker processes N jobs
 - Worker exits with code 99
@@ -880,20 +944,23 @@ Set `default_gc_limit` per worker to automatically restart after processing N jo
 - In multi-process mode (forks: 1+): parent process automatically restarts just that fork
 
 ```yaml
-production:
-  workers:
+production:  # <- environment config
+  default_forks: 2
+  default_threads: 10
+  default_gc_limit: 5000
+
+  workers:  # <- worker config
     imports:
-      default_forks: 4
-      default_threads: 1
-      default_gc_limit: 500  # Restart after 500 jobs (memory-intensive)
+      forks: 4           # Overrides default_forks
+      threads: 1         # Overrides default_threads
+      gc_limit: 500      # Overrides default_gc_limit (restart after 500 jobs, memory-intensive)
       queues:
         - imports
         - data_processing
 
     general:
-      default_forks: 2
-      default_threads: 100
-      default_gc_limit: 5000  # Restart after 5000 jobs
+      # Uses default_forks=2, default_threads=100, default_gc_limit=5000 (restart after 5000 jobs)
+      threads: 100       # Overrides default_threads
       queues:
         - default
         - mailers
@@ -913,48 +980,64 @@ production:
 default: &default
   beanstalk_url: <%= ENV['BEANSTALK_URL'] || 'beanstalk://localhost:11300' %>
 
-development:
+development:  # <- environment config
   <<: *default
-  # Defaults: forks: 0, threads: 1, gc_limit: nil
-  queues:
-    default: {}
-    mailers: {}
+  # Env defaults: forks=0, threads=1, gc_limit=nil
+
+  workers:  # <- worker config
+    default:
+      queues:
+        - default
+        - mailers
 
-test:
+test:  # <- environment config
   <<: *default
-  # Defaults: forks: 0, threads: 1, gc_limit: nil
-  queues:
-    default: {}
+  # Env defaults: forks=0, threads=1, gc_limit=nil
 
-staging:
+  workers:  # <- worker config
+    default:
+      queues:
+        - default
+
+staging:  # <- environment config
   <<: *default
   default_threads: 10      # Multi-threaded, single process
   default_gc_limit: 5000
-  queues:
-    default: {}
-    mailers: {}
 
-production:
+  workers:  # <- worker config
+    default:
+      # Uses env defaults: default_threads=10, default_gc_limit=5000
+      queues:
+        - default
+        - mailers
+
+production:  # <- environment config, i.e. defaults
   <<: *default
-  default_forks: 4         # Puma-style: multiple processes
-  default_threads: 10      # 10 threads per fork
-  default_gc_limit: 5000
+  default_forks: 2         # Default for workers
+  default_threads: 10      # Default for workers
+  default_gc_limit: 5000   # Default for workers
 
-  queues:
+  workers:  # <- worker config, i.e. overrides
     critical:
-      forks: 1
-      threads: 1           # 1 concurrent job
-      gc_limit: 100
+      forks: 1             # Overrides default_forks
+      threads: 1           # Overrides default_threads (1 concurrent job)
+      gc_limit: 100        # Overrides default_gc_limit
+      queues:
+        - critical
 
     default:
-      forks: 4
-      threads: 10          # 40 total concurrent jobs (4 × 10)
-      gc_limit: 1000
+      forks: 4             # Overrides default_forks
+      threads: 10          # Overrides default_threads (40 total concurrent jobs: 4 × 10)
+      gc_limit: 1000       # Overrides default_gc_limit
+      queues:
+        - default
 
     mailers:
-      forks: 2
-      threads: 5           # 10 total email senders (2 × 5)
-      gc_limit: 500
+      # forks uses default_forks=2
+      threads: 5           # Overrides default_threads (10 total email senders: 2 × 5)
+      gc_limit: 500        # Overrides default_gc_limit
+      queues:
+        - mailers
 ```
 
 ### Queue Configuration Methods
@@ -964,10 +1047,10 @@ For `Postburner::Job` subclasses (and backwards compatibility):
 ```ruby
 class CriticalJob < Postburner::Job
   queue 'critical'                  # Beanstalkd tube name
-  queue_priority 0                  # Lower = higher priority (0 is highest)
-  queue_ttr 300                     # TTR (time-to-run) in seconds
-  queue_max_job_retries 3           # Max retry attempts
-  queue_retry_delay 5               # Fixed delay: 5 seconds between retries
+  priority 0                  # Lower = higher priority (0 is highest)
+  ttr 300                     # TTR (time-to-run) in seconds
+  max_retries 3           # Max retry attempts
+  retry_delay 5               # Fixed delay: 5 seconds between retries
 
   def perform(args)
     # ...
@@ -980,8 +1063,8 @@ end
 ```ruby
 class BackgroundTask < Postburner::Job
   queue 'default'
-  queue_max_job_retries 5
-  queue_retry_delay ->(retries) { 2 ** retries }  # 2s, 4s, 8s, 16s, 32s
+  max_retries 5
+  retry_delay ->(retries) { 2 ** retries }  # 2s, 4s, 8s, 16s, 32s
 
   def perform(args)
     # ...
@@ -1341,6 +1424,10 @@ Key changes in v1.0:
 
 Submit a pull request. Follow project conventions. Be nice.
 
+There is a CLAUDE.md file for guidance when using Claude Code. Please use it or contribute to it. Do NOT reference AI tools in commits or code. Do not co-author with AI tools. Pull requests will be rejected if they violate these rules.
+
+We encourage AI tools, but do not vibe, as the code must look like it was written by a human. Code that contains AI agent idioms will be rejected. Code that doesn't follow the project conventions will be rejected.
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).