postburner 1.0.0.rc.3 → 1.0.0.rc.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9aa20d7fa2c867942efc6c70f7ad56548e651192068c9505a2854310cbb95043
-  data.tar.gz: 74606e227872a914c3a806227d74206d4b66a33b7160753cbbb48a15d48a7d9c
+  metadata.gz: 62187fb2fd681e87e1028c52cbd6f3756b32c895c05390bffca05d51e1287ec7
+  data.tar.gz: 31ab083d1e9d00bfb7dc6fabbf09d7065f5062d73a6cd738835a0b76f54426b6
 SHA512:
-  metadata.gz: cf4238e6bc83f1447de5c846032aad268d904fb5f74c1462424c69ffed8e40ffe316c51c2d248b2a6b46b7fb96df9b572f028368cc9e8e65e4baf8bffa7175b5
-  data.tar.gz: c93242ccd8501c5b337417f60633474104550360276c607a48308ed6bfcbcd5bd77aeafa9df7e6f0e6cb0f15094ffe4ada75ef13f2704e98c2fd31bc674fe791
+  metadata.gz: d2e7a422e53852490b5e2ec8502f3dd69a34f17975ee808f463658f20f2ca228c5467d149987caaf7e905433b364b30c7f92142e07e7f7da3bbd7763d99c6284
+  data.tar.gz: aa6ce11d8f045be5b259a1869339117ddf40fca8e6d3579ced469355d2c89fbd7f19da96903154dac11a0f0efed57db840c1581ba9512c2665300d58d0453e32

data/CHANGELOG.md

@@ -1,5 +1,78 @@
 # Changelog
 
+## v1.0.0.rc.5 - 2026-06-26
+
+### Fixed
+- `Postburner::Scheduler#ensure_future_execution!` no longer accumulates duplicate future `Postburner::ScheduleExecution` rows. The watchdog (every `scheduler_interval`, default 300s) created and enqueued one extra future execution on **every** cycle, building up dozens or hundreds of Beanstalkd delayed jobs that all eventually fire.
+
+## v1.0.0.rc.4 - 2026-06-10
+
+### Added
+- `Postburner::OrphanedJob` — inert STI placeholder loaded when a row's `type` column references a deleted or renamed class. Prevents `ActiveRecord::SubclassNotFound` from crashing callers (e.g. the admin queued-jobs index). `perform` and `queue!`/`requeue!` raise `Postburner::Job::OrphanedJobError`. `readonly?` returns `true` to prevent Rails' `ensure_proper_type` from overwriting the original `type` string. `remove!` (soft-delete) and `destroy` still work. No migration required.
+- `Postburner::Job::OrphanedJobError` — raised by `OrphanedJob#perform` and `#queue!`/`#requeue!`.
+- `Postburner::Job#orphaned?` — predicate; always `false` on the base class, `true` on `OrphanedJob`.
+- `Postburner::Job.find_sti_class` override — rescues `ActiveRecord::SubclassNotFound` and returns `Postburner::OrphanedJob` instead.
+- Partial indexes on `postburner_jobs` (`idx_pbjobs_active_run_at`, `idx_pbjobs_processed`, `idx_pbjobs_unqueued`) tuned to the admin UI's default queries, keeping the jobs list fast on large tables.
+- `JobsController#index` now accepts a `scope` param (`active`, `processed`, `unqueued`, `all`) and defaults to `active`, so the default page load matches the `idx_pbjobs_active_run_at` predicate.
+
+### Changed
+- `postburner_jobs.lag` is now `bigint` instead of `integer`, to avoid overflow on jobs that sit in the queue long enough to exceed the 32-bit range.
+
+### Upgrade notes
+- This is a **template mutation**, not an additive migration. Existing rc.x installs will need to either re-run `rails generate postburner:install` (and merge the migration by hand) or manually add an equivalent migration that creates the three partial indexes. Failing to do either is **non-catastrophic** — the admin UI will simply remain slow on large `postburner_jobs` tables until the indexes are in place.
+- The `lag` column type change (`integer` → `bigint`) is also a template mutation. Existing installs should add a migration along the lines of `change_column :postburner_jobs, :lag, :bigint` to avoid overflow once lag values exceed the 32-bit range.
+
+Example migration covering both the `lag` widening and the three partial indexes:
+
+```ruby
+class UpgradePostburnerJobsForRc4 < ActiveRecord::Migration[7.2]
+  disable_ddl_transaction!
+
+  def up
+    # Widen lag to bigint to avoid 32-bit overflow on long-queued jobs.
+    # ~24.9 days -> ~292.5 million years
+    change_column :postburner_jobs, :lag, :bigint
+
+    # Partial indexes tuned to the admin UI's default queries.
+    add_index :postburner_jobs, :run_at,
+      name: 'idx_pbjobs_active_run_at',
+      where: '(processed_at IS NULL AND removed_at IS NULL AND queued_at IS NOT NULL)',
+      algorithm: :concurrently
+
+    add_index :postburner_jobs, :processed_at,
+      name: 'idx_pbjobs_processed',
+      order: { processed_at: :desc },
+      where: '(processed_at IS NOT NULL)',
+      algorithm: :concurrently
+
+    add_index :postburner_jobs, [:removed_at, :created_at],
+      name: 'idx_pbjobs_unqueued',
+      order: { created_at: :desc },
+      where: '(queued_at IS NULL AND removed_at IS NULL)',
+      algorithm: :concurrently
+  end
+
+  def down
+    remove_index :postburner_jobs, name: 'idx_pbjobs_unqueued', algorithm: :concurrently
+    remove_index :postburner_jobs, name: 'idx_pbjobs_processed', algorithm: :concurrently
+    remove_index :postburner_jobs, name: 'idx_pbjobs_active_run_at', algorithm: :concurrently
+
+    change_column :postburner_jobs, :lag, :integer
+  end
+end
+```
+
+Notes:
+- `disable_ddl_transaction!` + `algorithm: :concurrently` is recommended for production tables of any size — it builds the indexes without holding a long write lock. Drop both if you're on a small dev table and want a single transactional migration.
+- `change_column ... :bigint` on PostgreSQL is an in-place type widening for `integer → bigint` and is fast (no full table rewrite), but it does take a brief `ACCESS EXCLUSIVE` lock — schedule accordingly on hot tables.
+- Adjust `ActiveRecord::Migration[7.2]` to match your app's Rails version.
+
+## v1.0.0.rc.3
+
+## v1.0.0.rc.2
+
+## v1.0.0.rc.1
+
 ## v0.7.0 - 2022-10-24
 
 ### Added

data/app/controllers/postburner/jobs_controller.rb

@@ -2,12 +2,41 @@ require_dependency "postburner/application_controller"
 
 module Postburner
   class JobsController < ApplicationController
+    # Scopes align with the partial indexes defined in the install
+    # migration. Keeping the controller's default query matched to an
+    # index predicate keeps the admin UI fast on large tables.
+    SCOPES = %w[ active processed unqueued all ].freeze
+    DEFAULT_SCOPE = 'active'.freeze
+
     def index
-      @jobs = Job.order(queued_at: :desc, created_at: :desc)
+      @scope = SCOPES.include?(params[:scope]) ? params[:scope] : DEFAULT_SCOPE
+      @jobs = scoped_jobs(@scope)
     end
 
     def show
       @job = Job.find(params[:id])
     end
+
+    private
+
+    def scoped_jobs(scope)
+      case scope
+      when 'active'
+        # Uses idx_pbjobs_active_run_at
+        Job.where(processed_at: nil, removed_at: nil)
+           .where.not(queued_at: nil)
+           .order(run_at: :asc)
+      when 'processed'
+        # Uses idx_pbjobs_processed
+        Job.where.not(processed_at: nil)
+           .order(processed_at: :desc)
+      when 'unqueued'
+        # Uses idx_pbjobs_unqueued
+        Job.where(queued_at: nil, removed_at: nil)
+           .order(created_at: :desc)
+      else
+        Job.order(queued_at: :desc, created_at: :desc)
+      end
+    end
   end
 end

data/app/models/postburner/job.rb

@@ -88,6 +88,45 @@ module Postburner
 
     validates :sid, presence: {strict: true}
 
+    # Jobs that have been picked up by a worker but have not yet completed or
+    # been removed, i.e. currently in-flight.
+    #
+    # @note A job whose worker crashed mid-run can remain in this scope even
+    #   though nothing is executing. Beanstalkd's reserved count
+    #   ({Postburner.stats}) is the ground truth for what is actually running.
+    #
+    # @example
+    #   Postburner::Job.processing          # all in-flight tracked jobs
+    #   Postburner::TrackedJob.processing   # in-flight tracked ActiveJobs
+    #
+    scope :processing, -> {
+      where.not(processing_at: nil).where(processed_at: nil, removed_at: nil)
+    }
+
+    # Resolves an STI type name to a class, falling back to OrphanedJob when the
+    # original class no longer exists. This tolerates rows whose `type` column
+    # references a deleted or renamed job class, preventing
+    # ActiveRecord::SubclassNotFound from crashing callers (e.g. the admin
+    # queued-jobs page).
+    #
+    # @param type_name [String] The value stored in the `type` column
+    # @return [Class] The resolved class, or {Postburner::OrphanedJob} if missing
+    #
+    def self.find_sti_class(type_name)
+      super
+    rescue ActiveRecord::SubclassNotFound
+      Postburner::OrphanedJob
+    end
+
+    # Returns whether this job instance is an orphaned placeholder.
+    #
+    # Always false on the base class. Overridden to return true in
+    # {Postburner::OrphanedJob}.
+    #
+    # @return [Boolean]
+    #
+    def orphaned? = false
+
     # Abstract method to be implemented by subclasses.
     #
     # This method contains the actual work logic for the job. It is called
@@ -220,6 +259,15 @@ module Postburner
     #
     class AlreadyProcessed < StandardError; end
 
+    # Exception raised when attempting to perform or enqueue an orphaned job whose
+    # original class no longer exists in the application.
+    #
+    # @example
+    #   job = Postburner::Job.find(id)  # loads as OrphanedJob if class is gone
+    #   job.perform  # => raises OrphanedJobError
+    #
+    class OrphanedJobError < StandardError; end
+
     # Exception raised when a job is executed before its scheduled run_at time.
     #
     # In production (Queue/NiceQueue), this is handled by re-inserting with delay.

data/app/models/postburner/orphaned_job.rb

@@ -0,0 +1,107 @@
+# frozen_string_literal: true
+
+module Postburner
+  # Inert STI placeholder loaded when a Postburner::Job row's `type` column
+  # references a class that no longer exists in the application.
+  #
+  # ## Why readonly?
+  #
+  # Rails' `ensure_proper_type` (called via `save`/`update`) would overwrite the
+  # original `type` value with `'Postburner::OrphanedJob'`, permanently destroying
+  # the record of which class was missing. `readonly?` returning true prevents all
+  # `save`/`update` paths and therefore blocks `ensure_proper_type` from running.
+  #
+  # Rails' `instantiate` reads the `type` column verbatim when loading records from
+  # the database and does NOT call `ensure_proper_type`, so the original class name
+  # is preserved on load. This class is only ever instantiated via the
+  # `find_sti_class` fallback in {Postburner::Job}, never created directly.
+  #
+  # ## Why remove! is overridden here
+  #
+  # In Rails 8.1+, `update_column` respects `readonly?` and raises
+  # `ActiveRecord::ReadOnlyRecord`. The base `Commands#remove!` calls
+  # `update_column(:removed_at, ...)` — so it would be blocked too. This subclass
+  # overrides `remove!` to use `self.class.where(id: self.id).update_all(...)` which
+  # operates at the relation level (bypasses both `readonly?` and all instance
+  # callbacks) while still soft-deleting the row. Admins can therefore safely
+  # soft-remove orphaned rows without restoring the missing class.
+  #
+  # ## Usage
+  #
+  # This class is loaded automatically by {Postburner::Job.find_sti_class} when a
+  # row's type is unresolvable. You should never instantiate it directly in
+  # application code.
+  #
+  # @example Typical lifecycle
+  #   # A DB row exists with type: 'Flex::TriggerNoShowDeliveriesJob' (class deleted)
+  #   job = Postburner::Job.find(42)
+  #   job.class         # => Postburner::OrphanedJob
+  #   job.type          # => 'Flex::TriggerNoShowDeliveriesJob'
+  #   job.orphaned?     # => true
+  #   job.missing_class_name  # => 'Flex::TriggerNoShowDeliveriesJob'
+  #   job.perform             # => raises Postburner::Job::OrphanedJobError
+  #   job.remove!             # soft-deletes fine despite readonly?
+  #
+  class OrphanedJob < Job
+    # Returns the original class name stored in the `type` column — the deleted
+    # or renamed class that this placeholder stands in for.
+    #
+    # @return [String]
+    #
+    def missing_class_name
+      self.type.to_s
+    end
+
+    # Always true; identifies this instance as an orphaned placeholder.
+    #
+    # @return [Boolean]
+    #
+    def orphaned? = true
+
+    # Raises {Postburner::Job::OrphanedJobError} because the original job class
+    # no longer exists and there is no implementation to run.
+    #
+    # @raise [Postburner::Job::OrphanedJobError] always
+    #
+    def perform(*)
+      raise Postburner::Job::OrphanedJobError,
+        "Cannot perform orphaned job ##{self.id}: class '#{self.missing_class_name}' no longer exists"
+    end
+
+    # Raises {Postburner::Job::OrphanedJobError} because re-enqueuing a job whose
+    # class is gone would only produce another unperformable entry.
+    #
+    # @raise [Postburner::Job::OrphanedJobError] always
+    #
+    def queue!(*)
+      raise Postburner::Job::OrphanedJobError,
+        "Cannot enqueue orphaned job ##{self.id}: class '#{self.missing_class_name}' no longer exists"
+    end
+    alias_method :requeue!, :queue!
+
+    # Soft-deletes this orphaned row by setting `removed_at`, bypassing the
+    # `readonly?` guard via a relation-level `update_all` call (which does not go
+    # through instance save/update and therefore does not trigger
+    # `ensure_proper_type` either).
+    #
+    # Idempotent: does nothing if already removed.
+    #
+    # @return [void]
+    #
+    def remove!
+      return if self.removed_at
+
+      self.delete!
+      now = Time.current
+      self.class.where(id: self.id).update_all(removed_at: now)
+      self.removed_at = now
+    end
+
+    # Prevents saves that would allow Rails' ensure_proper_type to overwrite the
+    # original `type` value with 'Postburner::OrphanedJob'.
+    #
+    # @return [Boolean] always true
+    #
+    def readonly? = true
+  end
+end

data/app/models/postburner/schedule.rb

@@ -366,6 +366,11 @@ module Postburner
 
       execution.save!
 
+      # Reload so instrumented timestamps reflect persisted (microsecond)
+      # precision rather than the in-memory nanosecond values, matching what
+      # consumers see after a DB roundtrip (see #enqueue! reload below).
+      execution.reload
+
       # Instrument execution creation
       ActiveSupport::Notifications.instrument('create.schedule_execution.postburner', {
         schedule: Postburner::Instrumentation.schedule_payload(self),

data/lib/generators/postburner/install/templates/migrations/create_postburner_jobs.rb.erb

@@ -13,7 +13,7 @@ class CreatePostburnerJobs < ActiveRecord::Migration<%= migration_version %>
       t.datetime :processing_at
       t.datetime :processed_at
       t.datetime :removed_at
-      t.integer :lag
+      t.bigint :lag
       t.integer :duration
       t.integer :attempt_count
       t.integer :error_count
@@ -37,6 +37,22 @@ class CreatePostburnerJobs < ActiveRecord::Migration<%= migration_version %>
       t.index [ :error_count ]
       t.index [ :log_count ]
 
+      # Partial indexes tuned to the admin UI's default queries. Kept
+      # alongside the single-column indexes above: the single-column
+      # indexes still serve ad-hoc queries, while these partials keep
+      # the hot-path listing queries fast on large tables.
+      t.index :run_at,
+        name: 'idx_pbjobs_active_run_at',
+        where: 'processed_at IS NULL AND removed_at IS NULL AND queued_at IS NOT NULL'
+      t.index :processed_at,
+        name: 'idx_pbjobs_processed',
+        order: { processed_at: :desc },
+        where: 'processed_at IS NOT NULL'
+      t.index [ :removed_at, :created_at ],
+        name: 'idx_pbjobs_unqueued',
+        order: { created_at: :desc },
+        where: 'queued_at IS NULL AND removed_at IS NULL'
+
       # Add these or more depending on what you want to search for.
       #t.index [ :attempts ], using: :gin
       #t.index [ :errata ], using: :gin

data/lib/postburner/scheduler.rb

@@ -484,7 +484,32 @@ module Postburner
         return true
       end
 
-      # Delegate to Schedule - it knows whether a future execution is needed
+      # GUARD: Only create a new future execution if none exists yet (relative to now).
+      #
+      # This guard intentionally uses Time.current, while the inner guard inside
+      # create_next_execution! (schedule.rb:199-200) uses after.run_at. They answer
+      # DIFFERENT questions:
+      #
+      #   Watchdog (here):         "Does ANY execution exist in the future?"
+      #   before_attempt callback: "Does an execution exist AFTER the one now running?"
+      #
+      # The before_attempt callback (job.rb:380) passes `after: current_execution` so
+      # create_next_execution!'s check_time ≈ Time.current — it correctly fires when
+      # the watchdog has already queued a successor. But when this method calls
+      # create_next_execution!(after: last_execution), check_time becomes
+      # last_execution.run_at — the highest run_at in the table by definition — so
+      # the inner guard can never find an execution beyond it and would always create
+      # another, accumulating one extra future execution per watchdog cycle.
+      #
+      # Note: > (not >=) is intentional. A just-fired execution at run_at == Time.current
+      # must NOT block creation of its successor.
+      #
+      # The inner guard in create_next_execution! is STILL REQUIRED for the
+      # before_attempt callback path and must NOT be removed even though it is
+      # effectively bypassed on the watchdog path now that this outer guard fires first.
+      return false if schedule.executions.where('run_at > ?', Time.current).exists?
+
+      # Delegate to Schedule - it knows how to calculate the next run time
       execution = schedule.create_next_execution!(after: last_execution)
 
       if execution

data/lib/postburner/version.rb

@@ -1,3 +1,3 @@
 module Postburner
-  VERSION = '1.0.0.rc.3'
+  VERSION = '1.0.0.rc.5'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: postburner
 version: !ruby/object:Gem::Version
-  version: 1.0.0.rc.3
+  version: 1.0.0.rc.5
 platform: ruby
 authors:
 - Matt Smith
@@ -138,6 +138,7 @@ files:
 - app/models/postburner/application_record.rb
 - app/models/postburner/job.rb
 - app/models/postburner/mailer.rb
+- app/models/postburner/orphaned_job.rb
 - app/models/postburner/schedule.rb
 - app/models/postburner/schedule_execution.rb
 - app/models/postburner/tracked_job.rb