postburner 1.0.0.rc.4 → 1.0.0.rc.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fc5a8c616b4bf393b03fdc5d2bef57eb03e1a9b9dc06fe592f89b3d1238c2606
-  data.tar.gz: 40fe3028ce331e7ecbc6facb5829b845272ce66c14578cdd12b79a4495de85ce
+  metadata.gz: b48c13a76dd9b1b6e54f825bbc50d95574c0decd4a57b8b6ba3d3fc8c5ea369c
+  data.tar.gz: 5a867eef5d36cce234b2874b5c18ff95d0ddb648fd1576e71c0a8c5946ddc12c
 SHA512:
-  metadata.gz: 2e3095d5efd3761db840b44461f42f127269608a6070ead2e9b5751054e3875e11098d34ac9232237baa27350ec46fbf8413a80adde99feae0115bedd9b08e68
-  data.tar.gz: 3a38ad87ba8692bae49462dbf6ac8e0013cb64209200c44201a25e5cb5d2c73ea14719753f279da4ce7da016d1af581a446bec7ad97af035f4884d645c12fefe
+  metadata.gz: 82e60e8af9d983550d7a07dac46b1c5611eb11245944e2ddac5a7411cf2d0b63a59697cab1a83965379bb92c274c1abd33a1a475498f981cbcb8589d520ac61c
+  data.tar.gz: 430829a786da22122f8ce4a47e8e966feb524877539f7a0e4086db912731235064c6888d0662d67c438e0c2e2d590a2a4496e33a56658b3022f70fbbb58ca050

data/CHANGELOG.md

@@ -1,5 +1,86 @@
 # Changelog
 
+## v1.0.0.rc.6 - 2026-06-27
+
+### Highlights
+
+- `Postburner::Schedule#reconcile!` — guarantees exactly one live future execution per enabled schedule and zero future executions for a disabled schedule. The watchdog delegates to it.
+- `Postburner::Schedule#enable!` / `#disable!` — convenience wrappers that drive reconciliation.
+- Auto-reconcile on schedule edits — an `after_update_commit` hook reconciles whenever a scheduling/snapshot attribute changes.
+- `Postburner::ScheduleExecution#supersede!` — tears down an execution (Beanstalkd job + `Postburner::Job` AR row) and marks it `superseded`.
+- `superseded` status (enum value `201`) on `Postburner::ScheduleExecution`, distinct from `skipped`. Neither counts as a live future execution.
+- `live` and `future_live` scopes on `Postburner::ScheduleExecution` (`pending`/`scheduled` only; `future_live` adds `run_at > now`).
+- Instrumentation events `supersede.schedule_execution.postburner` and `reconcile.schedule.postburner`.
+- **`ScheduleExecution#skip!` now skips one occurrence, then resumes -- tearing down down the associated `Postburner::Job` **and** the Beanstalkd job. The skipped row is retained as history.
+- **`schedule.destroy` now tears down each execution's Beanstalkd job and `Postburner::Job` (no orphans)
+- The watchdog sweeps **disabled** schedules with a lingering live future execution and supersedes them.
+
+### Fixed
+
+- Scheduled **tracked ActiveJob** executions had a `nil` `bkid`: the adapter stored the entire Beanstalkd put response hash (`{status:, id:}`) instead of the id, which cast to `nil` in the `bigint` column. Scheduled **non-tracked ActiveJob** executions discarded the Beanstalkd id entirely (`beanstalk_job_id` was always `nil`), so they could not be cancelled. Both now capture the real id via the adapter's `provider_job_id`.
+- Scheduled **`Postburner::Job`** executions stored the wrong value in `ScheduleExecution#beanstalk_job_id` (the return value of `queue!`, not the Beanstalkd id). It now holds the real `job.bkid`.
+- `Postburner::OrphanedJob#destroy` can be soft-removed (`remove!`) and destroyed (`destroy`), with Beanstalkd job, while `readonly?` still blocks changes.
+
+### Upgrade notes
+
+- This is a **template mutation** (like rc.4), not an additive migration. Existing installs must add a migration that:
+  1. Replaces the total unique index on `(schedule_id, run_at)` with a **live-only partial** unique index (`WHERE status IN (0, 11)`), so a superseded/skipped row can coexist with a freshly recreated live execution at the same `run_at`.
+  2. Drops `ON DELETE CASCADE` on the `schedule_id` foreign key (teardown is now owned by `dependent: :destroy` + `before_destroy`).
+- **Behavior change — deleting a schedule.** With `ON DELETE CASCADE` removed, deleting a schedule by any path other than ActiveRecord `destroy` — raw SQL, `Schedule.delete`, `where(...).delete_all` — now raises a foreign-key violation when the schedule has executions, instead of silently cascading. Always use `schedule.destroy` (which runs `dependent: :destroy` + the `before_destroy` teardown). This is intentional: it prevents orphaned Beanstalkd jobs and `Postburner::Job` rows.
+
+Standalone upgrade migration (written for zero-downtime on large `postburner_schedule_executions` tables — builds the index `CONCURRENTLY` and adds the FK unvalidated then validates it, neither of which blocks writes; requires `disable_ddl_transaction!`):
+
+```ruby
+class UpgradePostburnerSchedulesForReconcile < ActiveRecord::Migration[7.2]
+  disable_ddl_transaction!
+
+  # Existing installs created the unique index via the generator with Rails'
+  # default name. Adjust if your install named it differently.
+  OLD_INDEX = 'index_postburner_schedule_executions_on_schedule_id_and_run_at'
+  NEW_INDEX = 'index_pb_sched_exec_live_schedule_run_at'
+
+  def up
+    # 1. Add the live-only partial unique index CONCURRENTLY (no write lock) so a
+    #    superseded/skipped row can coexist with a live recreate at the same run_at.
+    #    Narrowing uniqueness can't conflict with existing data (the old total
+    #    index was stricter), so this build cannot fail on valid rows.
+    add_index :postburner_schedule_executions, [:schedule_id, :run_at],
+      unique: true, where: 'status IN (0, 11)',
+      name: NEW_INDEX, algorithm: :concurrently, if_not_exists: true
+
+    # Drop the old total unique index. Target by NAME: with the new index added,
+    # two indexes now match the (schedule_id, run_at) columns.
+    remove_index :postburner_schedule_executions,
+      name: OLD_INDEX, algorithm: :concurrently, if_exists: true
+
+    # 2. Replace the cascading FK with a plain one. Add unvalidated first (brief
+    #    metadata-only lock), then validate separately (does not block writes).
+    #    dependent: :destroy + before_destroy now own teardown.
+    remove_foreign_key :postburner_schedule_executions, column: :schedule_id
+    add_foreign_key :postburner_schedule_executions, :postburner_schedules,
+      column: :schedule_id, validate: false
+    validate_foreign_key :postburner_schedule_executions, column: :schedule_id
+  end
+
+  def down
+    remove_foreign_key :postburner_schedule_executions, column: :schedule_id
+    add_foreign_key :postburner_schedule_executions, :postburner_schedules,
+      column: :schedule_id, on_delete: :cascade, validate: false
+    validate_foreign_key :postburner_schedule_executions, column: :schedule_id
+
+    add_index :postburner_schedule_executions, [:schedule_id, :run_at],
+      unique: true, name: OLD_INDEX, algorithm: :concurrently, if_not_exists: true
+    remove_index :postburner_schedule_executions,
+      name: NEW_INDEX, algorithm: :concurrently, if_exists: true
+  end
+end
+```
+
+## v1.0.0.rc.5 - 2026-06-26
+
+### Fixed
+- `Postburner::Scheduler#ensure_future_execution!` no longer accumulates duplicate future `Postburner::ScheduleExecution` rows. The watchdog (every `scheduler_interval`, default 300s) created and enqueued one extra future execution on **every** cycle, building up dozens or hundreds of Beanstalkd delayed jobs that all eventually fire.
+
 ## v1.0.0.rc.4 - 2026-06-10
 
 ### Added

data/README.md

@@ -126,7 +126,7 @@ Postburner [beanstalkd](https://beanstalkd.github.io/) is used with PostgreSQL t
 
 ```ruby
 # Gemfile
-gem 'postburner', '~> 1.0.0.pre.18'
+gem 'postburner', '~> 1.0.0.rc.6'
 
 # config/application.rb
 config.active_job.queue_adapter = :postburner
@@ -499,8 +499,7 @@ The scheduler uses **immediate enqueue** combined with a **watchdog safety net**
    ```
 4. When a worker reserves the watchdog, it instantiates `Postburner::Scheduler` which:
    - Acquires a PostgreSQL advisory lock for coordination
-   - Auto-bootstraps any unstarted schedules
-   - Ensures each schedule has a future execution queued
+   - Reconciles every schedule (see [Editing & Pausing Schedules](#editing--pausing-schedules)) — auto-bootstrapping unstarted schedules, ensuring each enabled schedule has exactly one future execution queued, and tearing down lingering futures on disabled schedules
    - Re-queues a new watchdog with delay for the next interval
 
 NOTE: The watchdog is ephemeral data in Beanstalkd, not a database record. `Postburner::Scheduler` is the handler class that does the work. This design requires no dedicated scheduler process - existing workers handle everything.
@@ -687,8 +686,9 @@ Postburner::Schedule.create!(
 schedule = Postburner::Schedule.find_by(name: 'daily_cleanup')
 Postburner::Schedule.enabled  # All enabled schedules
 
-# Disable temporarily
-schedule.update!(enabled: false)
+# Pause / resume (preferred — see "Editing & Pausing Schedules" below)
+schedule.disable!  # Tears down the queued future execution
+schedule.enable!   # Resumes from the next grid point after now
 
 # Change catch-up policy
 schedule.update!(catch_up: true)
@@ -700,11 +700,68 @@ schedule.next_run_at  # => 2025-01-02 09:00:00 -0500
 schedule.next_run_at_times(5)  # Next 5 run times
 
 # View executions
+schedule.executions.live        # pending + scheduled (real queued work)
+schedule.executions.future_live # the live future execution(s)
 schedule.executions.pending
 schedule.executions.scheduled
-schedule.executions.skipped
+schedule.executions.skipped     # cancelled occurrences (history)
+schedule.executions.superseded  # replaced by a recreate (history)
+
+# Delete a schedule (ALWAYS use destroy — see note below)
+schedule.destroy  # Tears down every execution's Beanstalkd job + Postburner::Job row
 ```
 
+**Deleting schedules — always use `destroy`.** `schedule.destroy` runs `dependent: :destroy` plus a `before_destroy` teardown that removes each execution's Beanstalkd job and any `Postburner::Job` row, so nothing is orphaned. Deleting a schedule by any other path — raw SQL, `Schedule.delete`, `where(...).delete_all` — raises a foreign-key violation when the schedule has executions, by design. Reach for `schedule.destroy` (or `disable!` to keep the record and history).
+
+#### Editing & Pausing Schedules
+
+Postburner keeps a schedule's queued work in sync with its configuration through a single, idempotent convergence path: **reconciliation**. The guarantee is simple:
+
+- An **enabled** schedule always has **exactly one** live future execution, sitting on the current grid with a snapshot matching the live config.
+- A **disabled** schedule has **zero** future executions.
+
+You rarely call reconciliation directly (`schedule.reconcile!` exists if you need it). Instead it runs for you in three places: the watchdog reconciles every schedule on each pass, schedule edits trigger it automatically, and `enable!` / `disable!` drive it.
+
+##### Pausing & resuming
+
+Use `disable!` / `enable!` rather than `update!(enabled: ...)` (both work, but these read better and make intent obvious):
+
+```ruby
+schedule.disable!  # Supersedes the queued future execution → zero future executions
+schedule.enable!   # Recreates a future execution at the next grid point after now
+```
+
+Disabling now actually tears down the queued future execution in Beanstalkd — it doesn't just stop creating new ones. Enabling resumes from the **next grid point after now**; it does **not** backfill the gap while the schedule was paused (`catch_up` only governs the running-job chaining path, not resume).
+
+##### Editing a schedule
+
+Editing a scheduling/snapshot attribute — `args`, `queue`, `priority`, the grid (`anchor`, `interval`, `interval_unit`, `cron`), `timezone`, `enabled`, or `name` — automatically supersedes the stale queued execution and recreates it from the current config (via an `after_update_commit` hook):
+
+```ruby
+# Payload-only edit: recreates the future execution at the SAME run_at
+schedule.update!(args: { report_type: 'weekly' })
+
+# Grid edit: recreates the future execution at the NEW grid point
+schedule.update!(interval: 2, interval_unit: 'hours')
+```
+
+> **Best-effort, not synchronous.** This inline reconcile is non-blocking: it *tries* the global scheduler advisory lock without waiting, so it never holds up a web request. If the watchdog happens to hold the lock mid-sweep, the inline reconcile is skipped and the watchdog converges on its next pass. For UI-managed schedules, **don't assume the next run is updated immediately** — read it back after the watchdog interval (`scheduler_interval`) rather than right after the save.
+
+##### Skipping a single occurrence
+
+`skip!` cancels **one** upcoming occurrence and then resumes at the next grid point after the skipped slot. It tears down both the Beanstalkd job and the associated `Postburner::Job` row, marks the execution `skipped` (kept as history), and triggers a best-effort reconcile. The skipped occurrence does not run and is not recreated:
+
+```ruby
+execution = schedule.executions.future_live.first
+execution.skip!  # This one won't run; the schedule continues at the next grid point
+```
+
+To pause every upcoming run instead of just one, use `disable!`.
+
+##### Superseding (internal)
+
+`ScheduleExecution#supersede!` performs the same teardown as `skip!` but records a `superseded` status, meaning the execution was replaced by a recreate (config/grid drift resolved by reconciliation) rather than cancelled by an operator. Reconciliation uses it internally; you'll mostly encounter `superseded` rows when reading execution history. Like skipped rows, superseded rows are excluded from `live` / `future_live`.
+
 #### Starting Schedules
 
 When you create a schedule, it won't run until the first execution is created. You have two options:
@@ -748,19 +805,27 @@ Each scheduled run creates an execution record for tracking:
 ```ruby
 execution = Postburner::ScheduleExecution.find(123)
 
-execution.status        # pending, scheduled, skipped
+execution.status        # pending, scheduled, skipped, superseded
 execution.run_at        # Scheduled time
 execution.enqueued_at   # When job was queued
 execution.beanstalk_job_id  # Beanstalkd job ID
 execution.job_id            # Postburner::Job ID (if using Postburner::Job)
 ```
 
+**Status values:**
+- `pending` — created, not yet enqueued to Beanstalkd
+- `scheduled` — enqueued to Beanstalkd, waiting for `run_at`
+- `skipped` — one occurrence cancelled by an operator (`skip!`); retained as history
+- `superseded` — replaced by a recreate during reconciliation (`supersede!`); retained as history
+
+Only `pending` and `scheduled` count as a live future execution (the `live` / `future_live` scopes). `skipped` and `superseded` are inert history.
+
 **Execution lifecycle:**
 1. Execution created with `pending` status and immediately enqueued to Beanstalkd
 2. Status changes to `scheduled` once enqueued
 3. At `run_at` time, Beanstalkd releases job to worker
 4. For `Postburner::Job` (and `ActiveJob` with `Postburner::Tracked`) schedules: `before_attempt` callback creates next execution
-5. Watchdog periodically verifies future executions exist (safety net)
+5. Watchdog periodically reconciles each schedule, guaranteeing future executions exist (safety net)
 
 #### Timezone Handling
 
@@ -1572,7 +1637,8 @@ Postburner emits ActiveSupport::Notifications events following Rails conventions
 |-------|------|--------------|
 | `create.schedule.postburner` | When schedule is created | `:schedule` |
 | `update.schedule.postburner` | When schedule is updated | `:schedule`, `:changes` |
-| `audit.schedule.postburner` | When scheduler audits a schedule | `:schedule` |
+| `audit.schedule.postburner` | When scheduler audits a schedule | `:schedule`, `:execution_created`, `:orphans_enqueued` |
+| `reconcile.schedule.postburner` | When a schedule is reconciled (executions converged to config) | `:schedule`, `:superseded`, `:created` |
 
 **Schedule Payload Structure:**
 
@@ -1595,7 +1661,8 @@ Postburner emits ActiveSupport::Notifications events following Rails conventions
 |-------|------|--------------|
 | `create.schedule_execution.postburner` | When execution is created | `:schedule`, `:execution` |
 | `enqueue.schedule_execution.postburner` | When execution is enqueued to Beanstalkd | `:schedule`, `:execution`, `:beanstalk_job_id` |
-| `skip.schedule_execution.postburner` | When execution is skipped | `:schedule`, `:execution` |
+| `skip.schedule_execution.postburner` | When an operator skips one occurrence | `:schedule`, `:execution` |
+| `supersede.schedule_execution.postburner` | When an execution is superseded (replaced by a recreate during reconciliation) | `:schedule`, `:execution` |
 
 **Execution Payload Structure:**
 
@@ -2368,7 +2435,7 @@ Key changes in v1.0:
 
 1. **Update Gemfile:**
    ```ruby
-   gem 'postburner', '~> 1.0.0.pre.18'
+   gem 'postburner', '~> 1.0.0.rc.6'
    ```
 
 2. **Remove Backburner config:**

data/app/models/postburner/job.rb

@@ -88,6 +88,21 @@ module Postburner
 
     validates :sid, presence: {strict: true}
 
+    # Jobs that have been picked up by a worker but have not yet completed or
+    # been removed, i.e. currently in-flight.
+    #
+    # @note A job whose worker crashed mid-run can remain in this scope even
+    #   though nothing is executing. Beanstalkd's reserved count
+    #   ({Postburner.stats}) is the ground truth for what is actually running.
+    #
+    # @example
+    #   Postburner::Job.processing          # all in-flight tracked jobs
+    #   Postburner::TrackedJob.processing   # in-flight tracked ActiveJobs
+    #
+    scope :processing, -> {
+      where.not(processing_at: nil).where(processed_at: nil, removed_at: nil)
+    }
+
     # Resolves an STI type name to a class, falling back to OrphanedJob when the
     # original class no longer exists. This tolerates rows whose `type` column
     # references a deleted or renamed job class, preventing

data/app/models/postburner/orphaned_job.rb

@@ -84,6 +84,10 @@ module Postburner
     # through instance save/update and therefore does not trigger
     # `ensure_proper_type` either).
     #
+    # Uses the STI base class for the relation so the row's missing `type` value
+    # (which differs from OrphanedJob's sti_name) does not scope it out — a
+    # `self.class.where` would match zero rows and silently fail to persist.
+    #
     # Idempotent: does nothing if already removed.
     #
     # @return [void]
@@ -93,10 +97,31 @@ module Postburner
 
       self.delete!
       now = Time.current
-      self.class.where(id: self.id).update_all(removed_at: now)
+      self.class.base_class.where(id: self.id).update_all(removed_at: now)
       self.removed_at = now
     end
 
+    # Hard-deletes this orphaned row, removing BOTH its Beanstalkd job and its AR
+    # row, while keeping `readonly?` true for saves/updates.
+    #
+    # The base ActiveRecord#destroy raises ReadOnlyRecord on a readonly record,
+    # which would make ScheduleExecution#teardown_job! (which calls `job.destroy`
+    # uniformly across job shapes) fail for an execution whose job class was
+    # deleted/renamed. This override mirrors {#remove!}: it removes the Beanstalkd
+    # job via {#delete!} and deletes the row at the relation level — using the STI
+    # base class so the missing `type` value doesn't scope the row out — bypassing
+    # both `readonly?` and instance callbacks. The row is genuinely removed (not a
+    # soft-delete), matching destroy semantics.
+    #
+    # @return [self] frozen, with destroyed? == true
+    #
+    def destroy
+      self.delete!
+      self.class.base_class.where(id: self.id).delete_all
+      @destroyed = true
+      freeze
+    end
+
     # Prevents saves that would allow Rails' ensure_proper_type to overwrite the
     # original `type` value with 'Postburner::OrphanedJob'.
     #