sidekiq-batch-jobs 0.2.0 → 0.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9e70220429bbc1911259964dccf5cc7a67ac5ecdbb0d2ecafe98fb783d70c043
-  data.tar.gz: fb4d826947c4acf259e3641022031b656a6b4bd879ba748b1e0cc1f9342e691b
+  metadata.gz: 148b48f6e660baae58a41a709b7b43010e8c7a1c29585368d9d624d23ce7bfa7
+  data.tar.gz: 5c606573cc3b223bd29b6843dc041cb8296c157b2d063b5e6d5055a736dca4b7
 SHA512:
-  metadata.gz: 9b1bbb5b23cae02b4d51ab59c8a471f75cb46371236a6b9dd1494979cc9523c4a46c00e10e4402a713d6c9d426087e6f8ab7fd57bad48d0998b6457d90eb966a
-  data.tar.gz: d514d78ae12916da738b6d421a46eb4401d3e8ee3664dc8af67c84c21d26accb7ac2a8609450fe255ebb6ea38957d12a505c602402782fc69bb70c5976b59844
+  metadata.gz: e6b4be14970f3567194ab9f8c81fe8dde1493308c83b73165fdcfb7291f2b3d67644d0ca97168f90816955a9de78479d81111abfdf4e42c1522fb4bc6d6be8a3
+  data.tar.gz: 4c910d40ef9a6a183541a18c8fb5d8ff2f8483ea26ad5b1cc22b2d611f0728cc3f26c36d494715a36d64b0bce24cb51683ba672e967e01dd692b750ee004e96e

data/CHANGELOG.md

@@ -5,6 +5,58 @@ All notable changes to this project are documented here.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.3.0] - 2026-09-07
+
+### Breaking
+
+- **A migration is required, and the new code does not work without it.** The completion
+  statement writes both new columns, so against an unmigrated database every completion check
+  raises `PG::UndefinedColumn` and `#progress` raises `NameError` on a finished batch. See
+  *Upgrading from 0.2.0* below.
+
+### Added
+
+- `complete_count` and `failed_count` on `sidekiq_batches`, stamped by the completion statement
+  in the same `UPDATE` that transitions the batch. `#progress` reads them back, so a finished
+  batch reports its counts without touching a job row, and `#percentage_progress` now answers
+  100.0 from the status alone: a batch only transitions once nothing is pending, so a terminal
+  one has nothing left to count. A running batch counts rows exactly as before, so polling a
+  live batch is unchanged.
+
+  Written once, by the statement that already runs once per batch, rather than bumped from
+  `complete!` and `fail!`. A per-job counter would serialise every worker in a batch behind a
+  single row lock and churn a tuple per job, to speed up reads that were never on the hot path.
+  The completion statement takes that lock anyway, and a terminal batch transitions no further
+  jobs, so the tally cannot drift from the rows.
+
+- `rails g sidekiq:batch:jobs:upgrade`, which replaces the copy-paste migration these notes
+  used to hand you. It reads your tables and emits only what they are missing, so upgrading
+  from 0.1.0 and upgrading from 0.2.0 are the same command and each writes the migration that
+  applies. Nothing to do means no file, so it is safe to run against a current database, and
+  safe to run twice. `Sidekiq::Batch::Jobs::Schema` is the schema it diffs against.
+
+### Upgrading from 0.2.0
+
+```bash
+bin/rails g sidekiq:batch:jobs:upgrade
+bin/rails db:migrate
+```
+
+Two nullable columns, which on Postgres 11+ is a metadata-only change however large the table.
+
+Run it before the workers pick up the new code, not after. The completion statement names both
+columns, so until they exist every completion check fails.
+
+Nothing is lost if that order slips. The failure is caught where every completion check is
+caught: the job itself still succeeds, the batch keeps its rows and stays `running`, and the
+reason goes to `config.on_alert`. Once the columns exist, the next job to finish transitions
+its batch normally, and `SidekiqBatch::ReaperWorker` completes any batch whose last job already
+ran, once it has been quiet for `config.stuck_after` (two hours by default).
+
+No backfill. NULL means "no tally was stamped", which is every running batch and every batch
+that finished under 0.2.0; both fall back to counting job rows the way 0.2.0 always did.
+Batches that finish after the migration carry their own counts.
+
 ## [0.2.0] - 2026-08-31
 
 0.1.0 could enrol a batch, detect completion and fire a callback. This release keeps that core

data/README.md

@@ -9,6 +9,7 @@ Tested against Ruby 3.0–3.4, Rails 6.1–8.0, Sidekiq 7–8.
 ## Contents
 
 - [Installation](#installation)
+  - [Upgrading an existing install](#upgrading-an-existing-install)
   - [Configuration](#configuration)
 - [Usage](#usage)
   - [The four steps](#the-four-steps)
@@ -53,6 +54,26 @@ death handler for you at boot, and again after every code reload.
 (`Sidekiq.configure_embed`) never runs this gem's server middleware, so its jobs stay
 `pending` even when they succeed, and the reaper eventually marks them failed.
 
+### Upgrading an existing install
+
+Some releases add columns. Rather than a migration per release for you to apply in order,
+there is one command that reads your tables and writes only what they are missing:
+
+```bash
+bin/rails g sidekiq:batch:jobs:upgrade
+bin/rails db:migrate
+```
+
+It works from any earlier version, and writes no file at all when your schema is already
+current, so it is safe to run whenever a release mentions the schema, and safe to run twice.
+Read the migration before you run it, as you would any generated one. Do not use the install
+generator for this: that one builds the tables from scratch.
+
+Run it **before** the new code reaches your workers. The gem's SQL names every column it
+expects, so between deploying and migrating, completion checks fail. Nothing is lost if that
+order slips: the failure goes to `config.on_alert`, jobs still run, batches keep their rows,
+and everything stalled completes once the migration lands.
+
 ### Configuration
 
 Every setting is optional and has a working default. To change one, do it in an initializer:
@@ -316,6 +337,8 @@ batch.status            # "pending" | "running" | "succeeded" | "failed"
 batch.terminal?         # true once the batch has finished, either way
 batch.completed_at      # nil until the batch finishes
 batch.total_jobs        # stamped when the jobs {} block returns; 0 before that
+batch.complete_count    # the final tally, stamped when the batch finishes;
+batch.failed_count      # both nil while it is still running
 batch.context           # the jsonb hash you stashed when creating the batch
 
 batch.failure_policy    # "any_failure" | "all_failed" | "tolerate_jobs" | "tolerate_percent"
@@ -326,6 +349,19 @@ batch.enrollment_error  # nil normally; {"class" =>, "message" =>} if the
 batch.callbacks_fired   # event => when that callback went out
 ```
 
+**Reading a finished batch costs nothing.** The completion statement stamps `complete_count`
+and `failed_count` in the same `UPDATE` that transitions the batch, so `progress` on a terminal
+batch reads two attributes instead of counting job rows, and `percentage_progress` answers
+100.0 from the status alone: a batch only transitions once nothing is pending, so there is
+nothing left to count. This matters most where you are most likely to look, which is from
+inside a callback on a batch that just finished.
+
+Those are the only two places the numbers are written. Bumping a counter from `complete!` and
+`fail!` would put every worker in the batch behind one row lock and write a new version of the
+batch row per job, all to speed up a read that is not on the hot path. The completion statement
+takes that lock once per batch anyway, and a terminal batch transitions no further jobs, so its
+tally cannot drift. A batch that is still running counts rows, exactly as before.
+
 Each enrolled job has a row of its own:
 
 ```ruby

data/app/models/sidekiq_batch/completion_query.rb

@@ -9,9 +9,7 @@ class SidekiqBatch
       def sql
         <<~SQL.squish
           UPDATE sidekiq_batches
-          SET status       = #{outcome},
-              completed_at = NOW(),
-              updated_at   = NOW()
+          SET #{assignments}
           WHERE id = ?
             AND status = #{batch_status("running")}
             AND NOT EXISTS (#{jobs_with(job_status("pending"))})
@@ -21,6 +19,15 @@ class SidekiqBatch
 
       private
 
+      def assignments
+        <<~SQL
+          status                         = #{outcome},
+          (complete_count, failed_count) = (#{final_counts}),
+          completed_at                   = NOW(),
+          updated_at                     = NOW()
+        SQL
+      end
+
       def outcome
         <<~SQL
           CASE
@@ -62,6 +69,25 @@ class SidekiqBatch
         "GREATEST(COALESCE(failure_tolerance, 0), 0)"
       end
 
+      # The tally the batch keeps once it is finished, so reading its progress
+      # later costs no job rows at all. Free of the contention a per-job counter
+      # would buy: this statement's WHERE matches nothing until the last job
+      # lands, so it runs once per batch and takes the row lock once. Nothing
+      # can drift either, since a terminal batch transitions no further jobs.
+      #
+      # One multi-column assignment rather than two scalar subqueries, so the
+      # batch's rows are walked once for both numbers. The CASE in `status`
+      # cannot read them: every SET expression sees the pre-UPDATE row, which is
+      # why the tolerating branch below keeps a count of its own.
+      def final_counts
+        <<~SQL
+          SELECT COUNT(*) FILTER (WHERE status = #{job_status("complete")}),
+                 COUNT(*) FILTER (WHERE status = #{job_status("failed")})
+          FROM sidekiq_batch_jobs
+          WHERE sidekiq_batch_id = sidekiq_batches.id
+        SQL
+      end
+
       def failed_count
         "SELECT COUNT(*) FROM sidekiq_batch_jobs " \
           "WHERE sidekiq_batch_id = sidekiq_batches.id AND status = #{job_status("failed")}"

data/app/models/sidekiq_batch.rb

@@ -8,10 +8,12 @@
 #  callback_fired_at :datetime
 #  callbacks         :jsonb            not null
 #  callbacks_fired   :jsonb            not null
+#  complete_count    :integer
 #  completed_at      :datetime
 #  context           :jsonb            not null
 #  description       :string
 #  enrollment_error  :jsonb
+#  failed_count      :integer
 #  failure_policy    :string
 #  failure_tolerance :integer
 #  status            :integer          default("pending"), not null
@@ -112,6 +114,8 @@ class SidekiqBatch < ::Sidekiq::Batch::Jobs.base_class
   def percentage_progress
     return 0.0 if total_jobs.zero?
 
+    return 100.0 if terminal?
+
     ((finished_jobs_count.to_f / total_jobs) * 100).round(2)
   end
 
@@ -131,14 +135,7 @@ class SidekiqBatch < ::Sidekiq::Batch::Jobs.base_class
   end
 
   def progress
-    counts = sidekiq_batch_jobs.group(:status).count
-
-    {
-      total:    total_jobs,
-      complete: counts.fetch("complete", 0),
-      failed:   counts.fetch("failed", 0),
-      pending:  counts.fetch("pending", 0)
-    }
+    stamped_counts || count_by_status
   end
 
   def pending_jobs
@@ -192,6 +189,23 @@ class SidekiqBatch < ::Sidekiq::Batch::Jobs.base_class
     end
   end
 
+  def stamped_counts
+    return nil unless terminal? && complete_count && failed_count
+
+    { total: total_jobs, complete: complete_count, failed: failed_count, pending: 0 }
+  end
+
+  def count_by_status
+    counts = sidekiq_batch_jobs.group(:status).count
+
+    {
+      total:    total_jobs,
+      complete: counts.fetch("complete", 0),
+      failed:   counts.fetch("failed", 0),
+      pending:  counts.fetch("pending", 0)
+    }
+  end
+
   def finished_jobs_count
     sidekiq_batch_jobs.where(status: %w[complete failed]).count
   end

data/lib/generators/sidekiq/batch/jobs/install/templates/create_sidekiq_batch_tables.rb.tt

@@ -4,6 +4,8 @@ class CreateSidekiqBatchTables < ActiveRecord::Migration<%= migration_version %>
       t.string   :description
       t.integer  :status,            null: false, default: 0
       t.integer  :total_jobs,        null: false, default: 0
+      t.integer  :complete_count
+      t.integer  :failed_count
       t.jsonb    :callbacks,         null: false, default: {}
       t.jsonb    :callbacks_fired,   null: false, default: {}
       t.jsonb    :context,           null: false, default: {}

data/lib/generators/sidekiq/batch/jobs/upgrade/templates/upgrade_sidekiq_batch_tables.rb.tt

@@ -0,0 +1,5 @@
+class <%= migration_class_name %> < ActiveRecord::Migration<%= migration_version %>
+  def change
+<%= migration_body %>
+  end
+end

data/lib/generators/sidekiq/batch/jobs/upgrade/upgrade_generator.rb

@@ -0,0 +1,129 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+
+module Sidekiq
+  module Batch
+    module Jobs
+      module Generators
+        # Brings an existing installation up to the schema this version of the
+        # gem expects.
+        #
+        # It reads the host's real tables and emits only what they are missing,
+        # rather than shipping one migration per release. A user upgrading from
+        # 0.1.0 and a user upgrading from the release before this one run the
+        # same command and each get exactly the difference that applies to
+        # them, in one migration. Running it on a current database writes
+        # nothing at all, so it is safe to run whenever a release mentions the
+        # schema, and safe to run twice.
+        class UpgradeGenerator < ::Rails::Generators::Base
+          include ::Rails::Generators::Migration
+
+          source_root File.expand_path("templates", __dir__)
+
+          desc "Adds whatever the sidekiq-batch-jobs tables are missing for this version of the gem."
+
+          def self.next_migration_number(dirname)
+            ::ActiveRecord::Generators::Base.next_migration_number(dirname)
+          end
+
+          def copy_migration
+            return report_missing_tables if missing_tables.any?
+            return report_up_to_date     if changes.empty?
+
+            migration_template("upgrade_sidekiq_batch_tables.rb.tt", "db/migrate/#{migration_basename}.rb")
+          end
+
+          # NOTE: must stay private. Thor turns every public instance method on
+          # a generator into a step it invokes.
+          private
+
+          # Named for the version it upgrades to, so a later release's migration
+          # cannot collide with this one: two files called
+          # `upgrade_sidekiq_batch_tables` would be a duplicate migration name.
+          def migration_basename
+            "upgrade_sidekiq_batch_tables_to_v#{::Sidekiq::Batch::Jobs::VERSION.tr(".", "_")}"
+          end
+
+          # The template cannot hardcode a version: `ActiveRecord::Migration[7.1]`
+          # is unknown on Rails 6.1. Track whatever the host app runs instead.
+          def migration_version
+            "[#{::ActiveRecord::Migration.current_version}]"
+          end
+
+          # Read by the template.
+          def migration_body
+            changes.map { |line| "    #{line}" }.join("\n")
+          end
+
+          def changes
+            @changes ||= added_columns + added_indexes + dropped_indexes
+          end
+
+          def added_columns
+            Schema::COLUMNS.flat_map do |table, columns|
+              columns
+                .reject { |name, _type, _options| connection.column_exists?(table, name) }
+                .map    { |name, type, options| "add_column :#{table}, :#{name}, :#{type}#{arguments(options)}" }
+            end
+          end
+
+          def added_indexes
+            Schema::INDEXES.flat_map do |table, indexes|
+              indexes
+                .reject { |columns, _options| index?(table, columns) }
+                .map    { |columns, options| "add_index :#{table}, #{columns.inspect}#{arguments(options)}" }
+            end
+          end
+
+          def dropped_indexes
+            Schema::SUPERSEDED_INDEXES.flat_map do |table, indexes|
+              indexes
+                .select { |columns| index?(table, columns) }
+                .map    { |columns| "remove_index :#{table}, #{columns.inspect}" }
+            end
+          end
+
+          # Deliberately unqualified by the index's options. An index on the
+          # right columns is what the gem's queries need; rebuilding one a host
+          # has already tuned differently is not this generator's business.
+          def index?(table, columns)
+            connection.index_exists?(table, columns)
+          end
+
+          def arguments(options)
+            return "" if options.empty?
+
+            ", #{options.map { |key, value| "#{key}: #{value.inspect}" }.join(", ")}"
+          end
+
+          def missing_tables
+            @missing_tables ||= Schema::TABLES.reject { |table| connection.table_exists?(table) }
+          end
+
+          def report_missing_tables
+            say_status :skip,
+                       "#{missing_tables.join(" and ")} not found. This upgrades an existing installation; " \
+                       "run `rails g sidekiq:batch:jobs:install` to create the tables.",
+                       :yellow
+          end
+
+          def report_up_to_date
+            say_status :identical, "the sidekiq-batch-jobs tables are already current", :blue
+          end
+
+          # Reported rather than raised as a connection error, since the reason
+          # this generator needs a database at all is not obvious from one.
+          def connection
+            @connection ||= ::ActiveRecord::Base.connection
+          rescue ::ActiveRecord::ActiveRecordError => e
+            raise ::Rails::Generators::Error,
+                  "sidekiq-batch-jobs: this generator compares your schema against the one this version " \
+                  "expects, and the database could not be read (#{e.class}: #{e.message})."
+          end
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq/batch/jobs/schema.rb

@@ -0,0 +1,74 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module Batch
+    module Jobs
+      # The tables this version of the gem expects, as data rather than as a
+      # `create_table` block, so the upgrade generator can diff them against a
+      # host application's real schema and emit only what is missing. That is
+      # what lets one command upgrade a database from any earlier version,
+      # rather than a per-release migration the user has to apply in order.
+      #
+      # The install generator's template is this same schema written out for a
+      # fresh database. Two files that have to agree, and the upgrade generator
+      # spec catches it when they stop: run against a database built from the
+      # current schema, this diff has to come out empty.
+      module Schema
+        BATCHES = :sidekiq_batches
+        JOBS    = :sidekiq_batch_jobs
+
+        TABLES = [BATCHES, JOBS].freeze
+
+        # [name, type, options]. `id` and the timestamps are left out
+        # deliberately: both have been on both tables since 0.1.0, and neither
+        # is something an upgrade could sensibly add to a populated table.
+        COLUMNS = {
+          BATCHES => [
+            [:description,       :string,   {}],
+            [:status,            :integer,  { null: false, default: 0 }],
+            [:total_jobs,        :integer,  { null: false, default: 0 }],
+            [:complete_count,    :integer,  {}],
+            [:failed_count,      :integer,  {}],
+            [:callbacks,         :jsonb,    { null: false, default: {} }],
+            [:callbacks_fired,   :jsonb,    { null: false, default: {} }],
+            [:context,           :jsonb,    { null: false, default: {} }],
+            [:failure_policy,    :string,   {}],
+            [:failure_tolerance, :integer,  {}],
+            [:enrollment_error,  :jsonb,    {}],
+            [:completed_at,      :datetime, {}],
+            [:callback_fired_at, :datetime, {}]
+          ].freeze,
+          JOBS    => [
+            [:sidekiq_batch_id, :bigint,  { null: false }],
+            [:jid,              :string,  { null: false }],
+            [:worker_class,     :string,  { null: false }],
+            [:args,             :jsonb,   { null: false, default: [] }],
+            [:status,           :integer, { null: false, default: 0 }],
+            [:error_class,      :string,  {}],
+            [:error_message,    :text,    {}]
+          ].freeze
+        }.freeze
+
+        # [columns, options].
+        INDEXES = {
+          BATCHES => [
+            [%i[status callback_fired_at], {}],
+            [%i[status created_at],        {}]
+          ].freeze,
+          JOBS    => [
+            [%i[jid],                      { unique: true }],
+            [%i[sidekiq_batch_id status],  {}],
+            [%i[updated_at],               {}]
+          ].freeze
+        }.freeze
+
+        # Indexes an older version created that a current one should not keep.
+        # The bare `status` index goes because both composites above lead with
+        # `status`, and Postgres serves a status-only lookup from either.
+        SUPERSEDED_INDEXES = {
+          BATCHES => [%i[status]].freeze
+        }.freeze
+      end
+    end
+  end
+end

data/lib/sidekiq/batch/jobs/version.rb

@@ -3,7 +3,7 @@
 module Sidekiq
   module Batch
     module Jobs
-      VERSION = "0.2.0"
+      VERSION = "0.3.0"
     end
   end
 end

data/lib/sidekiq/batch/jobs.rb

@@ -2,6 +2,7 @@
 
 require "sidekiq"
 require_relative "jobs/version"
+require_relative "jobs/schema"
 require_relative "jobs/enum_compat"
 require_relative "jobs/failure_policy"
 require_relative "jobs/configuration"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-batch-jobs
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.3.0
 platform: ruby
 authors:
 - Douglas Greyling
@@ -101,11 +101,14 @@ files:
 - app/models/sidekiq_batch_job.rb
 - lib/generators/sidekiq/batch/jobs/install/install_generator.rb
 - lib/generators/sidekiq/batch/jobs/install/templates/create_sidekiq_batch_tables.rb.tt
+- lib/generators/sidekiq/batch/jobs/upgrade/templates/upgrade_sidekiq_batch_tables.rb.tt
+- lib/generators/sidekiq/batch/jobs/upgrade/upgrade_generator.rb
 - lib/sidekiq/batch/jobs.rb
 - lib/sidekiq/batch/jobs/configuration.rb
 - lib/sidekiq/batch/jobs/engine.rb
 - lib/sidekiq/batch/jobs/enum_compat.rb
 - lib/sidekiq/batch/jobs/failure_policy.rb
+- lib/sidekiq/batch/jobs/schema.rb
 - lib/sidekiq/batch/jobs/version.rb
 homepage: https://github.com/douglasgreyling/sidekiq-batch-jobs
 licenses: