custom_counter_cache 0.3.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c9162e152f23f2e1bcbe16bc8eac89109fb42e3710f056e280ffc7e2509972ea
-  data.tar.gz: 48a92f691a0d60be81222f930ee511cf7cc5af62b90a2d6dbb940d2bd6c40e83
+  metadata.gz: 56544a2488e48dc35c3ad64e7eb7cb0bc2c0be1bc3f5aa8ee5a52a349de2fcb8
+  data.tar.gz: 5d40812f87a2d9ce720e77558ac1cec17c81a607170d2a9e690b56bcc9481e29
 SHA512:
-  metadata.gz: a5140b21c810447e0f8593b57252ef42064cecfc26168a46df39fa2f813f042e54aabb32c602f275c6ead13f30fc4c05b929db0cb2ffbb557367be1a4b628ef9
-  data.tar.gz: 8a0101a516d27d8b076d97173fa1733b6d4b53f85877448229c30afaa60ea78bb7059fda11e41602de505148497f79a36622bb394f968340f8947fc788b68725
+  metadata.gz: 287db2bc86cca62e838cc32849eb1bb43fe66f9d40e771981bb56df221617e6bdf4392950808cafa031afb8a90fe9682ca68053bc0867ad5df802a76ca47f073
+  data.tar.gz: e096528b08c44673508784b4faa7bdaa500d5f5f30b5844d4d2a68a1dcf77ff241df30d4805ad58eb22f126f6fb821fc9214c5114fdf4ee520eca44e15ee3981

data/CHANGELOG.md

@@ -1,5 +1,90 @@
 # Changelog
 
+## 0.4.0
+
+### Breaking
+
+- Requires Ruby >= 3.3 and Rails (Active Record) >= 8.0.
+- `:if`/`:unless` on `update_counter_cache` now only gate the create and update callbacks; a
+  destroy always recounts. Use `except: [:destroy]` to skip it.
+- Column counters are written with `update_column`: recounts no longer save other unsaved
+  changes on the owner, touch its `updated_at`, run its callbacks or bump `lock_version`.
+- `update_counter_cache` raises `ArgumentError` for associations other than `belongs_to`
+  (they have never worked).
+- Every model that calls `define_counter_cache` gets the `counters` association, not only models
+  with a counter that has no column.
+- Defining counters on a table that doesn't exist no longer silently defines nothing.
+- Recounts triggered by `update_counter_cache` callbacks run after the saving transaction commits
+  instead of inside it: once per owner and counter per transaction, not at all on rollback, and
+  under a short lock on the owner's row. Counting inside the transaction couldn't see a concurrent
+  save's uncommitted child, so concurrent saves to the same owner left a stale count (on
+  PostgreSQL, 399 of 400 concurrent test runs). The counter now changes at commit rather than at
+  save; call `update_<name>` directly to recount immediately inside a transaction. Rails'
+  transactional tests commit each save within the test transaction, so tests see the recount.
+- A counter stored in the counters table raises `ArgumentError` for a value that isn't a whole
+  number (e.g. `2.5`, `'many'`, infinity) instead of silently truncating it with `to_i`. `nil`
+  still means 0, and whole values like `4.0` or `BigDecimal('4')` are still accepted. Use a column
+  of a suitable type or `store: :cache` for fractional values.
+
+### Changed
+
+- Whether a counter uses a column or the counters table is decided when it's read or written, so
+  loading models never queries the database, and a column added after the model has loaded is
+  picked up. The Heroku `DATABASE_URL` workaround is gone.
+- Counter rows are deleted on destroy only when the model has a counter without a column, so
+  column-only models never touch the counters table.
+- A recount that raises after commit is passed to `ActiveSupport.error_reporter.unexpected` (with
+  the owner and counter as context) and logged, instead of propagating: the save has already
+  committed, and one failure no longer skips the other recounts queued for that commit. In
+  development and test Rails' debug mode still raises it. Direct `update_<name>` calls,
+  `recount_counter_caches` and `CustomCounterCache::RecountJob` still raise.
+- Paranoia's `restore` now recounts the owner for callbacks declared after `acts_as_paranoid`; it
+  didn't before. Add `except: [:restore]` to keep the old behavior.
+
+### Added
+
+- `CustomCounterCache.counter_class_name` (default `'Counter'`) sets the model behind the
+  counters table.
+- `define_counter_cache :name, touch: true` (or `touch: :column`) sets a timestamp on the owner
+  whenever the counter is recounted. A column counter is written in the same UPDATE, and no
+  callbacks run.
+- `define_counter_cache :name, store: :cache, expires_in:` keeps a counter in
+  `CustomCounterCache.cache_store` (defaults to `Rails.cache`). It is computed on a miss and
+  deleted after commit when a child changes, instead of being recounted. Destroying the owner
+  deletes its keys.
+- `update_counter_cache ..., on_change: [:attr]` recounts on update only when a listed attribute
+  or the association's key columns changed. Create and destroy always recount.
+- `update_counter_cache [:article, :user], :comments_count` follows a path of `belongs_to`
+  associations and recounts both the old and the new owner at the end of it (grandparent counters).
+- `update_counter_cache ..., recount: :later` recounts in `CustomCounterCache::RecountJob`
+  (Active Job), enqueued after commit, once per owner and counter per transaction.
+- `CustomCounterCache.batch { }` recounts each owner and counter once after the block ends.
+- `CustomCounterCache.skip { }` drops recounts inside the block.
+- `Model.recount_counter_caches(*names, scope:, batch_size:)` recounts every record (or a scope) in
+  batches, repairing drift from `update_all`, `delete_all` and imports.
+- The `custom_counter_cache:recount` rake task (`MODEL=`, `COUNTERS=`, `BATCH_SIZE=`) wraps
+  `recount_counter_caches`.
+- Composite primary keys: owners and `belongs_to` associations with composite keys work with
+  column and cache counters, reassignment (including a change to only some key columns),
+  `on_change:`, `recount: :later` and `recount_counter_caches`. The counters table can't hold a
+  composite key, so a counter there raises `ArgumentError` naming the alternatives.
+- `only:` and `except:` on `update_counter_cache` accept `:restore`, the event fired by Paranoia's
+  `restore`.
+
+### Upgrading
+
+- If an `:if`/`:unless` condition was meant to skip destroys, add `except: [:destroy]`.
+- If you relied on a recount updating the owner's `updated_at`, pass `touch: true` to
+  `define_counter_cache`. Owner callbacks no longer run on a recount; call them explicitly if needed.
+- If a migration adds a counter column and backfills it, call `reset_column_information` on the
+  model before backfilling.
+- Counters now change when the saving transaction commits. Code that saves a child and reads the
+  owner's counter inside the same transaction should call `update_<name>` first.
+- A counters-table counter whose block can return a fractional value (an average, a sum of
+  decimals) now raises: move it to a column of a suitable type or to `store: :cache`.
+- On a model with a string or UUID primary key, the counters table's `countable_id` must be that
+  type (`t.references :countable, polymorphic: true, type: :uuid`).
+
 ## 0.3.2
 
 ### Fixed

data/README.rdoc

@@ -6,8 +6,8 @@ This is a simple approach to creating a custom counter cache in Rails that can b
 
 === Requirements
 
-* Ruby >= 3.1
-* Rails (Active Record) >= 7.2
+* Ruby >= 3.3
+* Rails (Active Record) >= 8.0
 
 CI covers the Ruby and Rails versions that are not yet end-of-life.
 
@@ -17,38 +17,180 @@ Add the following to your Gemfile:
 
   gem 'custom_counter_cache'
 
-=== Example
+== Defining a counter
 
-== Class with counter cache
-
-This is the block that will be used to calculate the value for the counter cache. It will be called by other models through their association via an after_save or after_destroy callback.
+The block calculates the counter's value from its owner. It is called whenever a model that
+declares +update_counter_cache+ (see below) changes.
 
   include CustomCounterCache::Model
   define_counter_cache :articles_count do |user|
     user.articles.where(state: 'published').count
   end
 
-== Class with callbacks
+Pass <tt>touch: true</tt> (or a column name, e.g. <tt>touch: :counted_at</tt>) to also set that
+timestamp, <tt>updated_at</tt> for +true+, on every recount. A column counter is written in the same
+UPDATE. No callbacks run.
+
+Where the value is kept depends on the counter: a column of the same name, a row in a shared
+counters table, or a cache store (<tt>store: :cache</tt>). See Choosing storage below.
 
-This will define the after_create, after_update and after_destroy callbacks. An :if option can be provided to limit when these callbacks get triggered.
+== Triggering recounts
+
+Declare which models change the counter with +update_counter_cache+, naming a +belongs_to+
+association and the counter on its owner. It defines the after_create, after_update and
+after_destroy callbacks (plus after_restore for Paranoia, see Soft deletes).
 
   include CustomCounterCache::Model
-  update_counter_cache :user, :articles_count, if: -> (article) { article.destroyed? || article.saved_change_to_state? }
+  update_counter_cache :user, :articles_count, if: -> (article) { article.saved_change_to_state? }
+
+These callbacks can be added to any number of models that might need to change the counter.
+When a record moves to another owner, both the old and the new owner are recounted.
+
+Options:
+
+* <tt>:if</tt>, <tt>:unless</tt>: limit when the create and update callbacks run. The callbacks
+  run after the record is saved, so use <tt>saved_change_to_state?</tt> rather than
+  <tt>state_changed?</tt>, which is always false by then. They don't apply to destroy or restore:
+  those have no saved changes, and an extra recount is never wrong.
+* <tt>on_change:</tt>: an attribute or an array. Recount on update only when one of them changed;
+  prefer it to +:if+ for plain attribute checks. Create and destroy always recount. It combines
+  with +:if+ and +:unless+ (both must allow the update).
+* <tt>only:</tt>, <tt>except:</tt>: limit the events, any of <tt>:create</tt>, <tt>:update</tt>,
+  <tt>:destroy</tt> and <tt>:restore</tt>, e.g. <tt>only: [:create, :destroy]</tt> or
+  <tt>except: [:destroy]</tt>.
+* <tt>:prepend</tt>: prepend the callbacks instead of appending them.
+* <tt>recount: :later</tt>: recount in a background job instead of right after commit. See
+  When recounts run.
+
+For example, to recount only when the state changes:
+
+  update_counter_cache :user, :articles_count, on_change: [:state]
+
+The association's key columns (plus the type column for a polymorphic one) are always watched
+by +on_change+, so moving the record to another owner recounts both owners.
+
+== Grandparent counters
+
+Pass a path of +belongs_to+ associations to recount an owner further up. For a
+<tt>comments_count</tt> on User across all of a user's articles:
+
+  class User < ApplicationRecord
+    include CustomCounterCache::Model
+    has_many :articles
+    has_many :comments, through: :articles
+    define_counter_cache(:comments_count) { |user| user.comments.count }
+  end
+
+  class Comment < ApplicationRecord
+    include CustomCounterCache::Model
+    belongs_to :article
+    update_counter_cache [:article, :user], :comments_count
+  end
+
+  class Article < ApplicationRecord
+    include CustomCounterCache::Model
+    belongs_to :user
+    update_counter_cache :user, :comments_count # an article moving users changes both counts
+  end
+
+Moving a comment recounts the old and the new user, once if they're the same user. The second
+declaration is needed because an Article changing user also changes both users' counts, and only
+Article's own callbacks see that. A step after a polymorphic one is skipped for records whose
+class has no such association; any other missing or non-+belongs_to+ step raises ArgumentError.
+
+== Soft deletes
+
+Discard soft-deletes with an ordinary update, so the callbacks already fire: filter with
+<tt>.kept</tt> in the counter block, and if you use <tt>on_change:</tt> include
+<tt>:discarded_at</tt> in it, or discarding won't recount.
+
+  define_counter_cache(:pages_count) { |notebook| notebook.pages.kept.count }
+  update_counter_cache :notebook, :pages_count, on_change: [:state, :discarded_at]
+
+Paranoia's +destroy+ runs the destroy callbacks, and +restore+ is recounted too, provided
++acts_as_paranoid+ is declared before +update_counter_cache+. Skip it with <tt>except: [:restore]</tt>
+(<tt>:restore</tt> is also accepted by <tt>only:</tt>).
+
+  class Parcel < ApplicationRecord
+    acts_as_paranoid
+    belongs_to :crate
+    update_counter_cache :crate, :parcels_count
+  end
+
+== Batching and skipping
+
+Every triggering save recounts its owner. For bulk work, batch the recounts so each owner and
+counter is recounted once, when the block ends (or when its transaction commits):
+
+  CustomCounterCache.batch do
+    rows.each { |row| article.comments.create!(row) } # one recount of article.comments_count
+  end
+
+Batches nest (the outermost one flushes) and still flush if the block raises, since a recount
+reflects whatever is in the database. To skip recounts entirely, e.g. for an import you'll
+recount afterwards:
+
+  CustomCounterCache.skip { import_comments }
 
-The callbacks run after the record is saved, so use <tt>saved_change_to_state?</tt> rather than
-<tt>state_changed?</tt> (which is always false by then). The :if condition also applies to
-after_destroy, where there are no saved changes, hence the <tt>destroyed?</tt> check.
+Both are per thread (and per fiber).
 
-Other options: <tt>:unless</tt>, <tt>:only</tt> / <tt>:except</tt> (e.g. <tt>only: [:create, :destroy]</tt>)
-and <tt>:prepend</tt>.
+== When recounts run
 
-These callbacks can be added to any number of models that might need to change the counter cache.
+Recounts triggered by these callbacks run after the saving transaction commits, once per owner
+and counter per transaction, and not at all if it rolls back. Each takes a short lock on the
+owner's row while it counts, so two saves committing at once can't leave a stale value: the
+later recount always counts after the earlier commit. (Counting inside the saving transaction
+can't see a concurrent save's uncommitted child, so one of them would overwrite the other.)
 
-== Counter Cache
+The counter therefore changes when the transaction commits, not when the child is saved. Inside
+a transaction, call the update method yourself if you need the new value straight away:
 
-To store the counter cache you need to create a column for the model with the counter cache (example: articles_count).
+  Article.transaction do
+    user.articles.create!(attrs)
+    user.update_articles_count # recounts now; update_* and recount_counter_caches never wait
+  end
+
+If a recount raises after commit, the save has already succeeded and the count can be rebuilt, so
+the error doesn't propagate to the caller or stop the other recounts queued for that commit. It's
+passed to <tt>Rails.error.unexpected</tt>, which raises in development and test (with
+<tt>consider_all_requests_local</tt>, as by default) and in production reports it to your error
+tracker with the owner and counter in the context. It's also logged. Calling
+<tt>update_<name></tt> or <tt>recount_counter_caches</tt> yourself raises as usual, and so does
+<tt>CustomCounterCache::RecountJob</tt>, so your queue's retries apply.
+
+To recount in a background job instead, enqueued after commit:
+
+  update_counter_cache :user, :articles_count, recount: :later # CustomCounterCache::RecountJob (Active Job)
+
+To pick a queue, <tt>CustomCounterCache::RecountJob.queue_as :low</tt>. Rails' transactional
+tests commit each save within the test transaction, so recounts happen there as they do in
+production.
+
+== Choosing storage
+
+[Column]         Add a column with the counter's name. Fastest to read, and you can sort and
+                 filter by it in SQL. Needs a migration per counter.
+[Counters table] Used automatically when there's no column. One shared table, no migration per
+                 counter; preload with <tt>includes(:counters)</tt>. See The counters table.
+[Cache]          <tt>define_counter_cache :x, store: :cache, expires_in: 12.hours</tt>. Kept in
+                 <tt>CustomCounterCache.cache_store</tt> (defaults to Rails.cache: Redis,
+                 Memcached, Solid Cache...). Reads compute on a miss; a child change deletes the
+                 key after commit instead of recounting, so writes are cheap and never lock the
+                 owner's row. Destroying the owner deletes its keys. Not visible to SQL, and a
+                 delete racing a concurrent read can leave a stale value for up to +expires_in+.
+
+Column or counters table is decided when the counter is read or written, not when the model
+loads, so defining counters never touches the database, and a column added later is picked up.
 
-If you would like to store all of your counter caches in a single table, you can use this migration:
+To use a column, add one:
+
+  def change
+    add_column :users, :articles_count, :integer, default: 0, null: false
+  end
+
+== The counters table
+
+To store counters in a single shared table instead, use this migration:
 
   create_table :counters do |t|
     t.references :countable, polymorphic: true
@@ -58,6 +200,17 @@ If you would like to store all of your counter caches in a single table, you can
   end
   add_index :counters, [ :countable_id, :countable_type, :key ], unique: true
 
+The table holds whole numbers. A block result of +nil+ is stored as 0, and whole values such as
+<tt>4.0</tt> or <tt>BigDecimal('4')</tt> are converted, but a fractional result such as an average
+raises ArgumentError rather than being truncated: give that counter a column of a suitable type
+(e.g. decimal) or use <tt>store: :cache</tt>.
+
++countable_id+ must match your models' primary key type: for string or UUID keys, use
+<tt>t.references :countable, polymorphic: true, type: :uuid</tt> (or <tt>:string</tt>). A model
+with a composite primary key can't use this table, since +countable_id+ holds one value: give its
+counters a column or <tt>store: :cache</tt> (it raises ArgumentError otherwise). Composite keys
+work everywhere else, including composite foreign keys on the +belongs_to+ side.
+
 Here is the example model to go with:
 
   class Counter < ActiveRecord::Base
@@ -65,23 +218,44 @@ Here is the example model to go with:
     validates :countable, presence: true
   end
 
-Note: do not add <tt>dependent: :destroy</tt> to that +belongs_to+. On a
-+belongs_to+, <tt>dependent: :destroy</tt> means "when this Counter is
-destroyed, also destroy the record it belongs to" -- the opposite of what you
-want, and it creates a destroy loop: destroying a countable record cascades
-into destroying its Counters (via the <tt>has_many :counters, dependent:
-:delete_all</tt> this gem defines), and a wrongly-configured +belongs_to+ would
-then cascade back into destroying the same countable record again. The
-countable side's <tt>dependent: :delete_all</tt> already handles cleanup and
-skips Counter's own callbacks entirely, so no +dependent+ option belongs on
-the +belongs_to+ side at all.
+To use a different model name (for example if +Counter+ is already taken), set
+<tt>CustomCounterCache.counter_class_name</tt> (see Configuration).
 
-If you would like to store your counter cache in an existing table, you can use this migration:
+When a record is destroyed, its counter rows are removed with a single DELETE, without loading
+them or running Counter's callbacks. Do not add <tt>dependent: :destroy</tt> to the +belongs_to+
+above: on a +belongs_to+ it means "destroying this Counter also destroys its owner".
 
-  def change
-    add_column :users, :articles_count, :integer, default: 0, null: false
-  end
+== Backfilling and repairing
+
+To backfill your counters, or repair them later, recount every record from the console or a migration:
+
+  User.recount_counter_caches
+
+or only some counters and records:
+
+  User.recount_counter_caches(:articles_count, scope: User.where(id: 1..1000), batch_size: 500)
+
+It returns the number of records processed. The same is available as a rake task:
+
+  bin/rails custom_counter_cache:recount MODEL=User COUNTERS=articles_count
+
++COUNTERS+ (comma-separated, default all) and +BATCH_SIZE+ (default 1000) are optional.
+
+Callbacks can't see +update_all+, +delete_all+, +insert_all+ or SQL imports, so counts drift after
+them. Run the recount periodically, or after an import, to repair that. It calls +update_<name>+
+directly, so it also works inside <tt>CustomCounterCache.skip { }</tt>.
+
+In a migration that also adds the column, call <tt>User.reset_column_information</tt> first so the
+backfill writes to the new column.
+
+== Configuration
+
+Set these in an initializer (<tt>config/initializers/custom_counter_cache.rb</tt>):
 
-To backfill your counters, run something like this either in a migration or in the console:
+* <tt>CustomCounterCache.counter_class_name</tt>: the model behind the counters table. Default
+  <tt>'Counter'</tt>. Must be set before your models load.
+* <tt>CustomCounterCache.cache_store</tt>: the store for <tt>store: :cache</tt> counters. Defaults
+  to Rails.cache.
 
-  User.select(:id).find_each(batch_size: 100) { |u| u.update_articles_count }
+  CustomCounterCache.counter_class_name = 'CounterCache'
+  CustomCounterCache.cache_store = ActiveSupport::Cache::MemoryStore.new

data/lib/custom_counter_cache/dispatcher.rb

@@ -0,0 +1,101 @@
+require 'active_support/isolated_execution_state'
+
+module CustomCounterCache
+  # Recounts each owner and counter once after the outermost batch ends (after commit, if inside a transaction).
+  def self.batch(&block)
+    Dispatcher.batch(&block)
+  end
+
+  # Drops recounts inside the block, e.g. for an import followed by recount_counter_caches.
+  def self.skip(&block)
+    Dispatcher.skip(&block)
+  end
+
+  # Single path from update_counter_cache's callbacks to an owner's recount.
+  module Dispatcher
+    # Always after commit: inside the saving transaction a recount can't see a concurrent save's
+    # uncommitted child, and the later of the two overwrites the count with a stale one.
+    def self.recount(owner, name, timing = :after_commit)
+      return if state[:skip]
+      # A cached counter is invalidated after commit, never recounted in place.
+      timing = :invalidate if owner.class.custom_counter_cache_storage(name) == :cache
+      # Keep the first instance seen, so loaded counters on it stay current.
+      return state[:batch][[owner.class, owner.id, name]] ||= [owner, timing] if state[:batch]
+      defer(owner, name, timing)
+    end
+
+    # Serializes recounts of one owner on its row, so each counts after any concurrent commit.
+    # Locks by query rather than with_lock, which would reload the owner and drop its unsaved changes.
+    def self.locked_recount(owner, name)
+      klass = owner.class
+      klass.transaction do
+        # Owner deleted meanwhile: nothing left to recount.
+        next unless klass.where(CustomCounterCache.key_conditions(klass.primary_key, owner.id)).lock.pick(*Array(klass.primary_key).first(1))
+        owner.public_send("update_#{name}")
+      end
+    end
+
+    def self.batch
+      outermost = state[:batch].nil?
+      state[:batch] ||= {}
+      yield
+    ensure
+      # A recount reflects whatever is in the database, so flushing after an exception is still correct.
+      if outermost
+        pending = state.delete(:batch)
+        pending.each { |(_, _, name), (owner, timing)| recount(owner, name, timing) }
+      end
+    end
+
+    def self.skip
+      skipping = state[:skip]
+      state[:skip] = true
+      yield
+    ensure
+      state[:skip] = skipping
+    end
+
+    # Once per owner and counter per transaction. A recount reads committed state, so running it late is still correct.
+    def self.defer(owner, name, timing)
+      key = [owner.class, owner.id, name, timing]
+      deferred = state[:deferred] ||= {}
+      return if deferred.key?(key)
+
+      transaction = owner.class.current_transaction
+      deferred[key] = true if transaction.open?
+      # A rolled-back savepoint drops its after_commit, so forget the key and let a later save schedule it again.
+      transaction.after_rollback { deferred.delete(key) }
+      transaction.after_commit do
+        deferred.delete(key)
+        perform(owner, name, timing)
+      rescue StandardError => error
+        failed(error, owner, name)
+      end
+    end
+
+    # The save has committed and the count can be rebuilt, so don't make the save look failed or
+    # stop the other recounts queued for this commit. unexpected still raises in development and test.
+    def self.failed(error, owner, name)
+      context = { owner: owner.class.name, owner_id: owner.id, counter: name.to_s }
+      owner.class.logger&.error("custom_counter_cache: recount of #{context[:owner]} #{context[:owner_id].inspect} " \
+                                "#{context[:counter]} failed: #{error.class}: #{error.message}")
+      ActiveSupport.error_reporter.unexpected(error, context: context, source: 'custom_counter_cache')
+    end
+
+    def self.perform(owner, name, timing)
+      case timing
+      when :later then RecountJob.perform_later(owner.class.name, owner.id, name.to_s)
+      when :invalidate
+        CustomCounterCache.cache_store.delete(CustomCounterCache.cache_key(owner, name))
+        touch_column = owner.class.custom_counter_cache_touch_column(name)
+        # A cascade destroy invalidates after the owner is gone; there's nothing left to touch.
+        owner.update_column(touch_column, Time.current) if touch_column && !owner.destroyed?
+      else locked_recount(owner, name)
+      end
+    end
+
+    def self.state
+      ActiveSupport::IsolatedExecutionState[:custom_counter_cache] ||= {}
+    end
+  end
+end

data/lib/custom_counter_cache/model.rb

@@ -3,90 +3,125 @@ require 'active_support/concern'
 module CustomCounterCache::Model
   extend ActiveSupport::Concern
 
+  included do
+    class_attribute :custom_counter_cache_names, instance_accessor: false, default: []
+    # Per counter name; a new merged hash is assigned so subclasses never mutate their parent's.
+    class_attribute :custom_counter_cache_options, instance_accessor: false, default: {}
+  end
+
   module ClassMethods
-    def define_counter_cache(cache_column, &block)
-      return unless table_exists?
-
-      # counter accessors
-      unless column_names.include?(cache_column.to_s)
-        # Declare once per class: redeclaring for each virtual counter triggers method-redefinition warnings.
-        has_many :counters, as: :countable, dependent: :delete_all unless reflect_on_association(:counters)
-        define_method "#{cache_column}" do
-          # Once loaded (e.g. includes(:counters)), a missing row means 0, not a query.
-          if counters.loaded?
-            counters.detect { |c| c.key == cache_column.to_s }.try(:value).to_i
+    # Column or counters row is decided on each call, not here, so defining never queries the database.
+    def define_counter_cache(cache_column, touch: nil, store: nil, expires_in: nil, &block)
+      raise ArgumentError, "store: must be :cache or omitted" unless store.nil? || store == :cache
+      name = cache_column.to_s
+      define_counters_association
+      define_cache_cleanup if store == :cache
+      self.custom_counter_cache_names += [name]
+      self.custom_counter_cache_options = custom_counter_cache_options.merge(name => { touch: touch, store: store, expires_in: expires_in })
+
+      custom_counter_cache_methods.module_eval do
+        define_method(name) do
+          case self.class.custom_counter_cache_storage(name)
+          when :column then super()
+          when :cache
+            CustomCounterCache.cache_store.fetch(CustomCounterCache.cache_key(self, name), expires_in: expires_in) { block.call(self) }
           else
-            counters.find_by(key: cache_column.to_s).try(:value).to_i
+            # Once loaded (e.g. includes(:counters)), a missing row means 0, not a query.
+            counter = counters.loaded? ? counters.detect { |c| c.key == name } : counters.find_by(key: name)
+            counter.try(:value).to_i
           end
         end
-        define_method "#{cache_column}=" do |count|
+
+        define_method("#{name}=") do |count|
+          case self.class.custom_counter_cache_storage(name)
+          when :column then return super(count)
+          when :cache then return CustomCounterCache.cache_store.write(CustomCounterCache.cache_key(self, name), count, expires_in: expires_in)
+          end
+          count = CustomCounterCache::Model.whole_number!(self, name, count)
           # Update the loaded Counter itself, or the reader keeps returning its stale value.
-          counter = counters.loaded? ? counters.detect { |c| c.key == cache_column.to_s } : counters.find_by(key: cache_column.to_s)
+          counter = counters.loaded? ? counters.detect { |c| c.key == name } : counters.find_by(key: name)
           if counter
-            counter.update_attribute :value, count.to_i
+            counter.update_attribute :value, count
           else
             begin
               # Savepoint: on PostgreSQL a failed INSERT would otherwise abort the caller's whole transaction.
-              self.class.transaction(requires_new: true) { counters.create key: cache_column.to_s, value: count.to_i }
+              self.class.transaction(requires_new: true) { counters.create key: name, value: count }
             rescue ActiveRecord::RecordNotUnique
               # Lost a create race. A locking read sees the winner's row even under REPEATABLE READ (MySQL).
               counters.reset
-              counters.lock.find_by!(key: cache_column.to_s).update_attribute :value, count.to_i
+              counters.lock.find_by!(key: name).update_attribute :value, count
             end
           end
         end
-      end
 
-      # counter update method
-      define_method "update_#{cache_column}" do
-        if self.class.column_names.include?(cache_column.to_s)
-          update_attribute cache_column, block.call(self)
-        else
-          send "#{cache_column}=", block.call(self)
+        define_method("update_#{name}") do
+          touch_column = self.class.custom_counter_cache_touch_column(name)
+          value = block.call(self)
+          if self.class.custom_counter_cache_storage(name) == :column
+            # One statement, and no callbacks: update_columns, not touch.
+            touch_column ? update_columns(name => value, touch_column => Time.current) : update_column(name, value)
+          else
+            send "#{name}=", value
+            update_column touch_column, Time.current if touch_column
+          end
         end
       end
+    end
 
-    rescue StandardError => e
-      # Support Heroku's database-less assets:precompile pre-deploy step:
-      raise e unless ENV['DATABASE_URL'].to_s.include?('//user:pass@127.0.0.1/')
+    # :cache when declared; otherwise a column if one exists, else a row in the counters table.
+    def custom_counter_cache_storage(name) # :nodoc:
+      return :cache if custom_counter_cache_options.dig(name.to_s, :store) == :cache
+      return :column if column_names.include?(name.to_s)
+      # The counters table's single countable_id can't hold a composite key.
+      if composite_primary_key?
+        raise ArgumentError, "#{self} has a composite primary key, so counter #{name} needs a column or store: :cache"
+      end
+      :counters
     end
 
-    def update_counter_cache(association, cache_column, options = {})
-      return unless table_exists?
+    def custom_counter_cache_touch_column(name) # :nodoc:
+      touch = custom_counter_cache_options.dig(name.to_s, :touch)
+      touch == true ? 'updated_at' : (touch.to_s if touch)
+    end
 
-      association  = association.to_sym
+    # Repairs drift no callback can see (update_all, delete_all, imports). Returns the records processed.
+    def recount_counter_caches(*names, scope: all, batch_size: 1000)
+      names = names.empty? ? custom_counter_cache_names : names.map(&:to_s)
+      unknown = names - custom_counter_cache_names
+      raise ArgumentError, "#{self} has no counter cache named #{unknown.join(', ')}" if unknown.any?
+
+      # Calls update_<name> directly, so CustomCounterCache.skip and .batch don't affect it.
+      # Preload :counters, which the virtual counter writer uses once loaded, to avoid a query per record.
+      scope = scope.includes(:counters) if names.any? { |name| custom_counter_cache_storage(name) == :counters }
+      processed = 0
+      scope.find_each(batch_size: batch_size) do |record|
+        names.each { |name| record.public_send("update_#{name}") }
+        processed += 1
+      end
+      processed
+    end
+
+    # association may be a belongs_to path, e.g. [:article, :user], to recount a grandparent.
+    def update_counter_cache(association, cache_column, options = {})
+      path         = Array(association).map(&:to_sym)
+      association  = path.first
       cache_column = cache_column.to_sym
-      method_name  = "callback_#{association}_#{cache_column}".to_sym
+      method_name  = "callback_#{path.join('_')}_#{cache_column}".to_sym
       reflection   = reflect_on_association(association)
-      raise ArgumentError, "#{self} must declare belongs_to :#{association} before update_counter_cache" unless reflection
+      raise ArgumentError, "#{self} must declare belongs_to :#{association} before update_counter_cache" unless reflection&.belongs_to?
       foreign_key  = reflection.foreign_key
+      rest         = path.drop(1)
+      path_checked = false
+      timing       = CustomCounterCache.check_timing!(options.fetch(:recount, :after_commit))
 
-      # define callback
       define_method method_name do
-        # update old association
-        if reflection.polymorphic?
-          type_key = reflection.foreign_type
-          id_key   = foreign_key
-          if saved_change_to_attribute?(id_key) || saved_change_to_attribute?(type_key)
-            old_type = saved_change_to_attribute?(type_key) ? attribute_before_last_save(type_key) : self[type_key]
-            old_id   = saved_change_to_attribute?(id_key)   ? attribute_before_last_save(id_key)   : self[id_key]
-            # safe_: the stored type may name a class that has since been renamed or removed.
-            old_klass = old_type&.safe_constantize
-            if ( old_klass && old_id && record = old_klass.find_by(reflection.association_primary_key(old_klass) => old_id) )
-              record.send("update_#{cache_column}")
-            end
-          end
-        else
-          if saved_change_to_attribute?(foreign_key)
-            old_id = attribute_before_last_save(foreign_key)
-            if ( old_id && record = reflection.klass.find_by(reflection.association_primary_key => old_id) )
-              record.send("update_#{cache_column}")
-            end
-          end
-        end
-        # update new association
-        if ( record = send(association) )
-          record.send("update_#{cache_column}")
+        # Later steps live on classes that may not be loaded at declaration time, so check them on first use.
+        path_checked ||= CustomCounterCache::Model.check_path!(reflection, rest)
+        owners = [CustomCounterCache::Model.previous_owner(self, reflection), public_send(association)]
+        owners = owners.map { |record| CustomCounterCache::Model.follow_path(record, rest) }
+        # Moving between two records under the same grandparent must recount it once, not twice.
+        owners.compact.uniq { |owner| [owner.class, owner.id] }.each do |owner|
+          CustomCounterCache::Dispatcher.recount(owner, cache_column, timing)
         end
       end
 
@@ -97,13 +132,104 @@ module CustomCounterCache::Model
 
       # set callbacks
       callback_opts = options.slice(:if, :unless, :prepend)
+      update_opts   = callback_opts
+      if options[:on_change]
+        # The keys are always watched so a reassignment recounts the old owner too.
+        watched = Array(options[:on_change]).map(&:to_s) + Array(foreign_key).map(&:to_s)
+        watched << reflection.foreign_type.to_s if reflection.polymorphic?
+        changed = ->(record) { watched.any? { |attribute| record.saved_change_to_attribute?(attribute) } }
+        update_opts = callback_opts.merge(if: Array(options[:if]) + [changed])
+      end
+      # Create isn't gated by :on_change: column defaults aren't saved changes, so new records would be missed.
       after_create  method_name, **callback_opts unless skip_callback.call(:create, options)
-      after_update  method_name, **callback_opts unless skip_callback.call(:update, options)
-      after_destroy method_name, **callback_opts unless skip_callback.call(:destroy, options)
+      after_update  method_name, **update_opts unless skip_callback.call(:update, options)
+      # Not :if/:unless: they test saved changes, which a destroy lacks. An extra recount is never wrong.
+      after_destroy method_name, **options.slice(:prepend) unless skip_callback.call(:destroy, options)
+      # Paranoia's restore skips update callbacks; it defines :restore only once acts_as_paranoid has run.
+      after_restore method_name, **options.slice(:prepend) if respond_to?(:after_restore) && !skip_callback.call(:restore, options)
+    end
+
+    private
+
+    # Included after Active Record's generated attribute methods, so super reaches a real column's accessor.
+    def custom_counter_cache_methods
+      @custom_counter_cache_methods ||= Module.new.tap { |mod| include mod }
+    end
+
+    def define_cache_cleanup
+      return if @custom_counter_cache_cleanup
+      @custom_counter_cache_cleanup = true
+      after_destroy_commit do
+        self.class.custom_counter_cache_names.each do |name|
+          next unless self.class.custom_counter_cache_storage(name) == :cache
+          CustomCounterCache.cache_store.delete(CustomCounterCache.cache_key(self, name))
+        end
+      end
+    end
+
+    def define_counters_association
+      return if reflect_on_association(:counters)
+      has_many :counters, as: :countable, class_name: CustomCounterCache.counter_class_name
+      # Not dependent: :delete_all, which would query the counters table even when every counter is a column.
+      before_destroy do
+        in_table = self.class.custom_counter_cache_names.any? { |name| self.class.custom_counter_cache_storage(name) == :counters }
+        counters.delete_all(:delete_all) if in_table
+      end
+    end
+  end
+
+  # The counters table holds integers. Converting 2.5 to 2 would store a wrong count without a word, so refuse.
+  def self.whole_number!(record, name, value)
+    number =
+      case value
+      when nil then 0
+      when Integer then value
+      when String then Integer(value, 10, exception: false)
+      when Numeric then value.to_i if (!value.respond_to?(:finite?) || value.finite?) && value == value.to_i
+      end
+    return number if number
+    raise ArgumentError, "#{record.class}##{name} is stored in the counters table, which holds whole numbers; got " \
+      "#{value.inspect}. Give it a column of a suitable type, or use store: :cache"
+  end
+
+  # Checks each step up to the first polymorphic one, past which the class varies per record.
+  def self.check_path!(reflection, steps)
+    steps.each do |step|
+      return true if reflection.polymorphic?
+      owner_class = reflection.klass
+      reflection = owner_class.reflect_on_association(step)
+      raise ArgumentError, "#{owner_class} must declare belongs_to :#{step} for update_counter_cache" unless reflection&.belongs_to?
+    end
+    true
+  end
+
+  def self.follow_path(record, steps)
+    steps.reduce(record) do |current, step|
+      break unless current
+      reflection = current.class.reflect_on_association(step)
+      # Only reachable past a polymorphic step: this type has no such parent, so there's nothing to recount.
+      break unless reflection
+      raise ArgumentError, "#{current.class} must declare belongs_to :#{step} for update_counter_cache" unless reflection.belongs_to?
+      current.public_send(step)
+    end
+  end
+
+  # The owner record belonged to before its last save moved it, or nil if it didn't move.
+  # Keys may be composite: only some of their columns may have changed.
+  def self.previous_owner(record, reflection)
+    keys = Array(reflection.foreign_key).map(&:to_s)
+    keys += [reflection.foreign_type] if reflection.polymorphic?
+    return unless keys.any? { |key| record.saved_change_to_attribute?(key) }
+
+    old_values = keys.map { |key| record.saved_change_to_attribute?(key) ? record.attribute_before_last_save(key) : record[key] }
+    return if old_values.any?(&:nil?)
 
-    rescue StandardError => e
-      # Support Heroku's database-less assets:precompile pre-deploy step:
-      raise e unless ENV['DATABASE_URL'].to_s.include?('//user:pass@127.0.0.1/')
+    if reflection.polymorphic?
+      # safe_: the stored type may name a class that has since been renamed or removed.
+      old_klass = old_values.pop.safe_constantize
+      old_klass&.find_by(CustomCounterCache.key_conditions(reflection.association_primary_key(old_klass), old_values))
+    else
+      reflection.klass.find_by(CustomCounterCache.key_conditions(reflection.association_primary_key, old_values))
     end
   end
 end

data/lib/custom_counter_cache/railtie.rb

@@ -0,0 +1,3 @@
+class CustomCounterCache::Railtie < Rails::Railtie
+  rake_tasks { load File.expand_path('../tasks/custom_counter_cache.rake', __dir__) }
+end

data/lib/custom_counter_cache/recount_job.rb

@@ -0,0 +1,16 @@
+begin
+  require 'active_job'
+rescue LoadError
+  raise LoadError, "custom_counter_cache: recount: :later needs Active Job; add 'activejob' to your Gemfile"
+end
+
+module CustomCounterCache
+  # Enqueued by update_counter_cache ..., recount: :later. Takes plain values, not the record, so no GlobalID is needed.
+  class RecountJob < ActiveJob::Base
+    def perform(class_name, id, name)
+      klass = class_name.safe_constantize
+      owner = klass&.find_by(CustomCounterCache.key_conditions(klass.primary_key, id))
+      Dispatcher.locked_recount(owner, name) if owner
+    end
+  end
+end

data/lib/custom_counter_cache/version.rb

@@ -1,5 +1,5 @@
 module CustomCounterCache
 
-  VERSION = '0.3.2'
+  VERSION = '0.4.0'
 
 end

data/lib/custom_counter_cache.rb

@@ -1,3 +1,37 @@
+require 'active_support'
+require 'active_support/core_ext/module/attribute_accessors'
+
 module CustomCounterCache
-  require 'custom_counter_cache/model'
+  # Model holding counters that have no column. Read when a model defines its first counter, so set it before models load.
+  mattr_accessor :counter_class_name, default: 'Counter'
+  mattr_writer :cache_store, default: nil
+
+  # Where store: :cache counters live: the configured store, else Rails.cache.
+  def self.cache_store
+    store = @@cache_store || (::Rails.cache if defined?(::Rails) && ::Rails.respond_to?(:cache))
+    store or raise ArgumentError, 'store: :cache needs a cache: set CustomCounterCache.cache_store (Rails.cache is used when available)'
+  end
+
+  def self.cache_key(owner, name)
+    "custom_counter_cache/v1/#{owner.class.polymorphic_name}/#{Array(owner.id).join('-')}/#{name}"
+  end
+
+  # Conditions matching columns to values; each may be a single key or a composite one.
+  def self.key_conditions(columns, values) # :nodoc:
+    Array(columns).map(&:to_s).zip(Array(values)).to_h
+  end
+
+  # Callback recounts always run after commit; :later moves them into a job.
+  TIMINGS = %i[after_commit later].freeze
+
+  def self.check_timing!(timing) # :nodoc:
+    raise ArgumentError, "recount: must be one of #{TIMINGS.join(', ')}" unless TIMINGS.include?(timing)
+    # Fail at class load, not on the first save, if Active Job is missing.
+    require 'custom_counter_cache/recount_job' if timing == :later
+    timing
+  end
 end
+
+require 'custom_counter_cache/dispatcher'
+require 'custom_counter_cache/model'
+require 'custom_counter_cache/railtie' if defined?(Rails::Railtie)

data/lib/tasks/custom_counter_cache.rake

@@ -0,0 +1,12 @@
+namespace :custom_counter_cache do
+  desc 'Recount counter caches: MODEL=User [COUNTERS=articles_count,comments_count] [BATCH_SIZE=1000]'
+  task recount: :environment do
+    abort 'MODEL is required, e.g. MODEL=User' if ENV['MODEL'].to_s.empty?
+    model = ENV['MODEL'].safe_constantize
+    abort "#{ENV['MODEL']} is not a model with counter caches" unless model.respond_to?(:recount_counter_caches)
+
+    names = ENV['COUNTERS'].to_s.split(',').map(&:strip).reject(&:empty?)
+    batch_size = Integer(ENV.fetch('BATCH_SIZE', 1000))
+    puts "Recounted #{model.recount_counter_caches(*names, batch_size: batch_size)} #{model} records"
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: custom_counter_cache
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.4.0
 platform: ruby
 authors:
 - Cedric Howe
@@ -15,7 +15,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '7.2'
+        version: '8.0'
     - - "<"
       - !ruby/object:Gem::Version
         version: '9.0'
@@ -25,7 +25,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '7.2'
+        version: '8.0'
     - - "<"
       - !ruby/object:Gem::Version
         version: '9.0'
@@ -35,7 +35,7 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '7.2'
+        version: '8.0'
     - - "<"
       - !ruby/object:Gem::Version
         version: '9.0'
@@ -45,10 +45,44 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '7.2'
+        version: '8.0'
     - - "<"
       - !ruby/object:Gem::Version
         version: '9.0'
+- !ruby/object:Gem::Dependency
+  name: activejob
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '8.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '9.0'
+- !ruby/object:Gem::Dependency
+  name: discard
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -69,6 +103,20 @@ dependencies:
     - - "<"
       - !ruby/object:Gem::Version
         version: '7'
+- !ruby/object:Gem::Dependency
+  name: paranoia
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.1'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -109,14 +157,19 @@ files:
 - LICENSE
 - README.rdoc
 - lib/custom_counter_cache.rb
+- lib/custom_counter_cache/dispatcher.rb
 - lib/custom_counter_cache/model.rb
+- lib/custom_counter_cache/railtie.rb
+- lib/custom_counter_cache/recount_job.rb
 - lib/custom_counter_cache/version.rb
+- lib/tasks/custom_counter_cache.rake
 homepage: https://github.com/cedric/custom_counter_cache
 licenses:
 - MIT
 metadata:
   source_code_uri: https://github.com/cedric/custom_counter_cache
   changelog_uri: https://github.com/cedric/custom_counter_cache/blob/main/CHANGELOG.md
+  rubygems_mfa_required: 'true'
 rdoc_options: []
 require_paths:
 - lib
@@ -124,7 +177,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '3.1'
+      version: '3.3'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="