counterwise 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4c53c61fffe70d6bed84b2d5e5da5349b4a4285dbd7c19a8523721c449dcdbcc
-  data.tar.gz: 23e702c8cc478833aa555ad28f4456912420036063bf53d30c35d61b426cbff9
+  metadata.gz: ac4f723a8cdebb397ea3c1035ee728599ce308ebe36fc5aaa9fb6afeff09474e
+  data.tar.gz: 5efc1f7fdbde64eab27962e0b37355ae16f8293c8173d101e85da97b81679dc7
 SHA512:
-  metadata.gz: 3dc224b1948220028726742754a51b4523147f86de67bac047a3159a0760782d68722ec5afcbd017ad5b5d3434430a6c8db960906360dae858f484f43177b8c6
-  data.tar.gz: 8ba0623f2cea3a11c538dbf10f8ac3cb0ebd49400b92a0617c630b62c297e34fefd6d6298a243f8615caeb238d47cffce4d5f1dd79e7701c01957737d958fb17
+  metadata.gz: 48b91a565a82de0da5dbdf3e6637721ae05ad4b5bea9e0b8c31366932e6b30f6460e526911a458ceda815910d1031046b2413eb0668207cd001c18afdb030002
+  data.tar.gz: 5d6fd86b0ab23cd378ed374c6bf91aab877626038f7c35acecd2b123777fb646340a4070e61e02cbbf98ce9ab521155c25d2e0b320de0879a927def69cf41b30

data/README.md

@@ -5,75 +5,73 @@
 Counting and aggregation library for Rails.
 
 - [Counter](#counter)
-  - [Main concepts](#main-concepts)
-  - [Defining a counter](#defining-a-counter)
-  - [Accessing counter values](#accessing-counter-values)
-  - [Anonymous counters](#anonymous-counters)
-  - [Defining a conditional counter](#defining-a-conditional-counter)
-  - [Aggregating a value (e.g. sum of order revenue)](#aggregating-a-value-eg-sum-of-order-revenue)
-  - [Recalculating a counter](#recalculating-a-counter)
-  - [Reset a counter](#reset-a-counter)
-  - [Verify a counter](#verify-a-counter)
-  - [Hooks](#hooks)
-  - [Testing the counters in production](#testing-the-counters-in-production)
-  - [TODO](#todo)
   - [Usage](#usage)
   - [Installation](#installation)
+  - [Main concepts](#main-concepts)
+  - [Basic usage](#basic-usage)
+    - [Define a counter](#define-a-counter)
+    - [Access counter values](#access-counter-values)
+    - [Recalculate a counter](#recalculate-a-counter)
+    - [Reset a counter](#reset-a-counter)
+    - [Verify a counter](#verify-a-counter)
+  - [Advanced usage](#advanced-usage)
+    - [Sort or filter parent models by a counter value](#sort-or-filter-parent-models-by-a-counter-value)
+    - [Aggregate a value (e.g. sum of order revenue)](#aggregate-a-value-eg-sum-of-order-revenue)
+    - [Hooks](#hooks)
+    - [Manual counters](#manual-counters)
+    - [Calculating a value from other counters](#calculating-a-value-from-other-counters)
+    - [Defining a conditional counter](#defining-a-conditional-counter)
+  - [Testing](#testing)
+    - [Using Rspec](#using-rspec)
+    - [In production](#in-production)
+  - [TODO](#todo)
   - [Contributing](#contributing)
   - [License](#license)
 
-By the time you need Rails counter_caches you probably have other needs too. You probably want to sum column values and you probably have enough throughput that updating a single column value will cause lock contention problems too.
+By the time you need Rails counter_caches you probably have other needs too. You probably want to sum column values, have conditional counters, and you probably have enough throughput that updating a single column value will cause lock contention problems.
 
 Counter is different from other solutions like [Rails counter caches](https://api.rubyonrails.org/classes/ActiveRecord/CounterCache/ClassMethods.html) and [counter_culture](https://github.com/magnusvk/counter_culture):
 
 - Counters are objects. This makes it possible for them to have an API that allows you to define them, reset, and recalculate them. The definition of a counter is seperate from the value
 - Counters are persisted as a ActiveRecord models (_not_ a column of an existing model)
-- Incrementing counters can be safely performed in a background job via a change event/deferred reconciliation pattern
+- Counters can also perform aggregation (e.g. sum of column values instead of counting rows) or be calculated from other counters
 - Avoids lock-contention found in other solutions. By storing the value in another object we reduce the contention on the main e.g. User instance. This is only a small improvement though. By using the background change event pattern, we can batch perform the updates reducing the number of processes requiring a lock.
-- Counters can also perform aggregation (e.g. sum of column values instead of counting rows)
+- Incrementing counters can be safely performed in a background job via a change event/deferred reconciliation pattern (coming in a future iteration)
 
+## Usage
 
-## Main concepts
+You probably shouldn't use it right now unless you're the sort of person that checks if something is poisonous by licking it—or you're working at Podia where we are testing it in production.
 
-![](docs/data_model.png)
+## Installation
 
-`Counter::Definition` defines what the counter is, what model it's connected to, what association it counts, how the count is performed etc. You create a subclass of `Counter::Definition` and call a few class methods to configure it. The definition is available through `counter.definition` for any counter value…
+Add this line to your application's Gemfile:
 
-`Counter::Value` is the value of a counter. So, for example, a User might have many Posts, so a User would have a `counters` association containing a `Counter::Value` for the number of posts. Counters can be accessed via their name `user.posts_counter` or via the `find_counter` method on the association, e.g. `user.counters.find_counter PostCounter`
+```ruby
+gem 'counterwise', require: 'counter'
+```
 
-`Counter::Change` is a temporary record that records a change to a counter. Instead of updating a counter directly, which requires obtaining a lock on it to perform it safely and atomically, a new `Change` event is inserted into the table. On regular intervals, the `Counter::Value` is updated by incrementing the value by the sum of all outstanding changes. This requires much less frequent locks at the expense of eventual consistency.
+And then execute:
 
-For example, you might have many background jobs running concurrently, inserting hundreds/thousands of rows. The would not need to fight for a lock to update the counter and would only need to insert Counter::Change rows. The counter would then be updated, in a single operation, by summing all the persisted change values.
+```bash
+$ bundle
+```
 
-Basically updating a counter value requires this SQL:
+Install the model migrations:
 
-```sql
-UPDATE counter_values
--- Update the counter with the sum of pending changes
-SET value = value + changes.sum
-FROM (
-  -- Find the pending changes for the counter
-  SELECT sum(value) as sum
-  FROM counter_changes
-  WHERE counter_id = 100
-) as changes
-WHERE id = 100
+```bash
+$ rails counter:install:migrations
 ```
 
-Or even reconcile all pending counters in a single statement:
-
-```sql
-UPDATE counter_values
-SET value = value + changes.sum
-FROM (
-  SELECT sum(value)
-  FROM counter_changes
-  GROUP BY counter_id
-) as changes
-WHERE counters.id = counter_id
-```
+## Main concepts
+
+![](docs/data_model.png)
+
+`Counter::Definition` defines what the counter is, what model it's connected to, what association it counts, how the count is performed etc. You create a subclass of `Counter::Definition` and call a few class methods to configure it. The definition is available through `counter.definition` for any counter value…
+
+`Counter::Value` is the value of a counter. So, for example, a User might have many Posts, so a User would have a `counters` association containing a `Counter::Value` for the number of posts. Counters can be accessed via their name `user.posts_counter` or via the `find_counter` method on the association, e.g. `user.counters.find_counter PostCounter`
 
-## Defining a counter
+## Basic usage
+### Define a counter
 
 Counters are defined in a seperate class using a small DSL.
 
@@ -94,20 +92,21 @@ end
 
 First we define the counter class itself using `count` to specify the association we're counting, then "attach" it to the parent Store model.
 
-By default, the counter will be available as `<association>_counter`, e.g. `store.orders_counter`. To customise this, pass a `as` parameter:
+By default, the counter will be available as `<association>_counter`, e.g. `store.orders_counter`. To customise this, use the `as` method:
 
 ```ruby
 class OrderCounter < Counter::Definition
   include Counter::Counters
-  count :orders, as: :total_orders
+  count :orders
+  as :total_orders
 end
 
 store.total_orders
 ```
 
-The counter's value with be stored as a `Counter::Value` with the name prefixed by the model name. e.g. `store_total_orders`
+The counter's value will be stored as a `Counter::Value` with the name prefixed by the model name. e.g. `store-total_orders`
 
-## Accessing counter values
+### Access counter values
 
 Since counters are represented as objects, you need to call `value` on them to retrieve the count.
 
@@ -116,78 +115,76 @@ store.total_orders        #=> Counter::Value
 store.total_orders.value  #=> 200
 ```
 
-## Anonymous counters
+### Recalculate a counter
 
-Most counters are associated with a model instance and association. These counters are automatically incremented when the associated collection changes  but sometimes you just need a global counter that you can increment.
+Counters have a habit of drifting over time, particularly if ActiveRecords hooks aren't run (e.g. with a pure SQL data migration) so you need a method of re-counting the metric. Counters make this easy because they are objects in their own right.
 
-```ruby
-class GlobalOrderCounter < Counter::Definition
-  global :my_custom_counter_name
-end
+You could refresh a store's revenue stats with:
 
-GlobalOrderCounter.counter.value #=> 5
-GlobalOrderCounter.counter.increment! #=> 6
+```ruby
+store.order_revenue.recalc!
 ```
 
-## Defining a conditional counter
+this would use the definition of the counter, including any option to sum a column. In the case of conditional counters, they are expected to be attached to an association which matched the conditions so the recalculated count remains accurate.
+
+### Reset a counter
 
-Consider this model that we'd like to count but we don't want to count all products, just the premium ones with a price >= 1000
+You can also reset a counter by calling `reset`. Since counters are ActiveRecord objects, you could also reset them using
 
 ```ruby
-class Product < ApplicationRecord
-  include Counter::Counters
-  include Counter::Changable
+store.order_revenue.reset
+Counter::Value.update value: 0
+```
 
-  belongs_to :user
+### Verify a counter
 
-  scope :premium, -> { where("price >= 1000") }
+You might like to check if a counter is correct
 
-  def premium?
-    price >= 1000
-  end
-end
+```ruby
+store.product_revenue.correct? #=> false
 ```
 
-Here's the counter to do that:
+This will re-count / re-calculate the value and compare it to the current one. If you wish to also update the value when it's not correct, use `correct!`:
 
 ```ruby
-class PremiumProductCounter < Counter::Definition
-  # Define the association we're counting
-  count :premium_products
+store.product_revenue #=>200
+store.product_revenue.reset!
+store.product_revenue #=>0
+store.product_revenue.correct? #=> false
+store.product_revenue.correct! #=> false
+store.product_revenue #=>200
+```
 
-  on :create do
-    increment_if ->(product) { product.premium? }
-  end
+## Advanced usage
 
-  on :delete do
-    decrement_if ->(product) { product.premium? }
-  end
+### Sort or filter parent models by a counter value
 
-  on :update do
-    increment_if ->(product) {
-      product.has_changed? :price, from: ->(price) { price < 1000 }, to: ->(price) { price >= 1000 }
-    }
+Say a Customer has a `total revenue` counter, and you'd like to sort the list of customers with the highest spenders at the top. Since the counts aren't stored on the Customer model, you can't just call `Customer.order(total_orders: :desc)`. Instead, Counterwise provides a convenience method to pull the counter values into the resultset.
 
-    decrement_if ->(product) {
-      product.has_changed? :price, from: ->(price) { price >= 1000 }, to: ->(price) { price < 1000 }
-    }
-  end
-end
-```
+```ruby
+Customer.order_by_counter TotalRevenueCounter => :desc
 
-There is a lot going on here!
+# You can sort by multiple counters or mix counters and model attributes
+Customer.order_by_counter TotalRevenueCounter => :desc, name: :asc
+```
 
-First, we define the counter on a scoped association. This ensures that when we call `counter.recalc()` we will count using the association's SQL.
+Under the hood, `order_by_counter` will uses `with_counter_data_from` to pull the counter values into the resultset. This is useful if you want to use the counter values in a `where` clause or `select` statement.
 
-We also define several conditions that operate on the instance level, i.e. when we create/update/delete an instance. On `create` and `delete` we define a block to determine if the counter should be updated. In this case, we only increment the counter when a premium product is created, and only decrement it when a premium product is deleted.
+```ruby
+Customer.with_counter_data_from(TotalRevenueCounter).where("total_revenue_data > 1000")
+```
 
-`update` is more complex because there are two scenarios: either a product has been updated to make it premium or  downgrade from premium to some other state. On update, we increment the counter if the price has gone above 1000; and decrement is the price has now gone below 1000.
+These methods pull in the counter data itself but don't include the counter instances themselves. To do this, call
 
-We use the `has_changed?` helper to query the ActiveRecord `previous_changes` hash and check what has changed. You can specify either Procs or values for `from`/`to`. If you only specify a `from` value, `to` will default to "any value" (Counter::Any.instance)
+```ruby
+customers = Customer.with_counters TotalRevenueCounter
+# Since the counters are now preloaded, this avoids an N+1 query
+customers.each &:total_revenue
+```
 
-Conditional counters work best with a single attribute. If the counter is conditional on e.g. confirmed and subscribed, the update tracking logic becomes very complex especially if the values are both updated at the same time. The solution to this is hopefully Rails generated columns in 7.1 so you can store a "subscribed_and_confirmed" column and check the value of that instead. Rails dirty tracking will need to work with generated columns though; see [this PR](https://github.com/rails/rails/pull/48628).
+### Aggregate a value (e.g. sum of order revenue)
 
-## Aggregating a value (e.g. sum of order revenue)
+Sometimes you don'y want to count the number of orders but instead sum the value of those orders..
 
 Given an ActiveRecord model `Order`, we can count a storefront's revenue like so
 
@@ -216,122 +213,209 @@ and access it like
   store.order_revenue.value #=> 200
 ```
 
-## Recalculating a counter
-
-Counters have a habit of drifting over time, particularly if ActiveRecords hooks aren't run (e.g. with a pure SQL data migration) so you need a method of re-counting the metric. Counters make this easy because they are objects in their own right.
+### Hooks
 
-You could refresh a store's revenue stats with:
+You can add an `after_change` hook to your counter definition to perform some action when the counter is updated. For example, you might want to send a notification when a counter reaches a certain value.
 
 ```ruby
-store.order_revenue.recalc!
+class OrderRevenueCounter < Counter::Definition
+  count :orders, as: :order_revenue
+  sum :price
+
+  after_change :send_congratulations_email
+
+  # Only send an email when they cross $1000
+  def send_congratulations_email counter, old_value, new_value
+    return unless old_value < 1000 && new_value >= 1000
+    send_email "Congratulations! You've made #{to} dollars!"
+  end
+end
 ```
 
-this would use the definition of the counter, including any option to sum a column. In the case of conditional counters, they are expected to be attached to an association which matched the conditions so the recalculated count remains accurate.
+### Manual counters
 
-## Reset a counter
+Most counters are associated with a model instance and association—these counters are automatically incremented when the associated collection changes but sometimes you just need a manual counter that you can increment.
 
-You can also reset a counter by calling `reset`. Since counters are ActiveRecord objects, you could also reset them using
+Manual counters just need a name
 
 ```ruby
-store.order_revenue.reset
-Counter::Value.update value: 0
+class TotalOrderCounter < Counter::Definition
+  as "total_orders"
+end
+
+TotalOrderCounter.counter.value #=> 5
+TotalOrderCounter.counter.increment! #=> 6
 ```
 
-## Verify a counter
+### Calculating a value from other counters
 
-You might like to check if a counter is correct
+You may also need have a common need to calculate a value from other counters. For example, given counters for the number of purchases and the number of visits, you might want to calculate the conversion rate. You can do this with a `calculate_from` block.
 
 ```ruby
-store.product_revenue.correct? #=> false
+class ConversionRateCounter < Counter::Definition
+  count nil, as: "conversion_rate"
+
+  calculated_from VisitsCounter, OrdersCounter do |visits, orders|
+    (orders.value.to_f / visits.value) * 100
+  end
+end
 ```
 
-This will re-count / re-calculate the value and compare it to the current one. If you wish to also update the value when it's not correct, use `correct!`:
+This recalculates the conversion rate each time the visits or order counters are updated. If either dependant counter is not present, the calculation will not be run (i.e., visits and order will never be nil).
+
+### Defining a conditional counter
+
+Conditional counters allow you to count a subset of an association, like just the premium product with a price >= 1000.
 
 ```ruby
-store.product_revenue #=>200
-store.product_revenue.reset!
-store.product_revenue #=>0
-store.product_revenue.correct? #=> false
-store.product_revenue.correct! #=> false
-store.product_revenue #=>200
-```
+class Product < ApplicationRecord
+  include Counter::Counters
+  include Counter::Changable
 
-## Hooks
+  belongs_to :user
 
-You can add an `after_change` hook to your counter definition to perform some action when the counter is updated. For example, you might want to send a notification when a counter reaches a certain value.
+  scope :premium, -> { where("price >= 1000") }
+
+  def premium?
+    price >= 1000
+  end
+end
+```
+
+Conditional counters are more complex to define since we also need to specify when the counter should be incremented or decremented, for each create/delete/update.
 
 ```ruby
-class OrderRevenueCounter < Counter::Definition
-  count :orders, as: :order_revenue
-  sum :price
+class PremiumProductCounter < Counter::Definition
+  # Define the association we're counting
+  count :premium_products
 
-  after_change :send_congratulations_email
+  on :create do
+    increment_if ->(product) { product.premium? }
+  end
 
-  def send_congratulations_email counter, from, to
-    return unless from < 1000 && to >= 1000
-    send_email "Congratulations! You've made #{to} dollars!"
+  on :delete do
+    decrement_if ->(product) { product.premium? }
+  end
+
+  on :update do
+    increment_if ->(product) {
+      product.has_changed? :price, from: ->(price) { price < 1000 }, to: ->(price) { price >= 1000 }
+    }
+
+    decrement_if ->(product) {
+      product.has_changed? :price, from: ->(price) { price >= 1000 }, to: ->(price) { price < 1000 }
+    }
   end
 end
 ```
 
-## Testing the counters in production
+There is a lot going on here!
+
+First, we define the counter on a scoped association. This ensures that when we call `counter.recalc()` we will count using the association's SQL to get the correct results.
+
+We also define several conditions that operate on the instance level, i.e. when we create/update/delete an instance. On `create` and `delete` we define a block to determine if the counter should be updated. In this case, we only increment the counter when a premium product is created, and only decrement it when a premium product is deleted.
+
+`update` is more complex because there are two scenarios: either a product has been updated to make it premium or downgrade from premium to some other state. On update, we increment the counter if the price has gone above 1000; and decrement is the price has now gone below 1000.
+
+We use the `has_changed?` helper to query the ActiveRecord `previous_changes` hash and check what has changed. You can specify either Procs or values for `from`/`to`. If you only specify a `from` value, `to` will default to "any value" (Counter::Any.instance)
+
+Conditional counters work best with a single attribute. If the counter is conditional on e.g. confirmed and subscribed, the update tracking logic becomes very complex especially if the values are both updated at the same time. The solution to this is hopefully Rails generated columns in 7.1 so you can store a "subscribed_and_confirmed" column and check the value of that instead. Rails dirty tracking will need to work with generated columns though; see [this PR](https://github.com/rails/rails/pull/48628).
+
 
-It may be useful to verify the accuracy of the counters in production, especially if you are concerned about conditional counters etc causing counter drift over time.
+## Testing
 
-This form of script takes a sampling approach suitable for large collections. It will randomly select a record and verify that the counter value is correct; if it's not, it stops giving you a chance to investigate.
+### Using Rspec
+
+If you use RSpec, you can include `Counter::RSpecMatchers` on your helpers and test your counter definitions.
 
 ```ruby
-site_range = Site.minimum(:id)..Site.maximum(:id)
-
-1000.times do
-  random_id = rand(site_range)
-  site = Site.where("id >= ?", random_id).limit(1).first
-  next if site.nil?
-  if site.confirmed_subscribers_counter.correct?
-    puts "✅ site #{site.id} has correct counter value"
-  else
-    puts "❌ site #{site.id} has incorrect counter value. Expected #{site.confirmed_subscribers_counter.value} but got #{site.confirmed_subscribers_counter.count_by_sql}"
-    break
-  end
-  sleep 0.1
+require "counter/rspec/matchers"
+
+RSpec.configure do |config|
+  config.include Counter::RSpecMatchers, type: :counter
 end
 ```
 
----
+Now you can test your counter definitions like so:
 
-## TODO
+```ruby
+require "rails_helper"
+
+RSpec.describe PremiumProductCounter, type: :counter do
+  let(:store) { create(:store) }
+
+  describe "on :create" do
+    context "when the product is premium" do
+      it "increments the counter" do
+        expect { create(:product, :premium, store: store) }.to increment_counter_for(described_class, store)
+      end
+    end
+
+    context "when the product is not premium" do
+      it "doesn't increment the counter" do
+        expect { create(:product, store: store) }.not_to increment_counter_for(described_class, store)
+      end
+    end
+  end
 
-See the asociated project in Github but roughly I'm thinking:
-- Hierarchical counters. For example, a Site sends many Newsletters and each Newsletter results in many EmailMessages. Each EmailMessage can be marked as spam. How do you create counters for how many spam emails were sent at the Newsletter level and the Site level?
-- Time-based counters for analytics. Instead of a User having one OrderRevenue counter, they would have an OrderRevenue counter for each day. These counters would then be used to produce a chart of their product revenue over the month. Not sure if these are just special counters or something else entirely? Do they use the same ActiveRecord model?
-- Can we support floating point values? Sounds useful but don't have a use case for it right now. Would they need to be a different ActiveRecord table?
-- In a similar vein of supporting different value types, can we support HLL values? Instead of increment an integer we add the items hash to a HyperLogLog so we can count unique items. An example would be counting site visits in a time-based daily counter, then combine the daily counts and still obtain an estimated number of monthly _unique_ visits. Again, not sure if this is the same ActiveRecord model or something different.
-- Actually start running this in production for basic use cases
+  describe "on :delete" do
+    context "when the product is premium" do
+      it "decrements the counter" do
+        expect { create(:product, :premium, store: store) }.to decrement_counter_for(described_class, store)
+      end
+    end
+
+    context "when the product is not premium" do
+      it "doesn't decrement the counter" do
+        expect { create(:product, store: store) }.not_to decrement_counter_for(described_class, store)
+      end
+    end
+  end
+end
+```
 
-## Usage
-No one has used this in production yet.
+### In production
 
-You probably shouldn't right now unless you're the sort of person that checks if something is poisonous by licking it.
+> test in prod or live a lie — Charity Majors
 
-## Installation
-Add this line to your application's Gemfile:
+It's very useful to verify the accuracy of the counters in production, especially if you are concerned about conditional counters etc causing counter drift over time.
+
+A simple approach would be:
 
 ```ruby
-gem 'counter'
+Counter::Value.all.each &:correct!
 ```
 
-And then execute:
-```bash
-$ bundle
-```
+If you have a large number of counters though it's best to take a sampling approach to randomly select a counter and verify that the value is correct
 
-Install the model migrations:
-```bash
-$ rails counter:install:migrations
+```ruby
+Counter::Value.sample_and_verify samples: 1000, verbose: true, on_error: :correct
 ```
 
+Options:
+
+- scope — allows you to scope the counters to a particular model or set of models, e.g. `scope: -> { where("name LIKE 'store-%'") }`. By default, all counters are sampled
+- samples — the number of counters to sample. Default: 1000
+- verbose — print out the counter details and whether it was correct. Default: true
+- on_error — what to do when a counter is incorrect. `:correct` will correct the counter, `:raise` will raise an error, `:log` will log the error to Rails.logger. Default: :raise
+
+---
+
+## TODO
+
+See the asociated project in Github but roughly I'm thinking:
+- Implement the background job pattern for incrementing counters
+- Hierarchical counters. For example, a Site sends many Newsletters and each Newsletter results in many EmailMessages. Each EmailMessage can be marked as spam. How do you create counters for how many spam emails were sent at the Newsletter level and the Site level?
+- Time-based counters for analytics. Instead of a User having one OrderRevenue counter, they would have an OrderRevenue counter for each day. These counters would then be used to produce a chart of their product revenue over the month. Not sure if these are just special counters or something else entirely? Do they use the same ActiveRecord model?
+- In a similar vein of supporting different value types, can we support HLL values? Instead of increment an integer we add the items hash to a HyperLogLog so we can count unique items. An example would be counting site visits in a time-based daily counter, then combine the daily counts and still obtain an estimated number of monthly _unique_ visits. Again, not sure if this is the same ActiveRecord model or something different.
+- Actually start running this in production for basic use cases
+
 ## Contributing
-Contribution directions go here.
+
+Bug reports and pull requests are welcome, especially around naming, internal APIs, bug fixes, and additional features. Please open an issue first if you're thinking of adding a new feature so we can discuss it.
+
+I'm unlikely to entertain suport for older Ruby or Rails versions, or databases other than Postgres.
 
 ## License
+
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/app/models/concerns/counter/calculated.rb

@@ -0,0 +1,25 @@
+module Counter::Calculated
+  extend ActiveSupport::Concern
+
+  included do
+    def calculate!
+      new_value = calculate
+      update! value: new_value unless new_value.nil?
+    end
+
+    def calculate
+      counters = counters_for_calculation
+      # If any of the counters are missing, we can't calculate
+      return if counters.any?(&:nil?)
+
+      definition.calculated_from.call(*counters)
+    end
+
+    def counters_for_calculation
+      # Fetch the dependant counters
+      definition.dependent_counters.map do |counter|
+        parent.counters.find_counter(counter)
+      end
+    end
+  end
+end

data/app/models/concerns/counter/recalculatable.rb

@@ -1,11 +1,16 @@
 module Counter::Recalculatable
   extend ActiveSupport::Concern
 
-  ####################################################### Support for regenerating the counters
   def recalc!
-    with_lock do
-      new_value = definition.sum? ? sum_by_sql : count_by_sql
-      update! value: new_value
+    if definition.calculated?
+      calculate!
+    elsif definition.manual?
+      raise Counter::Error.new("Can't recalculate a manual counter")
+    else
+      with_lock do
+        new_value = definition.sum? ? sum_by_sql : count_by_sql
+        update! value: new_value
+      end
     end
   end
 

data/app/models/concerns/counter/verifyable.rb

@@ -2,13 +2,66 @@ module Counter::Verifyable
   extend ActiveSupport::Concern
 
   def correct?
-    count_by_sql == value
+    # We can't verify these values
+    return true if definition.global?
+
+    old_value, new_value = verify
+    old_value == new_value
   end
 
   def correct!
-    requires_recalculation = !correct?
-    recalc! if requires_recalculation
+    # We can't verify these values
+    return true if definition.global?
+
+    old_value, new_value = verify
+
+    requires_recalculation = old_value != new_value
+    update! value: new_value if requires_recalculation
 
     !requires_recalculation
   end
+
+  def verify
+    if definition.calculated?
+      [calculate, value]
+    else
+      [count_by_sql, value]
+    end
+  end
+
+  class_methods do
+    # on_error: raise, log, correct
+    # Returns the number of incorrect counters
+    def sample_and_verify scope: -> { all }, samples: 1000, verbose: true, on_error: :raise
+      incorrect_counters = 0
+
+      counter_range = Counter::Value.minimum(:id)..Counter::Value.maximum(:id)
+
+      samples.times do
+        random_id = rand(counter_range)
+        counter = Counter::Value.merge(scope).where("id >= ?", random_id).limit(1).first
+        next if counter.nil?
+
+        if counter.definition.global? || counter.definition.calculated?
+          puts "➡️ Skipping counter #{counter.name} (#{counter.id})" if verbose
+          next
+        end
+
+        if counter.correct?
+          puts "✅ Counter #{counter.id} is correct" if verbose
+        else
+          incorrect_counters += 1
+          message = "❌ counter #{counter.name} (#{counter.id}) for #{counter.parent_type}##{counter.parent_id} has incorrect counter value. Expected #{counter.value} but got #{counter.count_by_sql}"
+
+          case on_error
+          when :raise then raise Counter::Error.new(message)
+          when :log then Rails.logger.error message
+          when :correct then counter.correct!
+          end
+        end
+        sleep 0.1
+      end
+      incorrect_counters
+    end
+  end
 end

data/app/models/counter/value.rb

@@ -43,5 +43,6 @@ class Counter::Value < ApplicationRecord
   include Counter::Verifyable
   include Counter::Summable
   include Counter::Conditional
+  include Counter::Calculated
   # include Counter::Hierarchical
 end

data/db/migrate/20210705154113_create_counter_values.rb

@@ -2,7 +2,7 @@ class CreateCounterValues < ActiveRecord::Migration[6.1]
   def change
     create_table :counter_values do |t|
       t.string :name, index: true
-      t.integer :value, default: 0
+      t.decimal :value, default: 0.0, null: false
       t.references :parent, polymorphic: true
 
       t.timestamps

data/lib/counter/definition.rb

@@ -5,11 +5,7 @@
 #   # This specifies the association we're counting
 #   count :products
 #   sum :price   # optional
-#   filters: {   # optional
-#     create: ->(product) { product.premium? }
-#     update: ->(product) { product.has_changed? :premium, to: :true }
-#     delete: ->(product) { product.premium? }
-#   }
+#   as "my_counter"
 # end
 class Counter::Definition
   include Singleton
@@ -34,21 +30,38 @@ class Counter::Definition
   attr_writer :global_counters
   # An array of Proc to run when the counter changes
   attr_writer :counter_hooks
-  # An array of all global counters
-  attr_writer :global_counters
+  # The counters this calculated counter depends on
+  attr_writer :dependent_counters
+  # The block to call to calculate the counter
+  attr_accessor :calculated_from
 
+  # Is this a counter which sums a column?
   def sum?
     column_to_count.present?
   end
 
+  # Is this a global counter? i.e., not attached to a model
   def global?
-    model.nil? && association_name.nil?
+    model.nil?
   end
 
+  # Is this counter conditional?
   def conditional?
     @conditional
   end
 
+  # Is this counter calculated from other counters?
+  def calculated?
+    !@calculated_from.nil?
+  end
+
+  # Is this a manual counter?
+  # Manual counters are not automatically updated from an association
+  # or calculated from other counters
+  def manual?
+    association_name.nil? && !calculated?
+  end
+
   # for global counter instances to find their definition
   def self.find_definition name
     Counter::Definition.instance.global_counters.find { |c| c.name == name }
@@ -64,7 +77,8 @@ class Counter::Definition
   # What we record in Counter::Value#name
   def record_name
     return name if global?
-    "#{model.name.underscore}-#{association_name}"
+    return "#{model.name.underscore}-#{association_name}" if association_name.present?
+    return "#{model.name.underscore}-#{name}"
   end
 
   def conditions
@@ -82,9 +96,9 @@ class Counter::Definition
     @counter_hooks
   end
 
-  def global_counters
-    @global_counters ||= []
-    @global_counters
+  def dependent_counters
+    @dependent_counters ||= []
+    @dependent_counters
   end
 
   # Set the association we're counting
@@ -95,12 +109,36 @@ class Counter::Definition
     instance.method_name = as.to_s
   end
 
-  def self.global name = nil
-    name ||= name.underscore
-    instance.name = name.to_s
+  def self.global
     Counter::Definition.instance.global_counters << instance
   end
 
+  def self.calculated_from *dependent_counters, &block
+    instance.dependent_counters = dependent_counters
+    instance.calculated_from = block
+
+    dependent_counters.each do |dependent_counter|
+      # Install after_change hooks on the dependent counters
+      dependent_counter.after_change :update_calculated_counters
+      dependent_counter.define_method :update_calculated_counters do |counter, _old_value, _new_value|
+        # Fetch all the counters which depend on this one
+        calculated_counters = counter.parent.class.counter_configs.select { |c|
+          c.dependent_counters.include?(counter.definition.class)
+        }
+
+        calculated_counters = calculated_counters.map { |c| counter.parent.counters.find_or_create_counter!(c) }
+        # calculate the new values
+        calculated_counters.each(&:calculate!)
+      end
+    end
+  end
+
+  # Set the name of the counter
+  def self.as name
+    instance.name = name.to_s
+    instance.method_name = name.to_s
+  end
+
   # Get the name of the association we're counting
   def self.association_name
     instance.association_name

data/lib/counter/integration/countable.rb

@@ -26,6 +26,8 @@ module Counter::Countable
     def each_counter_to_update
       # For each definition, find or create the counter on the parent
       self.class.counted_by.each do |counter_definition|
+        next unless counter_definition.inverse_association
+
         parent_association = association(counter_definition.inverse_association)
         parent_association.load_target unless parent_association.loaded?
         parent_model = parent_association.target

data/lib/counter/integration/counters.rb

@@ -55,34 +55,66 @@ module Counter::Counters
       counter_definitions = Array.wrap(counter_definitions)
       counter_definitions.each do |definition_class|
         definition = definition_class.instance
-        association_name = definition.association_name
-
-        # Find the association on this model
-        association_reflection = reflect_on_association(association_name)
-        # Find the association classes
-        association_class = association_reflection.class_name.constantize
-        inverse_association = association_reflection.inverse_of
-
-        raise Counter::Error.new("#{association_name} must have an inverse_of specified to be used in #{definition_class.name}") if inverse_association.nil?
-
-        # Add the after_commit hook to the association's class
-        association_class.include Counter::Countable
-        # association_class.include Counter::Changed
-
-        # Update the definition with the association class and inverse association
-        # gathered from the reflection
         definition.model = self
-        definition.inverse_association = inverse_association.name
-        definition.countable_model = association_class
+
+        scope :with_counter_data_from, ->(*counter_classes) {
+          subqueries = ["#{table_name}.*"]
+          counter_classes.each do |counter_class|
+            sql = Counter::Value.select("value")
+              .where("parent_id = #{table_name}.id AND parent_type = '#{name}' AND name = '#{counter_class.instance.record_name}'").to_sql
+            subqueries << "(#{sql}) AS #{counter_class.instance.name}_data"
+          end
+          select(subqueries)
+        }
+
+        # Expects a hash of counter classes and directions, like so:
+        # order_by_counter ProductCounter => :desc, PremiumProductCounter => :asc
+        scope :order_by_counter, ->(order_hash) {
+          counter_classes = order_hash.keys.select { |counter_class|
+            counter_class.is_a?(Class) &&
+              counter_class.ancestors.include?(Counter::Definition)
+          }
+          order_params = {}
+          order_hash.map do |counter_class, direction|
+            if counter_class.is_a?(String) || counter_class.is_a?(Symbol)
+              order_params[counter_class] = direction
+            elsif counter_class.ancestors.include?(Counter::Definition)
+              order_params["#{counter_class.instance.name}_data"] = direction
+            end
+          end
+          with_counter_data_from(*counter_classes).order(order_params)
+        }
+
+        scope :with_counters, -> { includes(:counters) }
 
         define_method definition.method_name do
           counters.find_or_create_counter!(definition)
         end
 
-        # Provide the Countable class with details about where it's counted
+        @counter_configs << definition unless @counter_configs.include?(definition)
 
-        @counter_configs << definition
-        association_class.add_counted_by definition
+        association_name = definition.association_name
+        if association_name.present?
+          # Find the association on this model
+          association_reflection = reflect_on_association(association_name)
+          raise Counter::Error.new("#{association_name} does not exist #{self.name}") if association_reflection.nil?
+
+          # Find the association classes
+          association_class = association_reflection.class_name.constantize
+          inverse_association = association_reflection.inverse_of
+          raise Counter::Error.new("#{association_name} must have an inverse_of specified to be used in #{definition_class.name}") if inverse_association.nil?
+
+          # Add the after_commit hook to the association's class
+          association_class.include Counter::Countable
+
+          # Update the definition with the association class and inverse association
+          # gathered from the reflection
+          definition.inverse_association = inverse_association.name
+          definition.countable_model = association_class
+
+          # Provide the Countable class with details about where it's counted
+          association_class.add_counted_by definition
+        end
       end
     end
 

data/lib/counter/rspec/matchers.rb

@@ -0,0 +1,29 @@
+module Counter
+  module RSpecMatchers
+    def increment_counter_for(...)
+      IncrementCounterFor.new(...)
+    end
+
+    def decrement_counter_for(...)
+      DecrementCounterFor.new(...)
+    end
+
+    class Base < RSpec::Matchers::BuiltIn::Change
+      def initialize(counter_class, parent)
+        super { parent.counters.find_or_create_counter!(counter_class).value }
+      end
+    end
+
+    class IncrementCounterFor < Base
+      def matches?(...)
+        by(1).matches?(...)
+      end
+    end
+
+    class DecrementCounterFor < Base
+      def matches?(...)
+        by(-1).matches?(...)
+      end
+    end
+  end
+end

data/lib/counter/version.rb

@@ -1,3 +1,3 @@
 module Counter
-  VERSION = "0.1.1"
+  VERSION = "0.1.2"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: counterwise
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - Jamie Lawrence
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-08-03 00:00:00.000000000 Z
+date: 2023-08-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -52,6 +52,7 @@ files:
 - app/controllers/counters_controller.rb
 - app/jobs/counter/reconciliation_job.rb
 - app/models/concerns/counter/Xhierarchical.rb
+- app/models/concerns/counter/calculated.rb
 - app/models/concerns/counter/changable.rb
 - app/models/concerns/counter/conditional.rb
 - app/models/concerns/counter/definable.rb
@@ -62,11 +63,9 @@ files:
 - app/models/concerns/counter/sidekiq_reconciliation.rb
 - app/models/concerns/counter/summable.rb
 - app/models/concerns/counter/verifyable.rb
-- app/models/counter/change.rb
 - app/models/counter/value.rb
 - config/routes.rb
 - db/migrate/20210705154113_create_counter_values.rb
-- db/migrate/20210709211056_create_counter_changes.rb
 - db/migrate/20210731224504_add_unique_index_to_counter_values.rb
 - lib/counter.rb
 - lib/counter/any.rb
@@ -77,6 +76,7 @@ files:
 - lib/counter/integration/countable.rb
 - lib/counter/integration/counters.rb
 - lib/counter/railtie.rb
+- lib/counter/rspec/matchers.rb
 - lib/counter/version.rb
 - lib/tasks/counter_tasks.rake
 homepage: https://github.com/podia/counter

data/app/models/counter/change.rb

@@ -1,23 +0,0 @@
-# == Schema Information
-#
-# Table name: counter_changes
-#
-#  id               :integer          not null, primary key
-#  counter_value_id :integer          indexed
-#  amount           :integer
-#  processed_at     :datetime         indexed
-#  created_at       :datetime         not null
-#  updated_at       :datetime         not null
-#
-class Counter::Change < ApplicationRecord
-  def self.table_name_prefix
-    "counter_"
-  end
-
-  belongs_to :counter, class_name: "Counter::Value"
-  validates_numericality_of :amount
-
-  scope :pending, -> { where(reconciled_at: nil) }
-  scope :reconciled, -> { where.not(reconciled_at: nil) }
-  scope :purgable, -> { reconciled.where(processed_at: 7.days.ago..) }
-end

data/db/migrate/20210709211056_create_counter_changes.rb

@@ -1,11 +0,0 @@
-class CreateCounterChanges < ActiveRecord::Migration[6.1]
-  def change
-    create_table :counter_changes do |t|
-      t.references :counter_value, foreign_key: true
-      t.integer :amount
-      t.timestamp :processed_at, index: true
-
-      t.timestamps
-    end
-  end
-end