counterwise 0.1.0 → 0.1.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +252 -144
- data/app/models/concerns/counter/calculated.rb +25 -0
- data/app/models/concerns/counter/recalculatable.rb +9 -4
- data/app/models/concerns/counter/verifyable.rb +56 -3
- data/app/models/counter/value.rb +1 -0
- data/db/migrate/20210705154113_create_counter_values.rb +1 -1
- data/lib/counter/definition.rb +53 -15
- data/lib/counter/integration/countable.rb +5 -2
- data/lib/counter/integration/counters.rb +53 -21
- data/lib/counter/rspec/matchers.rb +29 -0
- data/lib/counter/version.rb +1 -1
- metadata +4 -4
- data/app/models/counter/change.rb +0 -23
- data/db/migrate/20210709211056_create_counter_changes.rb +0 -11
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: ac4f723a8cdebb397ea3c1035ee728599ce308ebe36fc5aaa9fb6afeff09474e
|
4
|
+
data.tar.gz: 5efc1f7fdbde64eab27962e0b37355ae16f8293c8173d101e85da97b81679dc7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 48b91a565a82de0da5dbdf3e6637721ae05ad4b5bea9e0b8c31366932e6b30f6460e526911a458ceda815910d1031046b2413eb0668207cd001c18afdb030002
|
7
|
+
data.tar.gz: 5d6fd86b0ab23cd378ed374c6bf91aab877626038f7c35acecd2b123777fb646340a4070e61e02cbbf98ce9ab521155c25d2e0b320de0879a927def69cf41b30
|
data/README.md
CHANGED
@@ -5,74 +5,73 @@
|
|
5
5
|
Counting and aggregation library for Rails.
|
6
6
|
|
7
7
|
- [Counter](#counter)
|
8
|
-
- [Main concepts](#main-concepts)
|
9
|
-
- [Defining a counter](#defining-a-counter)
|
10
|
-
- [Accessing counter values](#accessing-counter-values)
|
11
|
-
- [Anonymous counters](#anonymous-counters)
|
12
|
-
- [Defining a conditional counter](#defining-a-conditional-counter)
|
13
|
-
- [Aggregating a value (e.g. sum of order revenue)](#aggregating-a-value-eg-sum-of-order-revenue)
|
14
|
-
- [Recalculating a counter](#recalculating-a-counter)
|
15
|
-
- [Reset a counter](#reset-a-counter)
|
16
|
-
- [Verify a counter](#verify-a-counter)
|
17
|
-
- [Hooks](#hooks)
|
18
|
-
- [TODO](#todo)
|
19
8
|
- [Usage](#usage)
|
20
9
|
- [Installation](#installation)
|
10
|
+
- [Main concepts](#main-concepts)
|
11
|
+
- [Basic usage](#basic-usage)
|
12
|
+
- [Define a counter](#define-a-counter)
|
13
|
+
- [Access counter values](#access-counter-values)
|
14
|
+
- [Recalculate a counter](#recalculate-a-counter)
|
15
|
+
- [Reset a counter](#reset-a-counter)
|
16
|
+
- [Verify a counter](#verify-a-counter)
|
17
|
+
- [Advanced usage](#advanced-usage)
|
18
|
+
- [Sort or filter parent models by a counter value](#sort-or-filter-parent-models-by-a-counter-value)
|
19
|
+
- [Aggregate a value (e.g. sum of order revenue)](#aggregate-a-value-eg-sum-of-order-revenue)
|
20
|
+
- [Hooks](#hooks)
|
21
|
+
- [Manual counters](#manual-counters)
|
22
|
+
- [Calculating a value from other counters](#calculating-a-value-from-other-counters)
|
23
|
+
- [Defining a conditional counter](#defining-a-conditional-counter)
|
24
|
+
- [Testing](#testing)
|
25
|
+
- [Using Rspec](#using-rspec)
|
26
|
+
- [In production](#in-production)
|
27
|
+
- [TODO](#todo)
|
21
28
|
- [Contributing](#contributing)
|
22
29
|
- [License](#license)
|
23
30
|
|
24
|
-
By the time you need Rails counter_caches you probably have other needs too. You probably want to sum column values and you probably have enough throughput that updating a single column value will cause lock contention problems
|
31
|
+
By the time you need Rails counter_caches you probably have other needs too. You probably want to sum column values, have conditional counters, and you probably have enough throughput that updating a single column value will cause lock contention problems.
|
25
32
|
|
26
33
|
Counter is different from other solutions like [Rails counter caches](https://api.rubyonrails.org/classes/ActiveRecord/CounterCache/ClassMethods.html) and [counter_culture](https://github.com/magnusvk/counter_culture):
|
27
34
|
|
28
35
|
- Counters are objects. This makes it possible for them to have an API that allows you to define them, reset, and recalculate them. The definition of a counter is seperate from the value
|
29
36
|
- Counters are persisted as a ActiveRecord models (_not_ a column of an existing model)
|
30
|
-
-
|
37
|
+
- Counters can also perform aggregation (e.g. sum of column values instead of counting rows) or be calculated from other counters
|
31
38
|
- Avoids lock-contention found in other solutions. By storing the value in another object we reduce the contention on the main e.g. User instance. This is only a small improvement though. By using the background change event pattern, we can batch perform the updates reducing the number of processes requiring a lock.
|
32
|
-
-
|
39
|
+
- Incrementing counters can be safely performed in a background job via a change event/deferred reconciliation pattern (coming in a future iteration)
|
33
40
|
|
41
|
+
## Usage
|
34
42
|
|
35
|
-
|
43
|
+
You probably shouldn't use it right now unless you're the sort of person that checks if something is poisonous by licking it—or you're working at Podia where we are testing it in production.
|
36
44
|
|
37
|
-
|
45
|
+
## Installation
|
38
46
|
|
39
|
-
|
47
|
+
Add this line to your application's Gemfile:
|
40
48
|
|
41
|
-
|
49
|
+
```ruby
|
50
|
+
gem 'counterwise', require: 'counter'
|
51
|
+
```
|
42
52
|
|
43
|
-
|
53
|
+
And then execute:
|
44
54
|
|
45
|
-
|
55
|
+
```bash
|
56
|
+
$ bundle
|
57
|
+
```
|
46
58
|
|
47
|
-
|
59
|
+
Install the model migrations:
|
48
60
|
|
49
|
-
```
|
50
|
-
|
51
|
-
-- Update the counter with the sum of pending changes
|
52
|
-
SET value = value + changes.sum
|
53
|
-
FROM (
|
54
|
-
-- Find the pending changes for the counter
|
55
|
-
SELECT sum(value) as sum
|
56
|
-
FROM counter_changes
|
57
|
-
WHERE counter_id = 100
|
58
|
-
) as changes
|
59
|
-
WHERE id = 100
|
61
|
+
```bash
|
62
|
+
$ rails counter:install:migrations
|
60
63
|
```
|
61
64
|
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
67
|
-
|
68
|
-
|
69
|
-
FROM counter_changes
|
70
|
-
GROUP BY counter_id
|
71
|
-
) as changes
|
72
|
-
WHERE counters.id = counter_id
|
73
|
-
```
|
65
|
+
## Main concepts
|
66
|
+
|
67
|
+
![](docs/data_model.png)
|
68
|
+
|
69
|
+
`Counter::Definition` defines what the counter is, what model it's connected to, what association it counts, how the count is performed etc. You create a subclass of `Counter::Definition` and call a few class methods to configure it. The definition is available through `counter.definition` for any counter value…
|
70
|
+
|
71
|
+
`Counter::Value` is the value of a counter. So, for example, a User might have many Posts, so a User would have a `counters` association containing a `Counter::Value` for the number of posts. Counters can be accessed via their name `user.posts_counter` or via the `find_counter` method on the association, e.g. `user.counters.find_counter PostCounter`
|
74
72
|
|
75
|
-
##
|
73
|
+
## Basic usage
|
74
|
+
### Define a counter
|
76
75
|
|
77
76
|
Counters are defined in a seperate class using a small DSL.
|
78
77
|
|
@@ -93,20 +92,21 @@ end
|
|
93
92
|
|
94
93
|
First we define the counter class itself using `count` to specify the association we're counting, then "attach" it to the parent Store model.
|
95
94
|
|
96
|
-
By default, the counter will be available as `<association>_counter`, e.g. `store.orders_counter`. To customise this,
|
95
|
+
By default, the counter will be available as `<association>_counter`, e.g. `store.orders_counter`. To customise this, use the `as` method:
|
97
96
|
|
98
97
|
```ruby
|
99
98
|
class OrderCounter < Counter::Definition
|
100
99
|
include Counter::Counters
|
101
|
-
count :orders
|
100
|
+
count :orders
|
101
|
+
as :total_orders
|
102
102
|
end
|
103
103
|
|
104
104
|
store.total_orders
|
105
105
|
```
|
106
106
|
|
107
|
-
The counter's value
|
107
|
+
The counter's value will be stored as a `Counter::Value` with the name prefixed by the model name. e.g. `store-total_orders`
|
108
108
|
|
109
|
-
|
109
|
+
### Access counter values
|
110
110
|
|
111
111
|
Since counters are represented as objects, you need to call `value` on them to retrieve the count.
|
112
112
|
|
@@ -115,22 +115,157 @@ store.total_orders #=> Counter::Value
|
|
115
115
|
store.total_orders.value #=> 200
|
116
116
|
```
|
117
117
|
|
118
|
-
|
118
|
+
### Recalculate a counter
|
119
|
+
|
120
|
+
Counters have a habit of drifting over time, particularly if ActiveRecords hooks aren't run (e.g. with a pure SQL data migration) so you need a method of re-counting the metric. Counters make this easy because they are objects in their own right.
|
121
|
+
|
122
|
+
You could refresh a store's revenue stats with:
|
123
|
+
|
124
|
+
```ruby
|
125
|
+
store.order_revenue.recalc!
|
126
|
+
```
|
127
|
+
|
128
|
+
this would use the definition of the counter, including any option to sum a column. In the case of conditional counters, they are expected to be attached to an association which matched the conditions so the recalculated count remains accurate.
|
129
|
+
|
130
|
+
### Reset a counter
|
119
131
|
|
120
|
-
|
132
|
+
You can also reset a counter by calling `reset`. Since counters are ActiveRecord objects, you could also reset them using
|
121
133
|
|
122
134
|
```ruby
|
123
|
-
|
124
|
-
|
135
|
+
store.order_revenue.reset
|
136
|
+
Counter::Value.update value: 0
|
137
|
+
```
|
138
|
+
|
139
|
+
### Verify a counter
|
140
|
+
|
141
|
+
You might like to check if a counter is correct
|
142
|
+
|
143
|
+
```ruby
|
144
|
+
store.product_revenue.correct? #=> false
|
145
|
+
```
|
146
|
+
|
147
|
+
This will re-count / re-calculate the value and compare it to the current one. If you wish to also update the value when it's not correct, use `correct!`:
|
148
|
+
|
149
|
+
```ruby
|
150
|
+
store.product_revenue #=>200
|
151
|
+
store.product_revenue.reset!
|
152
|
+
store.product_revenue #=>0
|
153
|
+
store.product_revenue.correct? #=> false
|
154
|
+
store.product_revenue.correct! #=> false
|
155
|
+
store.product_revenue #=>200
|
156
|
+
```
|
157
|
+
|
158
|
+
## Advanced usage
|
159
|
+
|
160
|
+
### Sort or filter parent models by a counter value
|
161
|
+
|
162
|
+
Say a Customer has a `total revenue` counter, and you'd like to sort the list of customers with the highest spenders at the top. Since the counts aren't stored on the Customer model, you can't just call `Customer.order(total_orders: :desc)`. Instead, Counterwise provides a convenience method to pull the counter values into the resultset.
|
163
|
+
|
164
|
+
```ruby
|
165
|
+
Customer.order_by_counter TotalRevenueCounter => :desc
|
166
|
+
|
167
|
+
# You can sort by multiple counters or mix counters and model attributes
|
168
|
+
Customer.order_by_counter TotalRevenueCounter => :desc, name: :asc
|
169
|
+
```
|
170
|
+
|
171
|
+
Under the hood, `order_by_counter` will uses `with_counter_data_from` to pull the counter values into the resultset. This is useful if you want to use the counter values in a `where` clause or `select` statement.
|
172
|
+
|
173
|
+
```ruby
|
174
|
+
Customer.with_counter_data_from(TotalRevenueCounter).where("total_revenue_data > 1000")
|
175
|
+
```
|
176
|
+
|
177
|
+
These methods pull in the counter data itself but don't include the counter instances themselves. To do this, call
|
178
|
+
|
179
|
+
```ruby
|
180
|
+
customers = Customer.with_counters TotalRevenueCounter
|
181
|
+
# Since the counters are now preloaded, this avoids an N+1 query
|
182
|
+
customers.each &:total_revenue
|
183
|
+
```
|
184
|
+
|
185
|
+
### Aggregate a value (e.g. sum of order revenue)
|
186
|
+
|
187
|
+
Sometimes you don'y want to count the number of orders but instead sum the value of those orders..
|
188
|
+
|
189
|
+
Given an ActiveRecord model `Order`, we can count a storefront's revenue like so
|
190
|
+
|
191
|
+
```ruby
|
192
|
+
class Store < ApplicationRecord
|
193
|
+
include Counter::Counters
|
194
|
+
|
195
|
+
counter OrderRevenue
|
196
|
+
end
|
197
|
+
```
|
198
|
+
|
199
|
+
Define the counter like so
|
200
|
+
|
201
|
+
```ruby
|
202
|
+
class OrderRevenue < Counter::Definition
|
203
|
+
count :orders
|
204
|
+
sum :total_price
|
125
205
|
end
|
206
|
+
```
|
126
207
|
|
127
|
-
|
128
|
-
|
208
|
+
and access it like
|
209
|
+
|
210
|
+
```ruby
|
211
|
+
store.orders.create total_price: 100
|
212
|
+
store.orders.create total_price: 100
|
213
|
+
store.order_revenue.value #=> 200
|
129
214
|
```
|
130
215
|
|
131
|
-
|
216
|
+
### Hooks
|
132
217
|
|
133
|
-
|
218
|
+
You can add an `after_change` hook to your counter definition to perform some action when the counter is updated. For example, you might want to send a notification when a counter reaches a certain value.
|
219
|
+
|
220
|
+
```ruby
|
221
|
+
class OrderRevenueCounter < Counter::Definition
|
222
|
+
count :orders, as: :order_revenue
|
223
|
+
sum :price
|
224
|
+
|
225
|
+
after_change :send_congratulations_email
|
226
|
+
|
227
|
+
# Only send an email when they cross $1000
|
228
|
+
def send_congratulations_email counter, old_value, new_value
|
229
|
+
return unless old_value < 1000 && new_value >= 1000
|
230
|
+
send_email "Congratulations! You've made #{to} dollars!"
|
231
|
+
end
|
232
|
+
end
|
233
|
+
```
|
234
|
+
|
235
|
+
### Manual counters
|
236
|
+
|
237
|
+
Most counters are associated with a model instance and association—these counters are automatically incremented when the associated collection changes but sometimes you just need a manual counter that you can increment.
|
238
|
+
|
239
|
+
Manual counters just need a name
|
240
|
+
|
241
|
+
```ruby
|
242
|
+
class TotalOrderCounter < Counter::Definition
|
243
|
+
as "total_orders"
|
244
|
+
end
|
245
|
+
|
246
|
+
TotalOrderCounter.counter.value #=> 5
|
247
|
+
TotalOrderCounter.counter.increment! #=> 6
|
248
|
+
```
|
249
|
+
|
250
|
+
### Calculating a value from other counters
|
251
|
+
|
252
|
+
You may also need have a common need to calculate a value from other counters. For example, given counters for the number of purchases and the number of visits, you might want to calculate the conversion rate. You can do this with a `calculate_from` block.
|
253
|
+
|
254
|
+
```ruby
|
255
|
+
class ConversionRateCounter < Counter::Definition
|
256
|
+
count nil, as: "conversion_rate"
|
257
|
+
|
258
|
+
calculated_from VisitsCounter, OrdersCounter do |visits, orders|
|
259
|
+
(orders.value.to_f / visits.value) * 100
|
260
|
+
end
|
261
|
+
end
|
262
|
+
```
|
263
|
+
|
264
|
+
This recalculates the conversion rate each time the visits or order counters are updated. If either dependant counter is not present, the calculation will not be run (i.e., visits and order will never be nil).
|
265
|
+
|
266
|
+
### Defining a conditional counter
|
267
|
+
|
268
|
+
Conditional counters allow you to count a subset of an association, like just the premium product with a price >= 1000.
|
134
269
|
|
135
270
|
```ruby
|
136
271
|
class Product < ApplicationRecord
|
@@ -147,7 +282,7 @@ class Product < ApplicationRecord
|
|
147
282
|
end
|
148
283
|
```
|
149
284
|
|
150
|
-
|
285
|
+
Conditional counters are more complex to define since we also need to specify when the counter should be incremented or decremented, for each create/delete/update.
|
151
286
|
|
152
287
|
```ruby
|
153
288
|
class PremiumProductCounter < Counter::Definition
|
@@ -176,138 +311,111 @@ end
|
|
176
311
|
|
177
312
|
There is a lot going on here!
|
178
313
|
|
179
|
-
First, we define the counter on a scoped association. This ensures that when we call `counter.recalc()` we will count using the association's SQL.
|
314
|
+
First, we define the counter on a scoped association. This ensures that when we call `counter.recalc()` we will count using the association's SQL to get the correct results.
|
180
315
|
|
181
316
|
We also define several conditions that operate on the instance level, i.e. when we create/update/delete an instance. On `create` and `delete` we define a block to determine if the counter should be updated. In this case, we only increment the counter when a premium product is created, and only decrement it when a premium product is deleted.
|
182
317
|
|
183
|
-
`update` is more complex because there are two scenarios: either a product has been updated to make it premium or
|
318
|
+
`update` is more complex because there are two scenarios: either a product has been updated to make it premium or downgrade from premium to some other state. On update, we increment the counter if the price has gone above 1000; and decrement is the price has now gone below 1000.
|
184
319
|
|
185
320
|
We use the `has_changed?` helper to query the ActiveRecord `previous_changes` hash and check what has changed. You can specify either Procs or values for `from`/`to`. If you only specify a `from` value, `to` will default to "any value" (Counter::Any.instance)
|
186
321
|
|
187
322
|
Conditional counters work best with a single attribute. If the counter is conditional on e.g. confirmed and subscribed, the update tracking logic becomes very complex especially if the values are both updated at the same time. The solution to this is hopefully Rails generated columns in 7.1 so you can store a "subscribed_and_confirmed" column and check the value of that instead. Rails dirty tracking will need to work with generated columns though; see [this PR](https://github.com/rails/rails/pull/48628).
|
188
323
|
|
189
|
-
## Aggregating a value (e.g. sum of order revenue)
|
190
324
|
|
191
|
-
|
325
|
+
## Testing
|
192
326
|
|
193
|
-
|
194
|
-
class Store < ApplicationRecord
|
195
|
-
include Counter::Counters
|
327
|
+
### Using Rspec
|
196
328
|
|
197
|
-
|
198
|
-
end
|
199
|
-
```
|
200
|
-
|
201
|
-
Define the counter like so
|
329
|
+
If you use RSpec, you can include `Counter::RSpecMatchers` on your helpers and test your counter definitions.
|
202
330
|
|
203
331
|
```ruby
|
204
|
-
|
205
|
-
|
206
|
-
|
332
|
+
require "counter/rspec/matchers"
|
333
|
+
|
334
|
+
RSpec.configure do |config|
|
335
|
+
config.include Counter::RSpecMatchers, type: :counter
|
207
336
|
end
|
208
337
|
```
|
209
338
|
|
210
|
-
|
339
|
+
Now you can test your counter definitions like so:
|
211
340
|
|
212
341
|
```ruby
|
213
|
-
|
214
|
-
|
215
|
-
|
216
|
-
|
217
|
-
|
218
|
-
|
219
|
-
|
220
|
-
|
221
|
-
|
222
|
-
|
342
|
+
require "rails_helper"
|
343
|
+
|
344
|
+
RSpec.describe PremiumProductCounter, type: :counter do
|
345
|
+
let(:store) { create(:store) }
|
346
|
+
|
347
|
+
describe "on :create" do
|
348
|
+
context "when the product is premium" do
|
349
|
+
it "increments the counter" do
|
350
|
+
expect { create(:product, :premium, store: store) }.to increment_counter_for(described_class, store)
|
351
|
+
end
|
352
|
+
end
|
353
|
+
|
354
|
+
context "when the product is not premium" do
|
355
|
+
it "doesn't increment the counter" do
|
356
|
+
expect { create(:product, store: store) }.not_to increment_counter_for(described_class, store)
|
357
|
+
end
|
358
|
+
end
|
359
|
+
end
|
223
360
|
|
224
|
-
|
225
|
-
|
361
|
+
describe "on :delete" do
|
362
|
+
context "when the product is premium" do
|
363
|
+
it "decrements the counter" do
|
364
|
+
expect { create(:product, :premium, store: store) }.to decrement_counter_for(described_class, store)
|
365
|
+
end
|
366
|
+
end
|
367
|
+
|
368
|
+
context "when the product is not premium" do
|
369
|
+
it "doesn't decrement the counter" do
|
370
|
+
expect { create(:product, store: store) }.not_to decrement_counter_for(described_class, store)
|
371
|
+
end
|
372
|
+
end
|
373
|
+
end
|
374
|
+
end
|
226
375
|
```
|
227
376
|
|
228
|
-
|
229
|
-
|
230
|
-
## Reset a counter
|
231
|
-
|
232
|
-
You can also reset a counter by calling `reset`. Since counters are ActiveRecord objects, you could also reset them using
|
377
|
+
### In production
|
233
378
|
|
234
|
-
|
235
|
-
store.order_revenue.reset
|
236
|
-
Counter::Value.update value: 0
|
237
|
-
```
|
379
|
+
> test in prod or live a lie — Charity Majors
|
238
380
|
|
239
|
-
|
381
|
+
It's very useful to verify the accuracy of the counters in production, especially if you are concerned about conditional counters etc causing counter drift over time.
|
240
382
|
|
241
|
-
|
383
|
+
A simple approach would be:
|
242
384
|
|
243
385
|
```ruby
|
244
|
-
|
386
|
+
Counter::Value.all.each &:correct!
|
245
387
|
```
|
246
388
|
|
247
|
-
|
389
|
+
If you have a large number of counters though it's best to take a sampling approach to randomly select a counter and verify that the value is correct
|
248
390
|
|
249
391
|
```ruby
|
250
|
-
|
251
|
-
store.product_revenue.reset!
|
252
|
-
store.product_revenue #=>0
|
253
|
-
store.product_revenue.correct? #=> false
|
254
|
-
store.product_revenue.correct! #=> false
|
255
|
-
store.product_revenue #=>200
|
392
|
+
Counter::Value.sample_and_verify samples: 1000, verbose: true, on_error: :correct
|
256
393
|
```
|
257
394
|
|
258
|
-
|
395
|
+
Options:
|
259
396
|
|
260
|
-
|
261
|
-
|
262
|
-
|
263
|
-
|
264
|
-
count :orders, as: :order_revenue
|
265
|
-
sum :price
|
266
|
-
|
267
|
-
after_change :send_congratulations_email
|
268
|
-
|
269
|
-
def send_congratulations_email counter, from, to
|
270
|
-
return unless from < 1000 && to >= 1000
|
271
|
-
send_email "Congratulations! You've made #{to} dollars!"
|
272
|
-
end
|
273
|
-
end
|
274
|
-
```
|
397
|
+
- scope — allows you to scope the counters to a particular model or set of models, e.g. `scope: -> { where("name LIKE 'store-%'") }`. By default, all counters are sampled
|
398
|
+
- samples — the number of counters to sample. Default: 1000
|
399
|
+
- verbose — print out the counter details and whether it was correct. Default: true
|
400
|
+
- on_error — what to do when a counter is incorrect. `:correct` will correct the counter, `:raise` will raise an error, `:log` will log the error to Rails.logger. Default: :raise
|
275
401
|
|
276
402
|
---
|
277
403
|
|
278
404
|
## TODO
|
279
405
|
|
280
406
|
See the asociated project in Github but roughly I'm thinking:
|
407
|
+
- Implement the background job pattern for incrementing counters
|
281
408
|
- Hierarchical counters. For example, a Site sends many Newsletters and each Newsletter results in many EmailMessages. Each EmailMessage can be marked as spam. How do you create counters for how many spam emails were sent at the Newsletter level and the Site level?
|
282
409
|
- Time-based counters for analytics. Instead of a User having one OrderRevenue counter, they would have an OrderRevenue counter for each day. These counters would then be used to produce a chart of their product revenue over the month. Not sure if these are just special counters or something else entirely? Do they use the same ActiveRecord model?
|
283
|
-
- Can we support floating point values? Sounds useful but don't have a use case for it right now. Would they need to be a different ActiveRecord table?
|
284
410
|
- In a similar vein of supporting different value types, can we support HLL values? Instead of increment an integer we add the items hash to a HyperLogLog so we can count unique items. An example would be counting site visits in a time-based daily counter, then combine the daily counts and still obtain an estimated number of monthly _unique_ visits. Again, not sure if this is the same ActiveRecord model or something different.
|
285
411
|
- Actually start running this in production for basic use cases
|
286
412
|
|
287
|
-
##
|
288
|
-
No one has used this in production yet.
|
289
|
-
|
290
|
-
You probably shouldn't right now unless you're the sort of person that checks if something is poisonous by licking it.
|
291
|
-
|
292
|
-
## Installation
|
293
|
-
Add this line to your application's Gemfile:
|
294
|
-
|
295
|
-
```ruby
|
296
|
-
gem 'counter'
|
297
|
-
```
|
298
|
-
|
299
|
-
And then execute:
|
300
|
-
```bash
|
301
|
-
$ bundle
|
302
|
-
```
|
413
|
+
## Contributing
|
303
414
|
|
304
|
-
|
305
|
-
```bash
|
306
|
-
$ rails counter:install:migrations
|
307
|
-
```
|
415
|
+
Bug reports and pull requests are welcome, especially around naming, internal APIs, bug fixes, and additional features. Please open an issue first if you're thinking of adding a new feature so we can discuss it.
|
308
416
|
|
309
|
-
|
310
|
-
Contribution directions go here.
|
417
|
+
I'm unlikely to entertain suport for older Ruby or Rails versions, or databases other than Postgres.
|
311
418
|
|
312
419
|
## License
|
420
|
+
|
313
421
|
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
@@ -0,0 +1,25 @@
|
|
1
|
+
module Counter::Calculated
|
2
|
+
extend ActiveSupport::Concern
|
3
|
+
|
4
|
+
included do
|
5
|
+
def calculate!
|
6
|
+
new_value = calculate
|
7
|
+
update! value: new_value unless new_value.nil?
|
8
|
+
end
|
9
|
+
|
10
|
+
def calculate
|
11
|
+
counters = counters_for_calculation
|
12
|
+
# If any of the counters are missing, we can't calculate
|
13
|
+
return if counters.any?(&:nil?)
|
14
|
+
|
15
|
+
definition.calculated_from.call(*counters)
|
16
|
+
end
|
17
|
+
|
18
|
+
def counters_for_calculation
|
19
|
+
# Fetch the dependant counters
|
20
|
+
definition.dependent_counters.map do |counter|
|
21
|
+
parent.counters.find_counter(counter)
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
25
|
+
end
|
@@ -1,11 +1,16 @@
|
|
1
1
|
module Counter::Recalculatable
|
2
2
|
extend ActiveSupport::Concern
|
3
3
|
|
4
|
-
####################################################### Support for regenerating the counters
|
5
4
|
def recalc!
|
6
|
-
|
7
|
-
|
8
|
-
|
5
|
+
if definition.calculated?
|
6
|
+
calculate!
|
7
|
+
elsif definition.manual?
|
8
|
+
raise Counter::Error.new("Can't recalculate a manual counter")
|
9
|
+
else
|
10
|
+
with_lock do
|
11
|
+
new_value = definition.sum? ? sum_by_sql : count_by_sql
|
12
|
+
update! value: new_value
|
13
|
+
end
|
9
14
|
end
|
10
15
|
end
|
11
16
|
|
@@ -2,13 +2,66 @@ module Counter::Verifyable
|
|
2
2
|
extend ActiveSupport::Concern
|
3
3
|
|
4
4
|
def correct?
|
5
|
-
|
5
|
+
# We can't verify these values
|
6
|
+
return true if definition.global?
|
7
|
+
|
8
|
+
old_value, new_value = verify
|
9
|
+
old_value == new_value
|
6
10
|
end
|
7
11
|
|
8
12
|
def correct!
|
9
|
-
|
10
|
-
|
13
|
+
# We can't verify these values
|
14
|
+
return true if definition.global?
|
15
|
+
|
16
|
+
old_value, new_value = verify
|
17
|
+
|
18
|
+
requires_recalculation = old_value != new_value
|
19
|
+
update! value: new_value if requires_recalculation
|
11
20
|
|
12
21
|
!requires_recalculation
|
13
22
|
end
|
23
|
+
|
24
|
+
def verify
|
25
|
+
if definition.calculated?
|
26
|
+
[calculate, value]
|
27
|
+
else
|
28
|
+
[count_by_sql, value]
|
29
|
+
end
|
30
|
+
end
|
31
|
+
|
32
|
+
class_methods do
|
33
|
+
# on_error: raise, log, correct
|
34
|
+
# Returns the number of incorrect counters
|
35
|
+
def sample_and_verify scope: -> { all }, samples: 1000, verbose: true, on_error: :raise
|
36
|
+
incorrect_counters = 0
|
37
|
+
|
38
|
+
counter_range = Counter::Value.minimum(:id)..Counter::Value.maximum(:id)
|
39
|
+
|
40
|
+
samples.times do
|
41
|
+
random_id = rand(counter_range)
|
42
|
+
counter = Counter::Value.merge(scope).where("id >= ?", random_id).limit(1).first
|
43
|
+
next if counter.nil?
|
44
|
+
|
45
|
+
if counter.definition.global? || counter.definition.calculated?
|
46
|
+
puts "➡️ Skipping counter #{counter.name} (#{counter.id})" if verbose
|
47
|
+
next
|
48
|
+
end
|
49
|
+
|
50
|
+
if counter.correct?
|
51
|
+
puts "✅ Counter #{counter.id} is correct" if verbose
|
52
|
+
else
|
53
|
+
incorrect_counters += 1
|
54
|
+
message = "❌ counter #{counter.name} (#{counter.id}) for #{counter.parent_type}##{counter.parent_id} has incorrect counter value. Expected #{counter.value} but got #{counter.count_by_sql}"
|
55
|
+
|
56
|
+
case on_error
|
57
|
+
when :raise then raise Counter::Error.new(message)
|
58
|
+
when :log then Rails.logger.error message
|
59
|
+
when :correct then counter.correct!
|
60
|
+
end
|
61
|
+
end
|
62
|
+
sleep 0.1
|
63
|
+
end
|
64
|
+
incorrect_counters
|
65
|
+
end
|
66
|
+
end
|
14
67
|
end
|
data/app/models/counter/value.rb
CHANGED
@@ -2,7 +2,7 @@ class CreateCounterValues < ActiveRecord::Migration[6.1]
|
|
2
2
|
def change
|
3
3
|
create_table :counter_values do |t|
|
4
4
|
t.string :name, index: true
|
5
|
-
t.
|
5
|
+
t.decimal :value, default: 0.0, null: false
|
6
6
|
t.references :parent, polymorphic: true
|
7
7
|
|
8
8
|
t.timestamps
|
data/lib/counter/definition.rb
CHANGED
@@ -5,11 +5,7 @@
|
|
5
5
|
# # This specifies the association we're counting
|
6
6
|
# count :products
|
7
7
|
# sum :price # optional
|
8
|
-
#
|
9
|
-
# create: ->(product) { product.premium? }
|
10
|
-
# update: ->(product) { product.has_changed? :premium, to: :true }
|
11
|
-
# delete: ->(product) { product.premium? }
|
12
|
-
# }
|
8
|
+
# as "my_counter"
|
13
9
|
# end
|
14
10
|
class Counter::Definition
|
15
11
|
include Singleton
|
@@ -34,21 +30,38 @@ class Counter::Definition
|
|
34
30
|
attr_writer :global_counters
|
35
31
|
# An array of Proc to run when the counter changes
|
36
32
|
attr_writer :counter_hooks
|
37
|
-
#
|
38
|
-
attr_writer :
|
33
|
+
# The counters this calculated counter depends on
|
34
|
+
attr_writer :dependent_counters
|
35
|
+
# The block to call to calculate the counter
|
36
|
+
attr_accessor :calculated_from
|
39
37
|
|
38
|
+
# Is this a counter which sums a column?
|
40
39
|
def sum?
|
41
40
|
column_to_count.present?
|
42
41
|
end
|
43
42
|
|
43
|
+
# Is this a global counter? i.e., not attached to a model
|
44
44
|
def global?
|
45
|
-
model.nil?
|
45
|
+
model.nil?
|
46
46
|
end
|
47
47
|
|
48
|
+
# Is this counter conditional?
|
48
49
|
def conditional?
|
49
50
|
@conditional
|
50
51
|
end
|
51
52
|
|
53
|
+
# Is this counter calculated from other counters?
|
54
|
+
def calculated?
|
55
|
+
!@calculated_from.nil?
|
56
|
+
end
|
57
|
+
|
58
|
+
# Is this a manual counter?
|
59
|
+
# Manual counters are not automatically updated from an association
|
60
|
+
# or calculated from other counters
|
61
|
+
def manual?
|
62
|
+
association_name.nil? && !calculated?
|
63
|
+
end
|
64
|
+
|
52
65
|
# for global counter instances to find their definition
|
53
66
|
def self.find_definition name
|
54
67
|
Counter::Definition.instance.global_counters.find { |c| c.name == name }
|
@@ -64,7 +77,8 @@ class Counter::Definition
|
|
64
77
|
# What we record in Counter::Value#name
|
65
78
|
def record_name
|
66
79
|
return name if global?
|
67
|
-
"#{model.name.underscore}-#{association_name}"
|
80
|
+
return "#{model.name.underscore}-#{association_name}" if association_name.present?
|
81
|
+
return "#{model.name.underscore}-#{name}"
|
68
82
|
end
|
69
83
|
|
70
84
|
def conditions
|
@@ -82,9 +96,9 @@ class Counter::Definition
|
|
82
96
|
@counter_hooks
|
83
97
|
end
|
84
98
|
|
85
|
-
def
|
86
|
-
@
|
87
|
-
@
|
99
|
+
def dependent_counters
|
100
|
+
@dependent_counters ||= []
|
101
|
+
@dependent_counters
|
88
102
|
end
|
89
103
|
|
90
104
|
# Set the association we're counting
|
@@ -95,12 +109,36 @@ class Counter::Definition
|
|
95
109
|
instance.method_name = as.to_s
|
96
110
|
end
|
97
111
|
|
98
|
-
def self.global
|
99
|
-
name ||= name.underscore
|
100
|
-
instance.name = name.to_s
|
112
|
+
def self.global
|
101
113
|
Counter::Definition.instance.global_counters << instance
|
102
114
|
end
|
103
115
|
|
116
|
+
def self.calculated_from *dependent_counters, &block
|
117
|
+
instance.dependent_counters = dependent_counters
|
118
|
+
instance.calculated_from = block
|
119
|
+
|
120
|
+
dependent_counters.each do |dependent_counter|
|
121
|
+
# Install after_change hooks on the dependent counters
|
122
|
+
dependent_counter.after_change :update_calculated_counters
|
123
|
+
dependent_counter.define_method :update_calculated_counters do |counter, _old_value, _new_value|
|
124
|
+
# Fetch all the counters which depend on this one
|
125
|
+
calculated_counters = counter.parent.class.counter_configs.select { |c|
|
126
|
+
c.dependent_counters.include?(counter.definition.class)
|
127
|
+
}
|
128
|
+
|
129
|
+
calculated_counters = calculated_counters.map { |c| counter.parent.counters.find_or_create_counter!(c) }
|
130
|
+
# calculate the new values
|
131
|
+
calculated_counters.each(&:calculate!)
|
132
|
+
end
|
133
|
+
end
|
134
|
+
end
|
135
|
+
|
136
|
+
# Set the name of the counter
|
137
|
+
def self.as name
|
138
|
+
instance.name = name.to_s
|
139
|
+
instance.method_name = name.to_s
|
140
|
+
end
|
141
|
+
|
104
142
|
# Get the name of the association we're counting
|
105
143
|
def self.association_name
|
106
144
|
instance.association_name
|
@@ -26,8 +26,11 @@ module Counter::Countable
|
|
26
26
|
def each_counter_to_update
|
27
27
|
# For each definition, find or create the counter on the parent
|
28
28
|
self.class.counted_by.each do |counter_definition|
|
29
|
-
|
30
|
-
|
29
|
+
next unless counter_definition.inverse_association
|
30
|
+
|
31
|
+
parent_association = association(counter_definition.inverse_association)
|
32
|
+
parent_association.load_target unless parent_association.loaded?
|
33
|
+
parent_model = parent_association.target
|
31
34
|
next unless parent_model
|
32
35
|
counter = parent_model.counters.find_or_create_counter!(counter_definition)
|
33
36
|
yield counter if counter
|
@@ -55,34 +55,66 @@ module Counter::Counters
|
|
55
55
|
counter_definitions = Array.wrap(counter_definitions)
|
56
56
|
counter_definitions.each do |definition_class|
|
57
57
|
definition = definition_class.instance
|
58
|
-
association_name = definition.association_name
|
59
|
-
|
60
|
-
# Find the association on this model
|
61
|
-
association_reflection = reflect_on_association(association_name)
|
62
|
-
# Find the association classes
|
63
|
-
association_class = association_reflection.class_name.constantize
|
64
|
-
inverse_association = association_reflection.inverse_of
|
65
|
-
|
66
|
-
raise Counter::Error.new("#{association_name} must have an inverse_of specified to be used in #{definition_class.name}") if inverse_association.nil?
|
67
|
-
|
68
|
-
# Add the after_commit hook to the association's class
|
69
|
-
association_class.include Counter::Countable
|
70
|
-
# association_class.include Counter::Changed
|
71
|
-
|
72
|
-
# Update the definition with the association class and inverse association
|
73
|
-
# gathered from the reflection
|
74
58
|
definition.model = self
|
75
|
-
|
76
|
-
|
59
|
+
|
60
|
+
scope :with_counter_data_from, ->(*counter_classes) {
|
61
|
+
subqueries = ["#{table_name}.*"]
|
62
|
+
counter_classes.each do |counter_class|
|
63
|
+
sql = Counter::Value.select("value")
|
64
|
+
.where("parent_id = #{table_name}.id AND parent_type = '#{name}' AND name = '#{counter_class.instance.record_name}'").to_sql
|
65
|
+
subqueries << "(#{sql}) AS #{counter_class.instance.name}_data"
|
66
|
+
end
|
67
|
+
select(subqueries)
|
68
|
+
}
|
69
|
+
|
70
|
+
# Expects a hash of counter classes and directions, like so:
|
71
|
+
# order_by_counter ProductCounter => :desc, PremiumProductCounter => :asc
|
72
|
+
scope :order_by_counter, ->(order_hash) {
|
73
|
+
counter_classes = order_hash.keys.select { |counter_class|
|
74
|
+
counter_class.is_a?(Class) &&
|
75
|
+
counter_class.ancestors.include?(Counter::Definition)
|
76
|
+
}
|
77
|
+
order_params = {}
|
78
|
+
order_hash.map do |counter_class, direction|
|
79
|
+
if counter_class.is_a?(String) || counter_class.is_a?(Symbol)
|
80
|
+
order_params[counter_class] = direction
|
81
|
+
elsif counter_class.ancestors.include?(Counter::Definition)
|
82
|
+
order_params["#{counter_class.instance.name}_data"] = direction
|
83
|
+
end
|
84
|
+
end
|
85
|
+
with_counter_data_from(*counter_classes).order(order_params)
|
86
|
+
}
|
87
|
+
|
88
|
+
scope :with_counters, -> { includes(:counters) }
|
77
89
|
|
78
90
|
define_method definition.method_name do
|
79
91
|
counters.find_or_create_counter!(definition)
|
80
92
|
end
|
81
93
|
|
82
|
-
|
94
|
+
@counter_configs << definition unless @counter_configs.include?(definition)
|
83
95
|
|
84
|
-
|
85
|
-
|
96
|
+
association_name = definition.association_name
|
97
|
+
if association_name.present?
|
98
|
+
# Find the association on this model
|
99
|
+
association_reflection = reflect_on_association(association_name)
|
100
|
+
raise Counter::Error.new("#{association_name} does not exist #{self.name}") if association_reflection.nil?
|
101
|
+
|
102
|
+
# Find the association classes
|
103
|
+
association_class = association_reflection.class_name.constantize
|
104
|
+
inverse_association = association_reflection.inverse_of
|
105
|
+
raise Counter::Error.new("#{association_name} must have an inverse_of specified to be used in #{definition_class.name}") if inverse_association.nil?
|
106
|
+
|
107
|
+
# Add the after_commit hook to the association's class
|
108
|
+
association_class.include Counter::Countable
|
109
|
+
|
110
|
+
# Update the definition with the association class and inverse association
|
111
|
+
# gathered from the reflection
|
112
|
+
definition.inverse_association = inverse_association.name
|
113
|
+
definition.countable_model = association_class
|
114
|
+
|
115
|
+
# Provide the Countable class with details about where it's counted
|
116
|
+
association_class.add_counted_by definition
|
117
|
+
end
|
86
118
|
end
|
87
119
|
end
|
88
120
|
|
@@ -0,0 +1,29 @@
|
|
1
|
+
module Counter
|
2
|
+
module RSpecMatchers
|
3
|
+
def increment_counter_for(...)
|
4
|
+
IncrementCounterFor.new(...)
|
5
|
+
end
|
6
|
+
|
7
|
+
def decrement_counter_for(...)
|
8
|
+
DecrementCounterFor.new(...)
|
9
|
+
end
|
10
|
+
|
11
|
+
class Base < RSpec::Matchers::BuiltIn::Change
|
12
|
+
def initialize(counter_class, parent)
|
13
|
+
super { parent.counters.find_or_create_counter!(counter_class).value }
|
14
|
+
end
|
15
|
+
end
|
16
|
+
|
17
|
+
class IncrementCounterFor < Base
|
18
|
+
def matches?(...)
|
19
|
+
by(1).matches?(...)
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
class DecrementCounterFor < Base
|
24
|
+
def matches?(...)
|
25
|
+
by(-1).matches?(...)
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
data/lib/counter/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: counterwise
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jamie Lawrence
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-08-16 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: rails
|
@@ -52,6 +52,7 @@ files:
|
|
52
52
|
- app/controllers/counters_controller.rb
|
53
53
|
- app/jobs/counter/reconciliation_job.rb
|
54
54
|
- app/models/concerns/counter/Xhierarchical.rb
|
55
|
+
- app/models/concerns/counter/calculated.rb
|
55
56
|
- app/models/concerns/counter/changable.rb
|
56
57
|
- app/models/concerns/counter/conditional.rb
|
57
58
|
- app/models/concerns/counter/definable.rb
|
@@ -62,11 +63,9 @@ files:
|
|
62
63
|
- app/models/concerns/counter/sidekiq_reconciliation.rb
|
63
64
|
- app/models/concerns/counter/summable.rb
|
64
65
|
- app/models/concerns/counter/verifyable.rb
|
65
|
-
- app/models/counter/change.rb
|
66
66
|
- app/models/counter/value.rb
|
67
67
|
- config/routes.rb
|
68
68
|
- db/migrate/20210705154113_create_counter_values.rb
|
69
|
-
- db/migrate/20210709211056_create_counter_changes.rb
|
70
69
|
- db/migrate/20210731224504_add_unique_index_to_counter_values.rb
|
71
70
|
- lib/counter.rb
|
72
71
|
- lib/counter/any.rb
|
@@ -77,6 +76,7 @@ files:
|
|
77
76
|
- lib/counter/integration/countable.rb
|
78
77
|
- lib/counter/integration/counters.rb
|
79
78
|
- lib/counter/railtie.rb
|
79
|
+
- lib/counter/rspec/matchers.rb
|
80
80
|
- lib/counter/version.rb
|
81
81
|
- lib/tasks/counter_tasks.rake
|
82
82
|
homepage: https://github.com/podia/counter
|
@@ -1,23 +0,0 @@
|
|
1
|
-
# == Schema Information
|
2
|
-
#
|
3
|
-
# Table name: counter_changes
|
4
|
-
#
|
5
|
-
# id :integer not null, primary key
|
6
|
-
# counter_value_id :integer indexed
|
7
|
-
# amount :integer
|
8
|
-
# processed_at :datetime indexed
|
9
|
-
# created_at :datetime not null
|
10
|
-
# updated_at :datetime not null
|
11
|
-
#
|
12
|
-
class Counter::Change < ApplicationRecord
|
13
|
-
def self.table_name_prefix
|
14
|
-
"counter_"
|
15
|
-
end
|
16
|
-
|
17
|
-
belongs_to :counter, class_name: "Counter::Value"
|
18
|
-
validates_numericality_of :amount
|
19
|
-
|
20
|
-
scope :pending, -> { where(reconciled_at: nil) }
|
21
|
-
scope :reconciled, -> { where.not(reconciled_at: nil) }
|
22
|
-
scope :purgable, -> { reconciled.where(processed_at: 7.days.ago..) }
|
23
|
-
end
|