idempotency_check 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/LICENSE.txt +21 -0
- data/README.md +134 -0
- data/lib/idempotency_check/active_job_perform_tracker.rb +29 -0
- data/lib/idempotency_check/active_job_snapshot.rb +23 -0
- data/lib/idempotency_check/checker.rb +78 -0
- data/lib/idempotency_check/http_tracker.rb +26 -0
- data/lib/idempotency_check/mailer_snapshot.rb +16 -0
- data/lib/idempotency_check/row_snapshot.rb +33 -0
- data/lib/idempotency_check/rspec.rb +21 -0
- data/lib/idempotency_check/sidekiq.rb +69 -0
- data/lib/idempotency_check/sql_tracker.rb +48 -0
- data/lib/idempotency_check/table_snapshot.rb +23 -0
- data/lib/idempotency_check/version.rb +5 -0
- data/lib/idempotency_check.rb +23 -0
- metadata +161 -0
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: fbbc8f01fa1d5065ce9b934f19f2906617273c926a1094bb70b801cadb534a12
|
|
4
|
+
data.tar.gz: 65e82375d324b949971364189c89895554502de934b7ba0ce35e0cdbd33ed2dc
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: 5ee41662951a3af3fa1f7c83a4217c2b624017d4fd3eed6b60f740ce224f3f055bbf6ca4796bed4f2a3f413a4761ab6d9ffeb5fab814349fc96f3c3e0b8d3969
|
|
7
|
+
data.tar.gz: bb28dc2ce5e986ae2f78bcf383614dff955086973cdeee8be5d86198f21f99c72a14100880a564cd4845122cc523103ff9a31c4076b8bf590c8bf9d62b09fbc8
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Agustin Fornio
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
data/README.md
ADDED
|
@@ -0,0 +1,134 @@
|
|
|
1
|
+
# idempotency_check
|
|
2
|
+
|
|
3
|
+
Detects whether a job — or any block of Ruby code — is idempotent, by running it multiple times and comparing the side effects it produces.
|
|
4
|
+
|
|
5
|
+
It works by running your block 3 times and comparing, between runs:
|
|
6
|
+
|
|
7
|
+
- database row counts and content (via ActiveRecord)
|
|
8
|
+
- ActionMailer deliveries
|
|
9
|
+
- ActiveJob activity (`perform_later` and `perform_now`), broken down by job class and arguments
|
|
10
|
+
- optionally, Sidekiq activity (`perform_async`), if you opt in
|
|
11
|
+
- outbound HTTP calls, which can't be verified, so they're flagged instead of assumed safe
|
|
12
|
+
|
|
13
|
+
It has **no runtime dependencies**.
|
|
14
|
+
It works on any block of code, not only jobs — `IdempotencyCheck.idempotent? { user.update!(...) }` works the same way as it does for a job's `perform`.
|
|
15
|
+
|
|
16
|
+
## Installation
|
|
17
|
+
|
|
18
|
+
```ruby
|
|
19
|
+
gem "idempotency_check"
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Usage
|
|
23
|
+
|
|
24
|
+
```ruby
|
|
25
|
+
require "idempotency_check"
|
|
26
|
+
|
|
27
|
+
IdempotencyCheck.idempotent? { MyJob.perform_now(user_id) }
|
|
28
|
+
# => true, false
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
Returns `true` only when running the block again produces no additional effect.
|
|
32
|
+
Any unconfirmed situation (see "HTTP calls" below) is treated as `false` by default — the safe assumption is "not confirmed idempotent", never a false positive.
|
|
33
|
+
|
|
34
|
+
### Ignoring timestamp-only changes
|
|
35
|
+
|
|
36
|
+
A job that calls `.touch`/`.save` on every run bumps `updated_at` even if nothing else changed.
|
|
37
|
+
By default that counts as non-idempotent (strict); opt out with:
|
|
38
|
+
|
|
39
|
+
```ruby
|
|
40
|
+
IdempotencyCheck.idempotent?(ignore_timestamps: true) { MyJob.perform_now }
|
|
41
|
+
# or, to ignore other specific columns:
|
|
42
|
+
IdempotencyCheck.idempotent?(ignore_columns: ["last_checked_at"]) { MyJob.perform_now }
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
### HTTP calls
|
|
46
|
+
|
|
47
|
+
Outbound HTTP calls can't be verified — the gem doesn't know whether the external service is itself idempotent.
|
|
48
|
+
By default, any HTTP call during the run makes the result `false` (unconfirmed = not safe to assume).
|
|
49
|
+
If you've independently verified a particular source is safe (e.g. it calls your own idempotent internal API), exclude it from the check:
|
|
50
|
+
|
|
51
|
+
```ruby
|
|
52
|
+
IdempotencyCheck.idempotent?(trust: [:http]) { MyJob.perform_now }
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
`trust:` accepts any of the sources the checker tracks: `:table`, `:row`, `:mailer`, `:active_job`, `:http`, and (if loaded) `:sidekiq`.
|
|
56
|
+
|
|
57
|
+
### Getting a reason, not just true/false
|
|
58
|
+
|
|
59
|
+
```ruby
|
|
60
|
+
IdempotencyCheck.idempotent?(verbose: true) { MyJob.perform_now }
|
|
61
|
+
# => { result: true, reason: :ok }
|
|
62
|
+
# => { result: false, reason: [:row] } # which source(s) changed
|
|
63
|
+
# => { result: false, reason: :suspicious_http } # DB looked fine, but there was an unverified HTTP call
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## Optional integrations
|
|
67
|
+
|
|
68
|
+
Neither of these are gemspec dependencies — they're only loaded (and only need their gem installed) if you explicitly require them.
|
|
69
|
+
|
|
70
|
+
### RSpec
|
|
71
|
+
|
|
72
|
+
```ruby
|
|
73
|
+
require "idempotency_check/rspec"
|
|
74
|
+
|
|
75
|
+
expect { MyJob.perform_now(user_id) }.to be_idempotent
|
|
76
|
+
expect { MyJob.perform_now(user_id) }.to be_idempotent(trust: [:http])
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
### Sidekiq (native `include Sidekiq::Job` / `perform_async`)
|
|
80
|
+
|
|
81
|
+
ActiveJob is supported out of the box, even when Sidekiq is its queue adapter.
|
|
82
|
+
This is only needed for jobs that use Sidekiq's own API directly (`include Sidekiq::Job`, `perform_async`), which doesn't go through ActiveJob at all.
|
|
83
|
+
|
|
84
|
+
```ruby
|
|
85
|
+
require "idempotency_check/sidekiq"
|
|
86
|
+
|
|
87
|
+
IdempotencyCheck.idempotent? { MyWorker.perform_async(user_id) }
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## The check does not undo anything
|
|
91
|
+
|
|
92
|
+
Running a check executes your block **3 times, for real**.
|
|
93
|
+
Nothing is rolled back.
|
|
94
|
+
A block that creates a row leaves 3 rows behind; one that sends a mail sends 3 mails.
|
|
95
|
+
|
|
96
|
+
Run checks inside a transactional test, which is what they're designed for:
|
|
97
|
+
|
|
98
|
+
```ruby
|
|
99
|
+
# RSpec, with use_transactional_fixtures = true (the default)
|
|
100
|
+
it "is safe to retry" do
|
|
101
|
+
expect { MyJob.perform_now(user.id) }.to be_idempotent
|
|
102
|
+
end
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
The surrounding transaction wraps all 3 runs and rolls the database back when the example ends.
|
|
106
|
+
Mailer deliveries and enqueued jobs aren't covered by the transaction, but the checker resets those itself at the start of every check, so they don't leak between examples.
|
|
107
|
+
|
|
108
|
+
Not undone by a transaction, in a test or anywhere else:
|
|
109
|
+
|
|
110
|
+
- **HTTP requests** — they really go out, 3 times.
|
|
111
|
+
This is also why an unverified HTTP call makes the result `false` by default.
|
|
112
|
+
- **Sidekiq's native `perform_async`** — pushes to Redis for real.
|
|
113
|
+
- **Redis and `Rails.cache`** — no rollback.
|
|
114
|
+
- **`after_commit`** — inside a test transaction the real commit never happens, so a job that depends on it may not behave the way it does in production.
|
|
115
|
+
|
|
116
|
+
Running a check against a development console is a bad idea for the same reason: it will triple whatever the block does.
|
|
117
|
+
|
|
118
|
+
## What it doesn't cover (yet)
|
|
119
|
+
|
|
120
|
+
- Automatic retries (`retry_on`/`discard_on`) — the checker runs the block cleanly 3 times, it doesn't simulate a mid-run failure and retry.
|
|
121
|
+
- Cache/Redis side effects (e.g. `Rails.cache.increment`).
|
|
122
|
+
- External storage (S3, ActiveStorage).
|
|
123
|
+
- Real concurrency — every run is sequential, not parallel.
|
|
124
|
+
- Writes that don't go through the Rails adapter: database triggers and `ON DELETE CASCADE`.
|
|
125
|
+
Those tables never show up in `sql.active_record`, so they're not snapshotted.
|
|
126
|
+
- HTTP clients that bypass `Net::HTTP` (`httpx`, `excon`, `typhoeus`, `curb`).
|
|
127
|
+
Anything built on `Net::HTTP` — Faraday's default adapter, HTTParty, RestClient — is seen.
|
|
128
|
+
|
|
129
|
+
## Development
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
bundle install
|
|
133
|
+
bundle exec rake test
|
|
134
|
+
```
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'active_support/notifications'
|
|
4
|
+
|
|
5
|
+
module IdempotencyCheck
|
|
6
|
+
# Accumulates every job actually performed (synchronously, e.g. via
|
|
7
|
+
# perform_now) since the last reset!, by subscribing to ActiveJob's core
|
|
8
|
+
# "perform.active_job" notification. This fires regardless of queue
|
|
9
|
+
# adapter, but with queue_adapter = :test it's the only way to see
|
|
10
|
+
# synchronous performs, since perform_later jobs just sit in the queue
|
|
11
|
+
# and never actually run.
|
|
12
|
+
module ActiveJobPerformTracker
|
|
13
|
+
@performed = []
|
|
14
|
+
|
|
15
|
+
class << self
|
|
16
|
+
attr_reader :performed
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def self.reset!
|
|
20
|
+
@performed = []
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
ActiveSupport::Notifications.subscribe('perform.active_job') do |*args|
|
|
24
|
+
event = ActiveSupport::Notifications::Event.new(*args)
|
|
25
|
+
job = event.payload[:job]
|
|
26
|
+
@performed << [job.class.name, job.arguments]
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module IdempotencyCheck
|
|
4
|
+
# Snapshots job activity so far (cumulative since the last reset!), broken
|
|
5
|
+
# down by job class and arguments so that swapping which job/args ran
|
|
6
|
+
# between runs is caught, not just a raw count. Covers both:
|
|
7
|
+
# - enqueued jobs (perform_later), via ActiveJob's :test queue adapter
|
|
8
|
+
# - performed jobs (perform_now), via ActiveJobPerformTracker
|
|
9
|
+
# Requires ActiveJob::Base.queue_adapter = :test in the host app.
|
|
10
|
+
class ActiveJobSnapshot
|
|
11
|
+
def self.take
|
|
12
|
+
{
|
|
13
|
+
enqueued: ActiveJob::Base.queue_adapter.enqueued_jobs.map { |job| [job[:job].name, job[:args]] },
|
|
14
|
+
performed: ActiveJobPerformTracker.performed.dup
|
|
15
|
+
}
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def self.reset!
|
|
19
|
+
ActiveJob::Base.queue_adapter.enqueued_jobs.clear
|
|
20
|
+
ActiveJobPerformTracker.reset!
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module IdempotencyCheck
|
|
4
|
+
class Checker
|
|
5
|
+
RUNS = 3
|
|
6
|
+
TIMESTAMP_COLUMNS = %w[created_at updated_at].freeze
|
|
7
|
+
|
|
8
|
+
@extra_snapshot_sources = {}
|
|
9
|
+
|
|
10
|
+
class << self
|
|
11
|
+
attr_reader :extra_snapshot_sources
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
# Lets optional integrations (e.g. idempotency_check/sidekiq) plug in
|
|
15
|
+
# their own snapshot source without the core Checker knowing about them.
|
|
16
|
+
# `name` is the symbol callers use to refer to this source via `trust:`.
|
|
17
|
+
# If the source responds to reset!, it's called before each check so
|
|
18
|
+
# cumulative trackers don't grow unbounded across an entire test suite.
|
|
19
|
+
def self.register_snapshot_source(name, source)
|
|
20
|
+
@extra_snapshot_sources[name] = source
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def self.idempotent?(ignore_columns: [], ignore_timestamps: false, trust: [], verbose: false, &block)
|
|
24
|
+
effective_ignore_columns = ignore_columns + (ignore_timestamps ? TIMESTAMP_COLUMNS : [])
|
|
25
|
+
trusted = Array(trust)
|
|
26
|
+
|
|
27
|
+
MailerSnapshot.reset!
|
|
28
|
+
ActiveJobSnapshot.reset!
|
|
29
|
+
extra_snapshot_sources.each_value { |source| source.reset! if source.respond_to?(:reset!) }
|
|
30
|
+
|
|
31
|
+
snapshots = {
|
|
32
|
+
table: [],
|
|
33
|
+
row: [],
|
|
34
|
+
mailer: [],
|
|
35
|
+
active_job: []
|
|
36
|
+
}
|
|
37
|
+
extra_snapshot_sources.each_key { |name| snapshots[name] = [] }
|
|
38
|
+
http_calls = 0
|
|
39
|
+
# Accumulated across runs: a run that writes a table no earlier run
|
|
40
|
+
# touched is itself a sign the block isn't settling down, and the
|
|
41
|
+
# snapshots will differ on that table anyway.
|
|
42
|
+
written_tables = []
|
|
43
|
+
|
|
44
|
+
RUNS.times do
|
|
45
|
+
http_calls += HttpTracker.track do
|
|
46
|
+
written_tables |= SqlTracker.track(&block)
|
|
47
|
+
end
|
|
48
|
+
snapshots[:table] << TableSnapshot.take(tables: written_tables)
|
|
49
|
+
snapshots[:row] << RowSnapshot.take(tables: written_tables, ignore_columns: effective_ignore_columns)
|
|
50
|
+
snapshots[:mailer] << MailerSnapshot.take
|
|
51
|
+
snapshots[:active_job] << ActiveJobSnapshot.take
|
|
52
|
+
extra_snapshot_sources.each { |name, source| snapshots[name] << source.take }
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
changed_sources = snapshots
|
|
56
|
+
.reject { |name, _| trusted.include?(name) }
|
|
57
|
+
.select { |_, list| list.uniq.size > 1 }
|
|
58
|
+
.keys
|
|
59
|
+
|
|
60
|
+
suspicious_http = http_calls.positive? && !trusted.include?(:http)
|
|
61
|
+
|
|
62
|
+
result = changed_sources.empty? && !suspicious_http
|
|
63
|
+
|
|
64
|
+
return result unless verbose
|
|
65
|
+
|
|
66
|
+
reason =
|
|
67
|
+
if changed_sources.any?
|
|
68
|
+
changed_sources
|
|
69
|
+
elsif suspicious_http
|
|
70
|
+
:suspicious_http
|
|
71
|
+
else
|
|
72
|
+
:ok
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
{ result: result, reason: reason }
|
|
76
|
+
end
|
|
77
|
+
end
|
|
78
|
+
end
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'net/http'
|
|
4
|
+
|
|
5
|
+
module IdempotencyCheck
|
|
6
|
+
# Counts how many outbound HTTP requests happen inside a block, by
|
|
7
|
+
# wrapping Net::HTTP#request. We can't verify idempotency of external
|
|
8
|
+
# side effects, so this is only used to flag results as suspicious.
|
|
9
|
+
class HttpTracker
|
|
10
|
+
def self.track
|
|
11
|
+
count = 0
|
|
12
|
+
original = Net::HTTP.instance_method(:request)
|
|
13
|
+
|
|
14
|
+
Net::HTTP.send(:define_method, :request) do |*args, &blk|
|
|
15
|
+
count += 1
|
|
16
|
+
original.bind(self).call(*args, &blk)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
yield
|
|
20
|
+
|
|
21
|
+
count
|
|
22
|
+
ensure
|
|
23
|
+
Net::HTTP.send(:define_method, :request, original) if original
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module IdempotencyCheck
|
|
4
|
+
# Snapshots how many mails have been delivered so far (cumulative since the
|
|
5
|
+
# last reset!), using ActionMailer's :test delivery method. Requires the
|
|
6
|
+
# host app to set config.action_mailer.delivery_method = :test.
|
|
7
|
+
class MailerSnapshot
|
|
8
|
+
def self.take
|
|
9
|
+
ActionMailer::Base.deliveries.size
|
|
10
|
+
end
|
|
11
|
+
|
|
12
|
+
def self.reset!
|
|
13
|
+
ActionMailer::Base.deliveries.clear
|
|
14
|
+
end
|
|
15
|
+
end
|
|
16
|
+
end
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module IdempotencyCheck
|
|
4
|
+
# Snapshots full row content per table, so two snapshots can be compared to
|
|
5
|
+
# catch changes that don't alter row counts (e.g. a counter being incremented).
|
|
6
|
+
#
|
|
7
|
+
# Only the tables the checked block wrote to are read — see SqlTracker.
|
|
8
|
+
class RowSnapshot
|
|
9
|
+
class << self
|
|
10
|
+
def take(tables:, ignore_columns: [])
|
|
11
|
+
connection = ActiveRecord::Base.connection
|
|
12
|
+
|
|
13
|
+
tables.each_with_object({}) do |table, rows|
|
|
14
|
+
rows[table] = fetch_rows(connection, table, ignore_columns)
|
|
15
|
+
end
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
private
|
|
19
|
+
|
|
20
|
+
def fetch_rows(connection, table, ignore_columns)
|
|
21
|
+
order_columns = Array(connection.primary_key(table))
|
|
22
|
+
sql = +"SELECT * FROM #{connection.quote_table_name(table)}"
|
|
23
|
+
sql << " ORDER BY #{order_columns.map { |c| connection.quote_column_name(c) }.join(', ')}" if order_columns.any?
|
|
24
|
+
|
|
25
|
+
rows = connection.select_all(sql).map { |row| row.except(*ignore_columns) }
|
|
26
|
+
# Tables without a usable primary key (e.g. composite/no-PK join
|
|
27
|
+
# tables) have no natural row order from the DB — sort the result
|
|
28
|
+
# ourselves so two snapshots taken back-to-back compare reliably.
|
|
29
|
+
order_columns.any? ? rows : rows.sort_by(&:to_s)
|
|
30
|
+
end
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
end
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require 'idempotency_check'
|
|
4
|
+
require 'rspec/expectations'
|
|
5
|
+
|
|
6
|
+
RSpec::Matchers.define :be_idempotent do |**opts|
|
|
7
|
+
supports_block_expectations
|
|
8
|
+
|
|
9
|
+
match do |block|
|
|
10
|
+
@result = IdempotencyCheck.idempotent?(**opts, &block)
|
|
11
|
+
@result == true
|
|
12
|
+
end
|
|
13
|
+
|
|
14
|
+
failure_message do
|
|
15
|
+
if @result == :suspicious
|
|
16
|
+
"expected the block to be idempotent, but it made external HTTP calls we can't verify"
|
|
17
|
+
else
|
|
18
|
+
'expected the block to be idempotent, but running it again produced different effects'
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative '../idempotency_check'
|
|
4
|
+
require 'sidekiq'
|
|
5
|
+
require 'sidekiq/test_api' # for Sidekiq::Testing.server_middleware, used under Sidekiq::Testing.inline!
|
|
6
|
+
|
|
7
|
+
module IdempotencyCheck
|
|
8
|
+
# Accumulates every job actually performed (via Sidekiq::Testing.inline! or
|
|
9
|
+
# a real worker process) for the lifetime of the process, by adding a
|
|
10
|
+
# server middleware. Sidekiq has no ActiveSupport::Notifications-style hook
|
|
11
|
+
# of its own, so we install our own.
|
|
12
|
+
module SidekiqPerformTracker
|
|
13
|
+
@performed = []
|
|
14
|
+
|
|
15
|
+
class << self
|
|
16
|
+
attr_reader :performed
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def self.record(job_class, args)
|
|
20
|
+
@performed << [job_class, args]
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
def self.reset!
|
|
24
|
+
@performed = []
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
class SidekiqServerTracker
|
|
29
|
+
def call(_job_instance, msg, _queue)
|
|
30
|
+
SidekiqPerformTracker.record(msg['class'], msg['args'])
|
|
31
|
+
yield
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# Snapshots Sidekiq job activity so far (cumulative), broken down by job
|
|
36
|
+
# class and arguments, covering both:
|
|
37
|
+
# - enqueued jobs (perform_async), via Sidekiq::Testing.fake!'s global
|
|
38
|
+
# Sidekiq::Queues registry (no middleware needed, Sidekiq tracks this
|
|
39
|
+
# itself)
|
|
40
|
+
# - performed jobs (perform_async under Sidekiq::Testing.inline!, or a
|
|
41
|
+
# real worker), via SidekiqServerTracker
|
|
42
|
+
class SidekiqSnapshot
|
|
43
|
+
def self.take
|
|
44
|
+
{
|
|
45
|
+
enqueued: Sidekiq::Queues.jobs_by_class.flat_map do |klass, jobs|
|
|
46
|
+
jobs.map { |job| [klass, job['args']] }
|
|
47
|
+
end,
|
|
48
|
+
performed: SidekiqPerformTracker.performed.dup
|
|
49
|
+
}
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def self.reset!
|
|
53
|
+
Sidekiq::Queues.clear_all
|
|
54
|
+
SidekiqPerformTracker.reset!
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Real worker process (production).
|
|
59
|
+
Sidekiq.default_configuration.server_middleware do |chain|
|
|
60
|
+
chain.add SidekiqServerTracker
|
|
61
|
+
end
|
|
62
|
+
|
|
63
|
+
# Sidekiq::Testing.inline! uses its own separate middleware chain.
|
|
64
|
+
Sidekiq::Testing.server_middleware do |chain|
|
|
65
|
+
chain.add SidekiqServerTracker
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
register_snapshot_source(:sidekiq, SidekiqSnapshot)
|
|
69
|
+
end
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module IdempotencyCheck
|
|
4
|
+
# Collects which tables received a write (INSERT/UPDATE/DELETE) inside a
|
|
5
|
+
# block, by subscribing to ActiveSupport::Notifications "sql.active_record".
|
|
6
|
+
#
|
|
7
|
+
# This only narrows down which tables are worth snapshotting — it never
|
|
8
|
+
# decides idempotency on its own. That matters: if the SQL parsing below
|
|
9
|
+
# reports a table that wasn't really written to, the cost is one extra
|
|
10
|
+
# table snapshotted, not a wrong answer.
|
|
11
|
+
class SqlTracker
|
|
12
|
+
IGNORED_TABLES = %w[schema_migrations ar_internal_metadata].freeze
|
|
13
|
+
|
|
14
|
+
class << self
|
|
15
|
+
def track(&block)
|
|
16
|
+
new.track(&block)
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def track
|
|
21
|
+
tables = []
|
|
22
|
+
|
|
23
|
+
subscriber = ActiveSupport::Notifications.subscribe('sql.active_record') do |*args|
|
|
24
|
+
event = ActiveSupport::Notifications::Event.new(*args)
|
|
25
|
+
table = written_table(event.payload[:sql].to_s)
|
|
26
|
+
tables << table if table
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
yield
|
|
30
|
+
|
|
31
|
+
tables.uniq
|
|
32
|
+
ensure
|
|
33
|
+
ActiveSupport::Notifications.unsubscribe(subscriber) if subscriber
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
private
|
|
37
|
+
|
|
38
|
+
def written_table(sql)
|
|
39
|
+
operation = sql[/\A\s*(INSERT|UPDATE|DELETE)/i, 1]
|
|
40
|
+
return unless operation
|
|
41
|
+
|
|
42
|
+
table = sql[/\A\s*#{operation}\s+(?:INTO\s+|FROM\s+)?["`\[]?(\w+)/i, 1]
|
|
43
|
+
return if table.nil? || IGNORED_TABLES.include?(table)
|
|
44
|
+
|
|
45
|
+
table
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module IdempotencyCheck
|
|
4
|
+
# Snapshots row counts per table, so two snapshots can be compared to see
|
|
5
|
+
# whether running something again changed the resulting state.
|
|
6
|
+
#
|
|
7
|
+
# `tables` is the set of tables the checked block actually wrote to — see
|
|
8
|
+
# SqlTracker. Snapshotting only those keeps a check proportional to what
|
|
9
|
+
# the job touches instead of to the size of the whole schema.
|
|
10
|
+
class TableSnapshot
|
|
11
|
+
class << self
|
|
12
|
+
def take(tables:)
|
|
13
|
+
connection = ActiveRecord::Base.connection
|
|
14
|
+
|
|
15
|
+
tables.each_with_object({}) do |table, counts|
|
|
16
|
+
counts[table] = connection.select_value(
|
|
17
|
+
"SELECT COUNT(*) FROM #{connection.quote_table_name(table)}"
|
|
18
|
+
).to_i
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative 'idempotency_check/version'
|
|
4
|
+
require_relative 'idempotency_check/sql_tracker'
|
|
5
|
+
require_relative 'idempotency_check/table_snapshot'
|
|
6
|
+
require_relative 'idempotency_check/row_snapshot'
|
|
7
|
+
require_relative 'idempotency_check/http_tracker'
|
|
8
|
+
require_relative 'idempotency_check/mailer_snapshot'
|
|
9
|
+
require_relative 'idempotency_check/active_job_perform_tracker'
|
|
10
|
+
require_relative 'idempotency_check/active_job_snapshot'
|
|
11
|
+
require_relative 'idempotency_check/checker'
|
|
12
|
+
|
|
13
|
+
module IdempotencyCheck
|
|
14
|
+
def self.idempotent?(**opts, &block)
|
|
15
|
+
Checker.idempotent?(**opts, &block)
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def self.register_snapshot_source(name, source)
|
|
19
|
+
Checker.register_snapshot_source(name, source)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
private_constant :Checker
|
|
23
|
+
end
|
metadata
ADDED
|
@@ -0,0 +1,161 @@
|
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
|
2
|
+
name: idempotency_check
|
|
3
|
+
version: !ruby/object:Gem::Version
|
|
4
|
+
version: 0.1.0
|
|
5
|
+
platform: ruby
|
|
6
|
+
authors:
|
|
7
|
+
- Agustin Fornio
|
|
8
|
+
bindir: bin
|
|
9
|
+
cert_chain: []
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
11
|
+
dependencies:
|
|
12
|
+
- !ruby/object:Gem::Dependency
|
|
13
|
+
name: actionmailer
|
|
14
|
+
requirement: !ruby/object:Gem::Requirement
|
|
15
|
+
requirements:
|
|
16
|
+
- - ">="
|
|
17
|
+
- !ruby/object:Gem::Version
|
|
18
|
+
version: '0'
|
|
19
|
+
type: :development
|
|
20
|
+
prerelease: false
|
|
21
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
22
|
+
requirements:
|
|
23
|
+
- - ">="
|
|
24
|
+
- !ruby/object:Gem::Version
|
|
25
|
+
version: '0'
|
|
26
|
+
- !ruby/object:Gem::Dependency
|
|
27
|
+
name: activejob
|
|
28
|
+
requirement: !ruby/object:Gem::Requirement
|
|
29
|
+
requirements:
|
|
30
|
+
- - ">="
|
|
31
|
+
- !ruby/object:Gem::Version
|
|
32
|
+
version: '0'
|
|
33
|
+
type: :development
|
|
34
|
+
prerelease: false
|
|
35
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
36
|
+
requirements:
|
|
37
|
+
- - ">="
|
|
38
|
+
- !ruby/object:Gem::Version
|
|
39
|
+
version: '0'
|
|
40
|
+
- !ruby/object:Gem::Dependency
|
|
41
|
+
name: activerecord
|
|
42
|
+
requirement: !ruby/object:Gem::Requirement
|
|
43
|
+
requirements:
|
|
44
|
+
- - ">="
|
|
45
|
+
- !ruby/object:Gem::Version
|
|
46
|
+
version: '0'
|
|
47
|
+
type: :development
|
|
48
|
+
prerelease: false
|
|
49
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
50
|
+
requirements:
|
|
51
|
+
- - ">="
|
|
52
|
+
- !ruby/object:Gem::Version
|
|
53
|
+
version: '0'
|
|
54
|
+
- !ruby/object:Gem::Dependency
|
|
55
|
+
name: rake
|
|
56
|
+
requirement: !ruby/object:Gem::Requirement
|
|
57
|
+
requirements:
|
|
58
|
+
- - ">="
|
|
59
|
+
- !ruby/object:Gem::Version
|
|
60
|
+
version: '0'
|
|
61
|
+
type: :development
|
|
62
|
+
prerelease: false
|
|
63
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
64
|
+
requirements:
|
|
65
|
+
- - ">="
|
|
66
|
+
- !ruby/object:Gem::Version
|
|
67
|
+
version: '0'
|
|
68
|
+
- !ruby/object:Gem::Dependency
|
|
69
|
+
name: rspec
|
|
70
|
+
requirement: !ruby/object:Gem::Requirement
|
|
71
|
+
requirements:
|
|
72
|
+
- - ">="
|
|
73
|
+
- !ruby/object:Gem::Version
|
|
74
|
+
version: '0'
|
|
75
|
+
type: :development
|
|
76
|
+
prerelease: false
|
|
77
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
78
|
+
requirements:
|
|
79
|
+
- - ">="
|
|
80
|
+
- !ruby/object:Gem::Version
|
|
81
|
+
version: '0'
|
|
82
|
+
- !ruby/object:Gem::Dependency
|
|
83
|
+
name: sidekiq
|
|
84
|
+
requirement: !ruby/object:Gem::Requirement
|
|
85
|
+
requirements:
|
|
86
|
+
- - ">="
|
|
87
|
+
- !ruby/object:Gem::Version
|
|
88
|
+
version: '0'
|
|
89
|
+
type: :development
|
|
90
|
+
prerelease: false
|
|
91
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
92
|
+
requirements:
|
|
93
|
+
- - ">="
|
|
94
|
+
- !ruby/object:Gem::Version
|
|
95
|
+
version: '0'
|
|
96
|
+
- !ruby/object:Gem::Dependency
|
|
97
|
+
name: sqlite3
|
|
98
|
+
requirement: !ruby/object:Gem::Requirement
|
|
99
|
+
requirements:
|
|
100
|
+
- - ">="
|
|
101
|
+
- !ruby/object:Gem::Version
|
|
102
|
+
version: '0'
|
|
103
|
+
type: :development
|
|
104
|
+
prerelease: false
|
|
105
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
106
|
+
requirements:
|
|
107
|
+
- - ">="
|
|
108
|
+
- !ruby/object:Gem::Version
|
|
109
|
+
version: '0'
|
|
110
|
+
description: |
|
|
111
|
+
idempotency_check runs a block of code multiple times and compares the
|
|
112
|
+
side effects it produces (database rows, ActionMailer deliveries,
|
|
113
|
+
ActiveJob enqueue/perform activity, and optionally Sidekiq) to detect
|
|
114
|
+
jobs that aren't safe to retry or re-run. It works on any block of Ruby
|
|
115
|
+
code, not just jobs, and has no runtime dependencies of its own.
|
|
116
|
+
email:
|
|
117
|
+
- agustinfornio@gmail.com
|
|
118
|
+
executables: []
|
|
119
|
+
extensions: []
|
|
120
|
+
extra_rdoc_files: []
|
|
121
|
+
files:
|
|
122
|
+
- LICENSE.txt
|
|
123
|
+
- README.md
|
|
124
|
+
- lib/idempotency_check.rb
|
|
125
|
+
- lib/idempotency_check/active_job_perform_tracker.rb
|
|
126
|
+
- lib/idempotency_check/active_job_snapshot.rb
|
|
127
|
+
- lib/idempotency_check/checker.rb
|
|
128
|
+
- lib/idempotency_check/http_tracker.rb
|
|
129
|
+
- lib/idempotency_check/mailer_snapshot.rb
|
|
130
|
+
- lib/idempotency_check/row_snapshot.rb
|
|
131
|
+
- lib/idempotency_check/rspec.rb
|
|
132
|
+
- lib/idempotency_check/sidekiq.rb
|
|
133
|
+
- lib/idempotency_check/sql_tracker.rb
|
|
134
|
+
- lib/idempotency_check/table_snapshot.rb
|
|
135
|
+
- lib/idempotency_check/version.rb
|
|
136
|
+
homepage: https://github.com/AFornio/idempotency_check
|
|
137
|
+
licenses:
|
|
138
|
+
- MIT
|
|
139
|
+
metadata:
|
|
140
|
+
source_code_uri: https://github.com/AFornio/idempotency_check
|
|
141
|
+
bug_tracker_uri: https://github.com/AFornio/idempotency_check/issues
|
|
142
|
+
rubygems_mfa_required: 'true'
|
|
143
|
+
rdoc_options: []
|
|
144
|
+
require_paths:
|
|
145
|
+
- lib
|
|
146
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
|
147
|
+
requirements:
|
|
148
|
+
- - ">="
|
|
149
|
+
- !ruby/object:Gem::Version
|
|
150
|
+
version: '3.0'
|
|
151
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
152
|
+
requirements:
|
|
153
|
+
- - ">="
|
|
154
|
+
- !ruby/object:Gem::Version
|
|
155
|
+
version: '0'
|
|
156
|
+
requirements: []
|
|
157
|
+
rubygems_version: 4.0.16
|
|
158
|
+
specification_version: 4
|
|
159
|
+
summary: Detects whether a job (and eventually other units of work) is idempotent
|
|
160
|
+
by running it twice and comparing effects
|
|
161
|
+
test_files: []
|