n_plus_one_control 0.3.0 → 0.6.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 292888473b7ae70cfdc9c8e9324170dfaf1d7048fc27a8c33a3ef09ee7b3fc3a
4
- data.tar.gz: 99bf992bbe635524c8eaea3f6536f4813e05b4420bc1f26ecea8ab26fecfc0f1
3
+ metadata.gz: 48ad0338e6227c2b71a09254f3613db415d64fe5b40ee7df55ee1daddc46c0f5
4
+ data.tar.gz: c55895975614627ea2308dde866079bad496f5ddef81f295267f5b0d8b2670ac
5
5
  SHA512:
6
- metadata.gz: cd45056252d7ae15236e8ae5d5880c0bda50f741ac3cf2079a42f5870cf378fb52d791b444c57c25218428d52e046841a5a762708a18ad868103d9e17f0d30f1
7
- data.tar.gz: b17d7be7d184ffa37ce4f6162b694928eb96968cc5f4e05e42971d68954709ed33c43b36c5549d49823ac2578103d9420cf713dd4f6cbc12880e885203a87bb4
6
+ metadata.gz: b4f6b44d7ecccd6e3f53d65e4df916982b144b8381049ebec5659cdf5792cc9c1d1c7fdee22cb8610df155020220286cc2daf78a0dbe8b76fe6bceb778446c81
7
+ data.tar.gz: 63ac722a3b82c6c3263a79f0065f554352bf4ce2043bdec98f37139f8b21f441e9327bc575ff6e3bdde05e8eac95ce636615e74ed4e3415a0ae28312da43d2a7
@@ -0,0 +1,33 @@
1
+ ## 0.6.0 (2020-11-27)
2
+
3
+ - Fix table stats summary when queries use backticks to surround table names ([@andrewhampton][])
4
+ - Add support to test for linear query. ([@caalberts][])
5
+
6
+ ## 0.5.0 (2020-09-07)
7
+
8
+ - **Ruby 2.5+ is required**. ([@palkan][])
9
+
10
+ - Add support for multiple backtrace lines in verbose output. ([@palkan][])
11
+
12
+ Could be specified via `NPLUSONE_BACKTRACE` env var.
13
+
14
+ - Add `NPLUSONE_TRUNCATE` env var to truncate queries in verbose mode. ([@palkan][])
15
+
16
+ - Support passing default filter via `NPLUSONE_FILTER` env var. ([@palkan][])
17
+
18
+ - Add location tracing to SQLs in verbose mode. ([@palkan][])
19
+
20
+ ## 0.4.1 (2020-09-04)
21
+
22
+ - Enhance failure message by showing differences in table hits. ([@palkan][])
23
+
24
+ ## 0.4.0 (2020-07-20)
25
+
26
+ - Make scale factor available in tests via `#current_scale` method. ([@Earendil95][])
27
+
28
+ - Start keeping a changelog. ([@palkan][])
29
+
30
+ [@Earendil95]: https://github.com/Earendil95
31
+ [@palkan]: https://github.com/palkan
32
+ [@caalberts]: https://github.com/caalberts
33
+ [@andrewhampton]: https://github.com/andrewhampton
@@ -1,6 +1,6 @@
1
1
  The MIT License (MIT)
2
2
 
3
- Copyright (c) 2017 palkan
3
+ Copyright (c) 2017-2020 Vladimir Dementyev
4
4
 
5
5
  Permission is hereby granted, free of charge, to any person obtaining a copy
6
6
  of this software and associated documentation files (the "Software"), to deal
data/README.md CHANGED
@@ -1,4 +1,5 @@
1
- [![Gem Version](https://badge.fury.io/rb/n_plus_one_control.svg)](https://rubygems.org/gems/n_plus_one_control) [![Build Status](https://travis-ci.org/palkan/n_plus_one_control.svg?branch=master)](https://travis-ci.org/palkan/n_plus_one_control)
1
+ [![Gem Version](https://badge.fury.io/rb/n_plus_one_control.svg)](https://rubygems.org/gems/n_plus_one_control)
2
+ ![Build](https://github.com/palkan/n_plus_one_control/workflows/Build/badge.svg)
2
3
 
3
4
  # N + 1 Control
4
5
 
@@ -31,7 +32,7 @@ Add this line to your application's Gemfile:
31
32
 
32
33
  ```ruby
33
34
  group :test do
34
- gem 'n_plus_one_control'
35
+ gem "n_plus_one_control"
35
36
  end
36
37
  ```
37
38
 
@@ -47,8 +48,6 @@ First, add NPlusOneControl to your `spec_helper.rb`:
47
48
 
48
49
  ```ruby
49
50
  # spec_helper.rb
50
- ...
51
-
52
51
  require "n_plus_one_control/rspec"
53
52
  ```
54
53
 
@@ -86,10 +85,45 @@ Availables modifiers:
86
85
  ```ruby
87
86
  # You can specify the RegExp to filter queries.
88
87
  # By default, it only considers SELECT queries.
89
- expect { ... }.to perform_constant_number_of_queries.matching(/INSERT/)
88
+ expect { get :index }.to perform_constant_number_of_queries.matching(/INSERT/)
90
89
 
91
90
  # You can also provide custom scale factors
92
- expect { ... }.to perform_constant_number_of_queries.with_scale_factors(10, 100)
91
+ expect { get :index }.to perform_constant_number_of_queries.with_scale_factors(10, 100)
92
+ ```
93
+
94
+ #### Using scale factor in spec
95
+
96
+ Let's suppose your action accepts parameter, which can make impact on the number of returned records:
97
+
98
+ ```ruby
99
+ get :index, params: {per_page: 10}
100
+ ```
101
+
102
+ Then it is enough to just change `per_page` parameter between executions and do not recreate records in DB. For this purpose, you can use `current_scale` method in your example:
103
+
104
+ ```ruby
105
+ context "N+1", :n_plus_one do
106
+ before { create_list :post, 3 }
107
+
108
+ specify do
109
+ expect { get :index, params: {per_page: current_scale} }.to perform_constant_number_of_queries
110
+ end
111
+ end
112
+ ```
113
+
114
+ #### Other available matchers
115
+
116
+ `perform_linear_number_of_queries(slope: 1)` allows you to test that a query generates linear number of queries with the given slope.
117
+
118
+ ```ruby
119
+ context "when has linear query", :n_plus_one do
120
+ populate { |n| create_list(:post, n) }
121
+
122
+ specify do
123
+ expect { Post.find_each { |p| p.user.name } }
124
+ .to perform_linear_number_of_queries(slope: 1)
125
+ end
126
+ end
93
127
  ```
94
128
 
95
129
  ### Minitest
@@ -98,8 +132,6 @@ First, add NPlusOneControl to your `test_helper.rb`:
98
132
 
99
133
  ```ruby
100
134
  # test_helper.rb
101
- ...
102
-
103
135
  require "n_plus_one_control/minitest"
104
136
  ```
105
137
 
@@ -115,6 +147,18 @@ def test_no_n_plus_one_error
115
147
  end
116
148
  ```
117
149
 
150
+ You can also use `assert_perform_linear_number_of_queries` to test for linear queries:
151
+
152
+ ```ruby
153
+ def test_no_n_plus_one_error
154
+ populate = ->(n) { create_list(:post, n) }
155
+
156
+ assert_perform_linear_number_of_queries(slope: 1, populate: populate) do
157
+ Post.find_each { |p| p.user.name }
158
+ end
159
+ end
160
+ ```
161
+
118
162
  You can also specify custom scale factors or filter patterns:
119
163
 
120
164
  ```ruby
@@ -133,6 +177,12 @@ assert_perform_constant_number_of_queries(
133
177
  end
134
178
  ```
135
179
 
180
+ It's possible to specify a filter via `NPLUSONE_FILTER` env var, e.g.:
181
+
182
+ ```ruby
183
+ NPLUSONE_FILTER = users bundle exec rake test
184
+ ```
185
+
136
186
  You can also specify `populate` as a test class instance method:
137
187
 
138
188
  ```ruby
@@ -145,6 +195,17 @@ def test_no_n_plus_one_error
145
195
  get :index
146
196
  end
147
197
  end
198
+
199
+ ```
200
+
201
+ As in RSpec, you can use `current_scale` factor instead of `populate` block:
202
+
203
+ ```ruby
204
+ def test_no_n_plus_one_error
205
+ assert_perform_constant_number_of_queries do
206
+ get :index, params: {per_page: current_scale}
207
+ end
208
+ end
148
209
  ```
149
210
 
150
211
  ### With caching
@@ -156,9 +217,9 @@ If you use caching you can face the problem when first request performs more DB
156
217
 
157
218
  context "N + 1", :n_plus_one do
158
219
  populate { |n| create_list :post, n }
159
-
220
+
160
221
  warmup { get :index } # cache something must be cached
161
-
222
+
162
223
  specify do
163
224
  expect { get :index }.to perform_constant_number_of_queries
164
225
  end
@@ -185,7 +246,7 @@ end
185
246
  def test_no_n_plus_one
186
247
  populate = ->(n) { create_list(:post, n) }
187
248
  warmup = -> { get :index }
188
-
249
+
189
250
  assert_perform_constant_number_of_queries population: populate, warmup: warmup do
190
251
  get :index
191
252
  end
@@ -193,6 +254,7 @@ end
193
254
  ```
194
255
 
195
256
  If your `warmup` and testing procs are identical, you can use:
257
+
196
258
  ```ruby
197
259
  expext { get :index }.to perform_constant_number_of_queries.with_warming_up # RSpec only
198
260
  ```
@@ -210,13 +272,21 @@ NPlusOneControl.default_scale_factors = [2, 3]
210
272
  # You can activate verbosity through env variable NPLUSONE_VERBOSE=1
211
273
  NPlusOneControl.verbose = false
212
274
 
275
+ # Print table hits difference, for example:
276
+ #
277
+ # Unmatched query numbers by tables:
278
+ # users (SELECT): 2 != 3
279
+ # events (INSERT): 1 != 2
280
+ #
281
+ self.show_table_stats = true
282
+
213
283
  # Ignore matching queries
214
284
  NPlusOneControl.ignore = /^(BEGIN|COMMIT|SAVEPOINT|RELEASE)/
215
285
 
216
286
  # ActiveSupport notifications event to track queries.
217
287
  # We track ActiveRecord event by default,
218
288
  # but can also track rom-rb events ('sql.rom') as well.
219
- NPlusOneControl.event = 'sql.active_record'
289
+ NPlusOneControl.event = "sql.active_record"
220
290
 
221
291
  # configure transactional behavour for populate method
222
292
  # in case of use multiple database connections
@@ -230,6 +300,21 @@ NPlusOneControl::Executor.tap do |executor|
230
300
  connections.each(&:rollback_transaction)
231
301
  end
232
302
  end
303
+
304
+ # Provide a backtrace cleaner callable object used to filter SQL caller location to display in the verbose mode
305
+ # Set it to nil to disable tracing.
306
+ #
307
+ # In Rails apps, we use Rails.backtrace_cleaner by default.
308
+ NPlusOneControl.backtrace_cleaner = ->(locations_array) { do_some_filtering(locations_array) }
309
+
310
+ # You can also specify the number of backtrace lines to show.
311
+ # MOTE: It could be specified via NPLUSONE_BACKTRACE env var
312
+ NPlusOneControl.backtrace_length = 1
313
+
314
+ # Sometime queries could be too large to provide any meaningful insight.
315
+ # You can configure an output length limit for quries in verbose mode by setting the follwing option
316
+ # NOTE: It could be specified via NPLUSONE_TRUNCATE env var
317
+ NPlusOneControl.truncate_query_size = 100
233
318
  ```
234
319
 
235
320
  ## How does it work?
@@ -238,22 +323,29 @@ Take a look at our [Executor](https://github.com/palkan/n_plus_one_control/blob/
238
323
 
239
324
  ## What's next?
240
325
 
326
+ - More matchers.
327
+
241
328
  It may be useful to provide more matchers/assertions, for example:
242
329
 
243
330
  ```ruby
244
331
 
245
332
  # Actually, that means that it is N+1))
246
- assert_linear_number_of_queries { ... }
333
+ assert_linear_number_of_queries { some_code }
247
334
 
248
335
  # But we can tune it with `coef` and handle such cases as selecting in batches
249
336
  assert_linear_number_of_queries(coef: 0.1) do
250
- Post.find_in_batches { ... }
337
+ Post.find_in_batches { some_code }
251
338
  end
252
339
 
253
340
  # probably, also make sense to add another curve types
254
- assert_logarithmic_number_of_queries { ... }
341
+ assert_logarithmic_number_of_queries { some_code }
255
342
  ```
256
343
 
344
+ - Support custom non-SQL events.
345
+
346
+ N+1 problem is not a database specific: we can have N+1 Redis calls, N+1 HTTP external requests, etc.
347
+ We can make `n_plus_one_control` customizable to support these scenarios (technically, we need to make it possible to handle different payload in the event subscriber).
348
+
257
349
  If you want to discuss or implement any of these, feel free to open an [issue](https://github.com/palkan/n_plus_one_control/issues) or propose a [pull request](https://github.com/palkan/n_plus_one_control/pulls).
258
350
 
259
351
  ## Development
@@ -5,17 +5,99 @@ require "n_plus_one_control/executor"
5
5
 
6
6
  # RSpec and Minitest matchers to prevent N+1 queries problem.
7
7
  module NPlusOneControl
8
+ # Used to extract a table name from a query
9
+ EXTRACT_TABLE_RXP = /(insert into|update|delete from|from) ['"`](\S+)['"`]/i.freeze
10
+
11
+ # Used to convert a query part extracted by the regexp above to the corresponding
12
+ # human-friendly type
13
+ QUERY_PART_TO_TYPE = {
14
+ "insert into" => "INSERT",
15
+ "update" => "UPDATE",
16
+ "delete from" => "DELETE",
17
+ "from" => "SELECT"
18
+ }.freeze
19
+
8
20
  class << self
9
- attr_accessor :default_scale_factors, :verbose, :ignore, :event
21
+ attr_accessor :default_scale_factors, :verbose, :show_table_stats, :ignore, :event,
22
+ :backtrace_cleaner, :backtrace_length, :truncate_query_size
23
+
24
+ attr_reader :default_matching
25
+
26
+ FAILURE_MESSAGES = {
27
+ constant_queries: "Expected to make the same number of queries",
28
+ linear_queries: "Expected to make linear number of queries"
29
+ }
10
30
 
11
- def failure_message(queries)
12
- msg = ["Expected to make the same number of queries, but got:\n"]
31
+ def failure_message(type, queries) # rubocop:disable Metrics/MethodLength
32
+ msg = ["#{FAILURE_MESSAGES[type]}, but got:\n"]
13
33
  queries.each do |(scale, data)|
14
34
  msg << " #{data.size} for N=#{scale}\n"
15
- msg << data.map { |sql| " #{sql}\n" }.join.to_s if verbose
16
35
  end
36
+
37
+ msg.concat(table_usage_stats(queries.map(&:last))) if show_table_stats
38
+
39
+ if verbose
40
+ queries.each do |(scale, data)|
41
+ msg << "Queries for N=#{scale}\n"
42
+ msg << data.map { |sql| " #{truncate_query(sql)}\n" }.join.to_s
43
+ end
44
+ end
45
+
17
46
  msg.join
18
47
  end
48
+
49
+ def table_usage_stats(runs) # rubocop:disable Metrics/MethodLength
50
+ msg = ["Unmatched query numbers by tables:\n"]
51
+
52
+ before, after = runs.map do |queries|
53
+ queries.group_by do |query|
54
+ matches = query.match(EXTRACT_TABLE_RXP)
55
+ next unless matches
56
+
57
+ " #{matches[2]} (#{QUERY_PART_TO_TYPE[matches[1].downcase]})"
58
+ end.transform_values(&:count)
59
+ end
60
+
61
+ before.keys.each do |k|
62
+ next if before[k] == after[k]
63
+
64
+ msg << "#{k}: #{before[k]} != #{after[k]}\n"
65
+ end
66
+
67
+ msg
68
+ end
69
+
70
+ def default_matching=(val)
71
+ unless val
72
+ @default_matching = nil
73
+ return
74
+ end
75
+
76
+ @default_matching =
77
+ if val.is_a?(Regexp)
78
+ val
79
+ else
80
+ Regexp.new(val, Regexp::MULTILINE | Regexp::IGNORECASE)
81
+ end
82
+ end
83
+
84
+ private
85
+
86
+ def truncate_query(sql)
87
+ return sql unless truncate_query_size
88
+
89
+ # Only truncate query, leave tracing (if any) as is
90
+ parts = sql.split(/(\s+↳)/)
91
+
92
+ parts[0] =
93
+ if truncate_query_size < 4
94
+ "..."
95
+ else
96
+ parts[0][0..(truncate_query_size - 4)] + "..."
97
+ end
98
+
99
+ parts.join
100
+ end
19
101
  end
20
102
 
21
103
  # Scale factors to use.
@@ -23,7 +105,10 @@ module NPlusOneControl
23
105
  self.default_scale_factors = [2, 3]
24
106
 
25
107
  # Print performed queries if true
26
- self.verbose = ENV['NPLUSONE_VERBOSE'] == '1'
108
+ self.verbose = ENV["NPLUSONE_VERBOSE"] == "1"
109
+
110
+ # Print table hits difference
111
+ self.show_table_stats = true
27
112
 
28
113
  # Ignore matching queries
29
114
  self.ignore = /^(BEGIN|COMMIT|SAVEPOINT|RELEASE)/
@@ -31,5 +116,16 @@ module NPlusOneControl
31
116
  # ActiveSupport notifications event to track queries.
32
117
  # We track ActiveRecord event by default,
33
118
  # but can also track rom-rb events ('sql.rom') as well.
34
- self.event = 'sql.active_record'
119
+ self.event = "sql.active_record"
120
+
121
+ # Default query filtering applied if none provided explicitly
122
+ self.default_matching = ENV["NPLUSONE_FILTER"] || /^SELECT/i
123
+
124
+ # Truncate queries in verbose mode to fit the length
125
+ self.truncate_query_size = ENV["NPLUSONE_TRUNCATE"]&.to_i
126
+
127
+ # Define the number of backtrace lines to show
128
+ self.backtrace_length = ENV.fetch("NPLUSONE_BACKTRACE", 1).to_i
35
129
  end
130
+
131
+ require "n_plus_one_control/railtie" if defined?(Rails::Railtie)
@@ -3,7 +3,7 @@
3
3
  module NPlusOneControl
4
4
  # Runs code for every scale factor
5
5
  # and returns collected queries.
6
- module Executor
6
+ class Executor
7
7
  # Subscribes to ActiveSupport notifications and collect matching queries.
8
8
  class Collector
9
9
  def initialize(pattern)
@@ -19,46 +19,86 @@ module NPlusOneControl
19
19
  @queries
20
20
  end
21
21
 
22
- def callback(_name, _start, _finish, _message_id, values)
22
+ def callback(_name, _start, _finish, _message_id, values) # rubocop:disable Metrics/CyclomaticComplexity,Metrics/LineLength
23
23
  return if %w[CACHE SCHEMA].include? values[:name]
24
- @queries << values[:sql] if @pattern.nil? || (values[:sql] =~ @pattern)
25
- end
26
- end
27
24
 
28
- class << self
29
- attr_accessor :transaction_begin
30
- attr_accessor :transaction_rollback
25
+ return unless @pattern.nil? || (values[:sql] =~ @pattern)
31
26
 
32
- def call(population:, scale_factors: nil, matching: nil)
33
- raise ArgumentError, "Block is required!" unless block_given?
27
+ query = values[:sql]
34
28
 
35
- results = []
36
- collector = Collector.new(matching)
29
+ if NPlusOneControl.backtrace_cleaner && NPlusOneControl.verbose
30
+ source = extract_query_source_location(caller)
37
31
 
38
- (scale_factors || NPlusOneControl.default_scale_factors).each do |scale|
39
- with_transaction do
40
- population.call(scale)
41
- results << [scale, collector.call { yield }]
42
- end
32
+ query = "#{query}\n ↳ #{source.join("\n")}" unless source.empty?
43
33
  end
44
- results
34
+
35
+ @queries << query
45
36
  end
46
37
 
47
38
  private
48
39
 
49
- def with_transaction
50
- transaction_begin.call
51
- yield
52
- ensure
53
- transaction_rollback.call
40
+ def extract_query_source_location(locations)
41
+ NPlusOneControl.backtrace_cleaner.call(locations.lazy)
42
+ .take(NPlusOneControl.backtrace_length).to_a
54
43
  end
55
44
  end
56
45
 
46
+ class << self
47
+ attr_accessor :transaction_begin
48
+ attr_accessor :transaction_rollback
49
+ end
50
+
51
+ attr_reader :current_scale
52
+
57
53
  self.transaction_begin = -> do
58
54
  ActiveRecord::Base.connection.begin_transaction(joinable: false)
59
55
  end
56
+
60
57
  self.transaction_rollback = -> do
61
58
  ActiveRecord::Base.connection.rollback_transaction
62
59
  end
60
+
61
+ def initialize(population: nil, scale_factors: nil, matching: nil)
62
+ @population = population
63
+ @scale_factors = scale_factors
64
+ @matching = matching
65
+ end
66
+
67
+ # rubocop:disable Metrics/MethodLength
68
+ def call
69
+ raise ArgumentError, "Block is required!" unless block_given?
70
+
71
+ results = []
72
+ collector = Collector.new(matching)
73
+
74
+ (scale_factors || NPlusOneControl.default_scale_factors).each do |scale|
75
+ @current_scale = scale
76
+ with_transaction do
77
+ population&.call(scale)
78
+ results << [scale, collector.call { yield }]
79
+ end
80
+ end
81
+ results
82
+ end
83
+ # rubocop:enable Metrics/MethodLength
84
+
85
+ private
86
+
87
+ def with_transaction
88
+ transaction_begin.call
89
+ yield
90
+ ensure
91
+ transaction_rollback.call
92
+ end
93
+
94
+ def transaction_begin
95
+ self.class.transaction_begin
96
+ end
97
+
98
+ def transaction_rollback
99
+ self.class.transaction_rollback
100
+ end
101
+
102
+ attr_reader :population, :scale_factors, :matching
63
103
  end
64
104
  end