prosopite 2.1.2 → 2.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +169 -0
- data/README.md +406 -183
- data/lib/generators/prosopite/install_generator.rb +53 -0
- data/lib/generators/prosopite/templates/prosopite.rb.tt +18 -0
- data/lib/prosopite/fixes.rb +181 -0
- data/lib/prosopite/middleware/rack.rb +212 -2
- data/lib/prosopite/middleware/sidekiq.rb +2 -2
- data/lib/prosopite/minitest.rb +23 -0
- data/lib/prosopite/rspec.rb +20 -0
- data/lib/prosopite/version.rb +1 -1
- data/lib/prosopite.rb +478 -95
- metadata +22 -24
- data/.github/workflows/ci.yml +0 -15
- data/.gitignore +0 -8
- data/Gemfile +0 -8
- data/Gemfile.lock +0 -152
- data/Rakefile +0 -9
- data/prosopite.gemspec +0 -34
data/README.md
CHANGED
|
@@ -1,139 +1,110 @@
|
|
|
1
1
|
# Prosopite  [](https://badge.fury.io/rb/prosopite)
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
<picture>
|
|
4
|
+
<source media="(prefers-reduced-motion: reduce)" srcset="docs/demo-still.png">
|
|
5
|
+
<img src="docs/demo.webp" width="840" alt="A Rails page runs 9 queries. Prosopite flags the N+1 in a footer and suggests Chair.includes(:legs) on line 2 of the view. After the fix, the page runs 2 queries.">
|
|
6
|
+
</picture>
|
|
4
7
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
SELECT `users`.* FROM `users` WHERE `users`.`id` = 21 LIMIT 1
|
|
9
|
-
SELECT `users`.* FROM `users` WHERE `users`.`id` = 22 LIMIT 1
|
|
10
|
-
SELECT `users`.* FROM `users` WHERE `users`.`id` = 23 LIMIT 1
|
|
11
|
-
SELECT `users`.* FROM `users` WHERE `users`.`id` = 24 LIMIT 1
|
|
12
|
-
Call stack:
|
|
13
|
-
app/controllers/thank_you_controller.rb:4:in `block in index'
|
|
14
|
-
app/controllers/thank_you_controller.rb:3:in `each'
|
|
15
|
-
app/controllers/thank_you_controller.rb:3:in `index':
|
|
16
|
-
app/controllers/application_controller.rb:8:in `block in <class:ApplicationController>'
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
The need for prosopite emerged after dealing with various false positives / negatives using the
|
|
20
|
-
[bullet](https://github.com/flyerhzm/bullet) gem.
|
|
21
|
-
|
|
22
|
-
## Compared to Bullet
|
|
23
|
-
|
|
24
|
-
Prosopite can auto-detect the following extra cases of N+1 queries:
|
|
8
|
+
Prosopite auto-detects N+1 queries in Rails apps, with zero false positives and false negatives. It finds every
|
|
9
|
+
query that repeats from the same place in your code, not only the lazy loading of associations, and suggests how
|
|
10
|
+
to fix it:
|
|
25
11
|
|
|
26
|
-
#### N+1 queries after record creations (usually in tests)
|
|
27
|
-
|
|
28
|
-
```ruby
|
|
29
|
-
FactoryBot.create_list(:leg, 10)
|
|
30
|
-
|
|
31
|
-
Leg.last(10).each do |l|
|
|
32
|
-
l.chair
|
|
33
|
-
end
|
|
34
12
|
```
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
Chair.
|
|
41
|
-
|
|
13
|
+
N+1 queries detected (1.2ms):
|
|
14
|
+
SELECT "legs".* FROM "legs" WHERE "legs"."chair_id" = $1 ORDER BY "legs"."id" ASC LIMIT $2
|
|
15
|
+
SELECT "legs".* FROM "legs" WHERE "legs"."chair_id" = $1 ORDER BY "legs"."id" ASC LIMIT $2
|
|
16
|
+
SELECT "legs".* FROM "legs" WHERE "legs"."chair_id" = $1 ORDER BY "legs"."id" ASC LIMIT $2
|
|
17
|
+
Possible fix:
|
|
18
|
+
Chair.includes(:legs) where the chairs are loaded, at app/views/chairs/index.html.erb:2
|
|
19
|
+
Context:
|
|
20
|
+
GET /chairs
|
|
21
|
+
Call stack:
|
|
22
|
+
app/views/chairs/index.html.erb:3
|
|
23
|
+
app/views/chairs/index.html.erb:2
|
|
42
24
|
```
|
|
43
25
|
|
|
44
|
-
|
|
26
|
+
* [Quick start](#quick-start)
|
|
27
|
+
* [How it works](#how-it-works)
|
|
28
|
+
* [Usage](#usage): [development](#development), [tests](#tests), [jobs](#jobs), [production](#production),
|
|
29
|
+
[any code](#scanning-any-code)
|
|
30
|
+
* [Reports](#reports): [loggers](#loggers), [fix suggestions](#fix-suggestions),
|
|
31
|
+
[structured notifications](#structured-notifications), [CI](#ci)
|
|
32
|
+
* [Ignoring N+1 queries](#ignoring-n1-queries)
|
|
33
|
+
* [Configuration](#configuration)
|
|
34
|
+
* Compared to [Bullet](#compared-to-bullet) and [strict_loading](#compared-to-strict_loading)
|
|
45
35
|
|
|
46
|
-
|
|
47
|
-
Chair.last(20).each do |c|
|
|
48
|
-
c.legs.first
|
|
49
|
-
c.legs.last
|
|
50
|
-
c.legs.pluck(:id)
|
|
51
|
-
end
|
|
52
|
-
```
|
|
36
|
+
## Quick start
|
|
53
37
|
|
|
54
|
-
|
|
38
|
+
Add Prosopite to your Gemfile:
|
|
55
39
|
|
|
56
40
|
```ruby
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
end
|
|
41
|
+
gem 'prosopite'
|
|
42
|
+
gem 'pg_query' # Optional, with PostgreSQL
|
|
60
43
|
```
|
|
61
44
|
|
|
62
|
-
|
|
45
|
+
With PostgreSQL, [pg_query](https://github.com/pganalyze/pg_query) makes the fingerprints of queries more precise,
|
|
46
|
+
but it's a native extension that doesn't install everywhere, and Prosopite works without it.
|
|
63
47
|
|
|
64
|
-
|
|
65
|
-
class Leg::Design
|
|
66
|
-
include Mongoid::Document
|
|
67
|
-
...
|
|
68
|
-
field :cid, as: :chair_id, type: Integer
|
|
69
|
-
...
|
|
70
|
-
def chair
|
|
71
|
-
@chair ||= Chair.where(id: chair_id).first!
|
|
72
|
-
end
|
|
73
|
-
end
|
|
48
|
+
Then install it and set it up:
|
|
74
49
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
end
|
|
78
|
-
```
|
|
50
|
+
$ bundle install
|
|
51
|
+
$ bin/rails generate prosopite:install
|
|
79
52
|
|
|
80
|
-
|
|
53
|
+
The generator creates `config/initializers/prosopite.rb`, which shows N+1 queries on the pages in development and
|
|
54
|
+
raises for them in tests, and it scans each test. N+1 queries also go to the Rails log, unless another logger is set.
|
|
55
|
+
The sections below show how to set it up yourself.
|
|
81
56
|
|
|
82
|
-
|
|
83
|
-
different from prosopite's.
|
|
57
|
+
Prosopite needs Ruby 2.7+ and Rails 6.1+. The footer on pages needs Rails 7.1+, and the scanning of fibers Ruby 3.2+.
|
|
84
58
|
|
|
85
59
|
## How it works
|
|
86
60
|
|
|
87
|
-
Prosopite
|
|
88
|
-
|
|
61
|
+
Prosopite subscribes to the SQL queries of Active Record. In a scan, e.g. of a request, a job or a test, it reports
|
|
62
|
+
the queries that run more than once from the same call stack and have the same fingerprint, i.e. that are the same
|
|
63
|
+
query apart from its values. Queries on different databases, e.g. on shards, never count together.
|
|
89
64
|
|
|
90
|
-
|
|
65
|
+
## Usage
|
|
91
66
|
|
|
92
|
-
|
|
67
|
+
### Development
|
|
93
68
|
|
|
94
|
-
|
|
69
|
+
The Rack middleware scans every request:
|
|
95
70
|
|
|
96
71
|
```ruby
|
|
97
|
-
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
If you're **not** using MySQL/MariaDB, you should also add:
|
|
72
|
+
# config/initializers/prosopite.rb
|
|
101
73
|
|
|
102
|
-
|
|
103
|
-
|
|
74
|
+
if Rails.env.development?
|
|
75
|
+
require 'prosopite/middleware/rack'
|
|
76
|
+
Rails.configuration.middleware.use(Prosopite::Middleware::Rack)
|
|
77
|
+
end
|
|
104
78
|
```
|
|
105
79
|
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
$ bundle install
|
|
109
|
-
|
|
110
|
-
Or install it yourself as:
|
|
80
|
+
N+1 queries go to the Rails log, with the method and the path of the request as their context, e.g. `GET /chairs`.
|
|
81
|
+
In tests, the scan of each test covers its requests instead, see [Tests](#tests).
|
|
111
82
|
|
|
112
|
-
|
|
83
|
+
To see the N+1 queries of a page on the page itself, in a footer and/or the browser console:
|
|
113
84
|
|
|
114
|
-
|
|
85
|
+
```ruby
|
|
86
|
+
# config/environments/development.rb
|
|
115
87
|
|
|
116
|
-
|
|
88
|
+
config.after_initialize do
|
|
89
|
+
Prosopite.add_footer = true
|
|
90
|
+
Prosopite.browser_console = true
|
|
91
|
+
end
|
|
92
|
+
```
|
|
117
93
|
|
|
118
|
-
|
|
119
|
-
* `Prosopite.raise = true`: Raise warnings as exceptions. Defaults to `false`.
|
|
120
|
-
* `Prosopite.start_raise`: Raises warnings as exceptions from when this is called. Overrides `Proposite.raise`.
|
|
121
|
-
* `Propsoite.stop_raise`: Disables raising warnings as exceptions if previously enabled with `Proposite.start_raise`.
|
|
122
|
-
* `Prosopite.local_raise?`: Returns `true` if `Prosopite.start_raise` has been called previously.
|
|
123
|
-
* `Prosopite.rails_logger = true`: Send warnings to the Rails log. Defaults to `false`.
|
|
124
|
-
* `Prosopite.prosopite_logger = true`: Send warnings to `log/prosopite.log`. Defaults to `false`.
|
|
125
|
-
* `Prosopite.stderr_logger = true`: Send warnings to STDERR. Defaults to `false`.
|
|
126
|
-
* `Prosopite.backtrace_cleaner = my_custom_backtrace_cleaner`: use a different [ActiveSupport::BacktraceCleaner](https://api.rubyonrails.org/classes/ActiveSupport/BacktraceCleaner.html). Defaults to `Rails.backtrace_cleaner`.
|
|
127
|
-
* `Prosopite.custom_logger = my_custom_logger`: Set a custom logger. See the following section for the details. Defaults to `false`.
|
|
128
|
-
* `Prosopite.enabled = true`: Enables or disables the gem. Defaults to `true`.
|
|
94
|
+

|
|
129
95
|
|
|
130
|
-
|
|
96
|
+
They also get the N+1 queries of later requests, like the ones of Turbo frames and streams or of other fetch and
|
|
97
|
+
XHR requests, which send them in an `x-prosopite-notifications` header. Requests that redirect, like form
|
|
98
|
+
submissions, can't show theirs, which go to the log only. The footer and the console need Rails 7.1+, show only when
|
|
99
|
+
`Prosopite.raise` is off, and follow your Content Security Policy: they use its nonce, if any, and leave out a style
|
|
100
|
+
or a script that it would block.
|
|
131
101
|
|
|
132
|
-
|
|
102
|
+
Instead of the middleware, an `around_action` can scan the actions of controllers. Don't use a `before_action` and an
|
|
103
|
+
`after_action` for this: an exception skips the `after_action`, leaving the scan open into the next request.
|
|
133
104
|
|
|
134
105
|
```ruby
|
|
135
106
|
class ApplicationController < ActionController::Base
|
|
136
|
-
|
|
107
|
+
if Rails.env.development?
|
|
137
108
|
around_action :n_plus_one_detection
|
|
138
109
|
|
|
139
110
|
def n_plus_one_detection
|
|
@@ -145,64 +116,99 @@ class ApplicationController < ActionController::Base
|
|
|
145
116
|
end
|
|
146
117
|
end
|
|
147
118
|
```
|
|
148
|
-
|
|
119
|
+
|
|
120
|
+
### Tests
|
|
121
|
+
|
|
122
|
+
To make the tests with N+1 queries fail, raise for them and scan each test:
|
|
149
123
|
|
|
150
124
|
```ruby
|
|
151
|
-
# config/environments/
|
|
125
|
+
# config/environments/test.rb
|
|
152
126
|
|
|
153
127
|
config.after_initialize do
|
|
154
|
-
Prosopite.
|
|
128
|
+
Prosopite.raise = true
|
|
129
|
+
end
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
```ruby
|
|
133
|
+
# spec/rails_helper.rb
|
|
134
|
+
|
|
135
|
+
RSpec.configure do |config|
|
|
136
|
+
config.before { |example| Prosopite.scan(example.location) unless example.metadata[:prosopite] == false }
|
|
137
|
+
config.after { Prosopite.finish }
|
|
155
138
|
end
|
|
156
139
|
```
|
|
140
|
+
|
|
141
|
+
```ruby
|
|
142
|
+
# test/test_helper.rb
|
|
143
|
+
|
|
144
|
+
class ActiveSupport::TestCase
|
|
145
|
+
setup { Prosopite.scan("#{self.class}##{name}") }
|
|
146
|
+
teardown { Prosopite.finish }
|
|
147
|
+
end
|
|
157
148
|
```
|
|
158
|
-
## Test Environment Usage
|
|
159
149
|
|
|
160
|
-
|
|
150
|
+
Each test is a scan of its own, with its location or name as the context, and specs can opt out with
|
|
151
|
+
`prosopite: false`. Don't scan the whole suite, as that would count the queries of different tests together.
|
|
152
|
+
|
|
153
|
+
The N+1 queries of [FactoryBot](https://github.com/thoughtbot/factory_bot) factories are usually only test setup.
|
|
154
|
+
To leave them out, pause the scans while factories run:
|
|
161
155
|
|
|
162
156
|
```ruby
|
|
163
157
|
# config/environments/test.rb
|
|
164
158
|
|
|
165
159
|
config.after_initialize do
|
|
166
|
-
Prosopite.
|
|
167
|
-
Prosopite.raise = true
|
|
160
|
+
Prosopite.pause_during('factory_bot.run_factory')
|
|
168
161
|
end
|
|
169
162
|
```
|
|
170
163
|
|
|
171
|
-
|
|
164
|
+
#### Assertions
|
|
172
165
|
|
|
173
|
-
|
|
174
|
-
# spec/spec_helper.rb
|
|
166
|
+
To check specific code for N+1 queries, or to check that it has some, use the RSpec matcher:
|
|
175
167
|
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
168
|
+
```ruby
|
|
169
|
+
# spec/rails_helper.rb
|
|
170
|
+
require 'prosopite/rspec'
|
|
179
171
|
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
end
|
|
172
|
+
# in a spec
|
|
173
|
+
expect { get :index }.not_to have_n_plus_one_queries
|
|
183
174
|
```
|
|
184
175
|
|
|
185
|
-
|
|
176
|
+
or the Minitest assertions:
|
|
186
177
|
|
|
187
|
-
|
|
178
|
+
```ruby
|
|
179
|
+
# test/test_helper.rb
|
|
180
|
+
require 'prosopite/minitest'
|
|
188
181
|
|
|
189
|
-
|
|
182
|
+
# in a test
|
|
183
|
+
assert_no_n_plus_one_queries { get chairs_url }
|
|
184
|
+
assert_n_plus_one_queries { Chair.all.each { |chair| chair.legs.first } }
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
The block runs in a scan of its own, so its N+1 queries are reported only by the assertion, even when each test
|
|
188
|
+
is scanned too. `Prosopite.n_plus_one_queries { ... }` returns them for any other use.
|
|
190
189
|
|
|
191
|
-
|
|
192
|
-
implementing auto detect for all controllers.
|
|
190
|
+
#### Adopting Prosopite gradually
|
|
193
191
|
|
|
194
|
-
|
|
192
|
+
To start using Prosopite on an app that already has N+1 queries, record them in a todo file. Its N+1 queries are
|
|
193
|
+
then ignored, while new ones are reported:
|
|
195
194
|
|
|
196
195
|
```ruby
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
196
|
+
# config/environments/test.rb
|
|
197
|
+
|
|
198
|
+
config.after_initialize do
|
|
199
|
+
Prosopite.todo_file = Rails.root.join('.prosopite_todo.yml')
|
|
200
|
+
Prosopite.record_todo = ENV['PROSOPITE_RECORD_TODO'].present?
|
|
200
201
|
end
|
|
201
202
|
```
|
|
202
203
|
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
204
|
+
$ PROSOPITE_RECORD_TODO=1 bin/rails test
|
|
205
|
+
|
|
206
|
+
Entries match by query and file, so remove the ones of the N+1 queries you fix, or delete the file and record it
|
|
207
|
+
again. The assertions above don't use the todo file.
|
|
208
|
+
|
|
209
|
+
### Jobs
|
|
210
|
+
|
|
211
|
+
The Sidekiq middleware, for Sidekiq 6.5+, scans every job, with its class, ID and queue as context:
|
|
206
212
|
|
|
207
213
|
```ruby
|
|
208
214
|
Sidekiq.configure_server do |config|
|
|
@@ -215,99 +221,107 @@ Sidekiq.configure_server do |config|
|
|
|
215
221
|
end
|
|
216
222
|
```
|
|
217
223
|
|
|
218
|
-
|
|
224
|
+
Other jobs can be scanned with a callback, e.g. for Active Job:
|
|
225
|
+
|
|
219
226
|
```ruby
|
|
220
|
-
|
|
221
|
-
|
|
227
|
+
class ApplicationJob < ActiveJob::Base
|
|
228
|
+
around_perform do |job, block|
|
|
229
|
+
Prosopite.scan(job.class.name) { block.call }
|
|
230
|
+
end
|
|
222
231
|
end
|
|
223
232
|
```
|
|
224
233
|
|
|
225
|
-
|
|
234
|
+
### Production
|
|
226
235
|
|
|
227
|
-
|
|
236
|
+
To find N+1 queries in production with little overhead, scan a sample of the requests and jobs, and report only
|
|
237
|
+
the N+1 queries that take long enough to matter:
|
|
228
238
|
|
|
229
239
|
```ruby
|
|
230
|
-
|
|
231
|
-
```
|
|
240
|
+
# config/environments/production.rb
|
|
232
241
|
|
|
233
|
-
|
|
242
|
+
require 'prosopite/middleware/rack'
|
|
234
243
|
|
|
235
|
-
|
|
236
|
-
Prosopite
|
|
244
|
+
Rails.application.configure do
|
|
245
|
+
config.middleware.use(Prosopite::Middleware::Rack)
|
|
246
|
+
|
|
247
|
+
config.after_initialize do
|
|
248
|
+
Prosopite.sample_rate = 0.01 # Scan 1% of the requests and jobs
|
|
249
|
+
Prosopite.min_duration_ms = 50
|
|
250
|
+
end
|
|
251
|
+
end
|
|
237
252
|
```
|
|
238
253
|
|
|
239
|
-
|
|
254
|
+
### Scanning any code
|
|
240
255
|
|
|
241
|
-
|
|
256
|
+
Wrap the code in a scan:
|
|
242
257
|
|
|
243
258
|
```ruby
|
|
244
259
|
Prosopite.scan
|
|
245
|
-
<code to scan>
|
|
260
|
+
# <code to scan>
|
|
246
261
|
Prosopite.finish
|
|
247
262
|
```
|
|
248
263
|
|
|
249
|
-
In block form
|
|
264
|
+
In block form, `Prosopite.finish` runs at the end of the block, and the scan returns the result of the block:
|
|
250
265
|
|
|
251
266
|
```ruby
|
|
252
|
-
Prosopite.scan do
|
|
253
|
-
|
|
267
|
+
my_object = Prosopite.scan do
|
|
268
|
+
MyObjectFactory.create(params)
|
|
254
269
|
end
|
|
255
270
|
```
|
|
256
271
|
|
|
257
|
-
|
|
272
|
+
A context passed to `Prosopite.scan` is added to the notifications, to tell where they came from:
|
|
258
273
|
|
|
259
274
|
```ruby
|
|
260
|
-
|
|
261
|
-
|
|
275
|
+
Prosopite.scan("ImportJob #{import.id}") do
|
|
276
|
+
# <code to scan>
|
|
262
277
|
end
|
|
263
278
|
```
|
|
264
279
|
|
|
265
|
-
|
|
280
|
+
### Pausing scans
|
|
266
281
|
|
|
267
|
-
|
|
282
|
+
To leave out code that has N+1 queries, pause the scan:
|
|
268
283
|
|
|
269
284
|
```ruby
|
|
270
285
|
Prosopite.scan
|
|
271
286
|
# <code to scan>
|
|
272
287
|
Prosopite.pause
|
|
273
|
-
# <code that has
|
|
288
|
+
# <code that has N+1 queries>
|
|
274
289
|
Prosopite.resume
|
|
275
290
|
# <code to scan>
|
|
276
291
|
Prosopite.finish
|
|
277
292
|
```
|
|
278
293
|
|
|
279
|
-
|
|
280
|
-
for you automatically:
|
|
294
|
+
or pause it for a block, which can be nested:
|
|
281
295
|
|
|
282
296
|
```ruby
|
|
283
|
-
Prosopite.scan
|
|
284
|
-
# <code to scan>
|
|
285
|
-
|
|
286
297
|
result = Prosopite.pause do
|
|
287
|
-
# <code that has
|
|
298
|
+
# <code that has N+1 queries>
|
|
288
299
|
end
|
|
289
|
-
|
|
290
|
-
Prosopite.finish
|
|
291
300
|
```
|
|
292
301
|
|
|
293
|
-
|
|
302
|
+
or pause it while the code instrumented as an Active Support event runs, e.g. the factories of FactoryBot:
|
|
294
303
|
|
|
295
|
-
|
|
296
|
-
|
|
304
|
+
```ruby
|
|
305
|
+
Prosopite.pause_during('factory_bot.run_factory')
|
|
306
|
+
```
|
|
297
307
|
|
|
298
|
-
|
|
308
|
+
`Prosopite.finish` ends a scan even while it's paused, and reports the N+1 queries found before the pause, while
|
|
309
|
+
`Prosopite.scan` doesn't start another scan, so the pause applies to its code too. With
|
|
310
|
+
`Prosopite.ignore_pauses = true`, pauses don't apply, e.g. to report all the N+1 queries for a while.
|
|
299
311
|
|
|
300
|
-
|
|
301
|
-
|
|
312
|
+
For example, when [testing Active Jobs inline](https://guides.rubyonrails.org/testing.html#testing-jobs), an
|
|
313
|
+
[Active Job callback](https://guides.rubyonrails.org/active_job_basics.html#callbacks) can pause the scan while a
|
|
314
|
+
job runs, to check only the code of the app.
|
|
302
315
|
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
316
|
+
### Raising in specific places
|
|
317
|
+
|
|
318
|
+
To raise only for some code, e.g. some controller actions, instead of setting `Prosopite.raise`, call
|
|
319
|
+
`Prosopite.start_raise` and `Prosopite.stop_raise` around it. `Prosopite.local_raise?` tells whether it's on.
|
|
306
320
|
|
|
307
321
|
```ruby
|
|
308
322
|
# app/controllers/application_controller.rb
|
|
309
323
|
class ApplicationController < ActionController::Base
|
|
310
|
-
def raise_on_n_plus_ones!(**options)
|
|
324
|
+
def self.raise_on_n_plus_ones!(**options)
|
|
311
325
|
return if Rails.env.production?
|
|
312
326
|
|
|
313
327
|
prepend_around_action(:_raise_on_n_plus_ones, **options)
|
|
@@ -317,11 +331,14 @@ class ApplicationController < ActionController::Base
|
|
|
317
331
|
around_action :n_plus_one_detection
|
|
318
332
|
|
|
319
333
|
def n_plus_one_detection
|
|
320
|
-
|
|
334
|
+
Prosopite.scan
|
|
335
|
+
yield
|
|
336
|
+
ensure
|
|
337
|
+
Prosopite.finish
|
|
321
338
|
end
|
|
322
339
|
|
|
323
340
|
def _raise_on_n_plus_ones
|
|
324
|
-
|
|
341
|
+
Prosopite.start_raise
|
|
325
342
|
yield
|
|
326
343
|
ensure
|
|
327
344
|
Prosopite.stop_raise
|
|
@@ -329,43 +346,249 @@ class ApplicationController < ActionController::Base
|
|
|
329
346
|
end
|
|
330
347
|
end
|
|
331
348
|
```
|
|
349
|
+
|
|
332
350
|
```ruby
|
|
333
351
|
# app/controllers/books_controller.rb
|
|
334
352
|
class BooksController < ApplicationController
|
|
335
353
|
raise_on_n_plus_ones!(only: [:index])
|
|
336
354
|
|
|
337
355
|
def index
|
|
338
|
-
@books = Book.all.map(&:author) # This
|
|
356
|
+
@books = Book.all.map(&:author) # This raises for the N+1 queries
|
|
339
357
|
end
|
|
340
358
|
|
|
341
359
|
def show
|
|
342
360
|
@book = Book.find(params[:id])
|
|
343
|
-
@book.reviews.map(&:author) # This
|
|
361
|
+
@book.reviews.map(&:author) # This only logs them
|
|
344
362
|
end
|
|
345
363
|
end
|
|
364
|
+
```
|
|
365
|
+
|
|
366
|
+
### Fibers and threads
|
|
367
|
+
|
|
368
|
+
On Ruby 3.2+, a scan also covers the fibers spawned while it runs, e.g. [Async](https://github.com/socketry/async)
|
|
369
|
+
tasks, unless they start a scan of their own. The queries of `ActionController::Live` actions count towards the scan
|
|
370
|
+
of their request too, while the ones of other threads don't. Pausing a scan pauses it in all of them.
|
|
371
|
+
|
|
372
|
+
## Reports
|
|
373
|
+
|
|
374
|
+
### Loggers
|
|
375
|
+
|
|
376
|
+
N+1 queries are reported to any of:
|
|
377
|
+
|
|
378
|
+
```ruby
|
|
379
|
+
Prosopite.raise = true # Raise Prosopite::NPlusOneQueriesError
|
|
380
|
+
Prosopite.rails_logger = true # The Rails log, in red
|
|
381
|
+
Prosopite.stderr_logger = true # STDERR, in red
|
|
382
|
+
Prosopite.prosopite_logger = true # log/prosopite.log, or a path or a logger
|
|
383
|
+
Prosopite.custom_logger = logger # Any logger, e.g. Rails.logger for the Rails log without the red
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
When no logger is set, they go to the Rails log, unless `Prosopite.rails_logger = false`.
|
|
387
|
+
|
|
388
|
+
The Rails backtrace cleaner leaves out the frames of gems, including engines. To see them, e.g. when developing an
|
|
389
|
+
engine, set another cleaner:
|
|
390
|
+
|
|
391
|
+
```ruby
|
|
392
|
+
Prosopite.backtrace_cleaner = ActiveSupport::BacktraceCleaner.new
|
|
393
|
+
```
|
|
394
|
+
|
|
395
|
+
To show fewer queries of each N+1, e.g. when they're many, and their count instead:
|
|
396
|
+
|
|
397
|
+
```ruby
|
|
398
|
+
Prosopite.display_max_n_queries = 3
|
|
399
|
+
Prosopite.display_query_count = true
|
|
400
|
+
```
|
|
401
|
+
|
|
402
|
+
### Fix suggestions
|
|
403
|
+
|
|
404
|
+
Notifications suggest how to fix an N+1, from the associations whose queries look like its own, and show where the
|
|
405
|
+
query that loaded its records ran, like in the example at the top. If that query was built elsewhere, e.g.
|
|
406
|
+
`@chairs = Chair.order(:name)` in a controller, add the `includes` there. For `COUNT` queries, they suggest a counter
|
|
407
|
+
cache too.
|
|
408
|
+
|
|
409
|
+
Several associations can run the same query, so check which one your code uses. To turn them off, set
|
|
410
|
+
`Prosopite.suggest_fixes = false`.
|
|
411
|
+
|
|
412
|
+
### Structured notifications
|
|
413
|
+
|
|
414
|
+
`Prosopite.finish` returns the N+1 queries it found. Each one is also published as an `n_plus_one_queries.prosopite`
|
|
415
|
+
event through [Active Support Instrumentation](https://guides.rubyonrails.org/active_support_instrumentation.html),
|
|
416
|
+
whether a logger is configured or not, with these keys:
|
|
417
|
+
|
|
418
|
+
| Key | Value |
|
|
419
|
+
| -------------- | ---------------------------------------------------------- |
|
|
420
|
+
| `:queries` | The SQL queries of the N+1 |
|
|
421
|
+
| `:call_stack` | The call stack, cleaned with `Prosopite.backtrace_cleaner` |
|
|
422
|
+
| `:duration_ms` | The total time of the queries in milliseconds |
|
|
423
|
+
| `:fixes` | Suggested fixes, see [Fix suggestions](#fix-suggestions) |
|
|
424
|
+
| `:fingerprint` | The fingerprint of the queries |
|
|
425
|
+
| `:context` | The context passed to `Prosopite.scan`, if any |
|
|
426
|
+
|
|
427
|
+
For example, to log them as JSON for Datadog, the ELK stack or similar:
|
|
428
|
+
|
|
429
|
+
```ruby
|
|
430
|
+
Prosopite.rails_logger = false # Instead of the text ones
|
|
431
|
+
|
|
432
|
+
ActiveSupport::Notifications.subscribe('n_plus_one_queries.prosopite') do |event|
|
|
433
|
+
Rails.logger.warn(event.payload.to_json)
|
|
434
|
+
end
|
|
435
|
+
```
|
|
436
|
+
|
|
437
|
+
### CI
|
|
438
|
+
|
|
439
|
+
```ruby
|
|
440
|
+
Prosopite.github_annotations = true # Show each N+1 on its line of code in GitHub Actions
|
|
441
|
+
Prosopite.json_report = 'tmp/prosopite.jsonl' # Add each N+1 to a file as a line of JSON, e.g. for coding agents
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
## Ignoring N+1 queries
|
|
445
|
+
|
|
446
|
+
Ignore the N+1 queries whose call stack contains one or more substrings or regexes, e.g. of specific files or
|
|
447
|
+
directories:
|
|
448
|
+
|
|
449
|
+
```ruby
|
|
450
|
+
Prosopite.allow_stack_paths = ['app/models/legacy_report.rb', %r{lib/generated/}]
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
Report only the N+1 queries whose call stack contains one of them, e.g. to fail tests for the N+1 queries of the app
|
|
454
|
+
code, but not of the test code itself. `allow_stack_paths` takes precedence:
|
|
455
|
+
|
|
456
|
+
```ruby
|
|
457
|
+
Prosopite.focus_stack_paths = [Rails.root.join('app').to_s]
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
Ignore the N+1 queries matching a regex, or equal to a string:
|
|
461
|
+
|
|
462
|
+
```ruby
|
|
463
|
+
Prosopite.ignore_queries = [/regex_match/, 'SELECT * from EXACT_STRING_MATCH']
|
|
464
|
+
```
|
|
465
|
+
|
|
466
|
+
Some gems run code through different files on the first and the later calls of a method, e.g. `sorbet-runtime`
|
|
467
|
+
(`call_validation.rb` and `call_validation_2_7.rb`), so the queries of an N+1 get different call stacks and aren't
|
|
468
|
+
counted together. Leave such frames out of the call stacks that Prosopite compares:
|
|
469
|
+
|
|
470
|
+
```ruby
|
|
471
|
+
Prosopite.ignore_location_paths = ['sorbet-runtime', /observability/]
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
The notifications still show the full call stack.
|
|
475
|
+
|
|
476
|
+
## Configuration
|
|
477
|
+
|
|
478
|
+
| Reporting | Default | Description |
|
|
479
|
+
| ---------------------------- | ------- | ------------------------------------------------------------------------ |
|
|
480
|
+
| `Prosopite.raise` | `false` | Raise `Prosopite::NPlusOneQueriesError` |
|
|
481
|
+
| `Prosopite.rails_logger` | when no other logger is set | Log to the Rails log |
|
|
482
|
+
| `Prosopite.stderr_logger` | `false` | Log to STDERR |
|
|
483
|
+
| `Prosopite.prosopite_logger` | `false` | Log to `log/prosopite.log`, or to a path or a logger |
|
|
484
|
+
| `Prosopite.custom_logger` | none | Log to this logger |
|
|
485
|
+
| `Prosopite.add_footer` | `false` | Show them in a footer of pages, with the [Rack middleware](#development) |
|
|
486
|
+
| `Prosopite.browser_console` | `false` | Show them in the browser console, with the Rack middleware |
|
|
487
|
+
| `Prosopite.github_annotations` | `false` | Show them on the code in GitHub Actions |
|
|
488
|
+
| `Prosopite.json_report` | none | Add them to this file as lines of JSON |
|
|
489
|
+
|
|
490
|
+
| Messages | Default | Description |
|
|
491
|
+
| --------------------------------- | ------------------------- | ------------------------------------------ |
|
|
492
|
+
| `Prosopite.suggest_fixes` | `true` | Suggest fixes |
|
|
493
|
+
| `Prosopite.display_max_n_queries` | all | Show at most this many queries of each N+1 |
|
|
494
|
+
| `Prosopite.display_query_count` | `false` | Show the number of queries of each N+1 |
|
|
495
|
+
| `Prosopite.backtrace_cleaner` | `Rails.backtrace_cleaner` | Clean the call stacks |
|
|
496
|
+
|
|
497
|
+
| Scanning | Default | Description |
|
|
498
|
+
| --------------------------- | ------- | --------------------------------------------------------------- |
|
|
499
|
+
| `Prosopite.enabled` | `true` | Scan at all |
|
|
500
|
+
| `Prosopite.min_n_queries` | `2` | Report N+1s of at least this many queries |
|
|
501
|
+
| `Prosopite.min_duration_ms` | none | Report N+1s whose queries take at least this long in total |
|
|
502
|
+
| `Prosopite.sample_rate` | all | Run only this share of the scans, from 0 to 1 |
|
|
503
|
+
| `Prosopite.ignore_pauses` | `false` | Scan [paused code](#pausing-scans) too |
|
|
504
|
+
|
|
505
|
+
| Ignoring | Default | Description |
|
|
506
|
+
| --------------------------------- | ------- | --------------------------------------------------------------- |
|
|
507
|
+
| `Prosopite.allow_stack_paths` | none | Ignore N+1s from these places, see [Ignoring N+1 queries](#ignoring-n1-queries) |
|
|
508
|
+
| `Prosopite.focus_stack_paths` | none | Report only N+1s from these places |
|
|
509
|
+
| `Prosopite.ignore_queries` | none | Ignore these queries |
|
|
510
|
+
| `Prosopite.ignore_location_paths` | none | Leave these frames out of the call stacks that are compared |
|
|
511
|
+
| `Prosopite.todo_file` | none | Ignore the N+1s in this file, see [Adopting Prosopite gradually](#adopting-prosopite-gradually) |
|
|
512
|
+
| `Prosopite.record_todo` | `false` | Add the N+1s to the todo file instead of reporting them |
|
|
513
|
+
|
|
514
|
+
## Compared to Bullet
|
|
515
|
+
|
|
516
|
+
Prosopite started after dealing with false positives and false negatives of [bullet](https://github.com/flyerhzm/bullet),
|
|
517
|
+
whose core mechanism is completely different: bullet tracks the loading of Active Record associations, while
|
|
518
|
+
Prosopite looks at the SQL queries themselves. This way it detects these N+1 queries too:
|
|
519
|
+
|
|
520
|
+
#### After creating records, usually in tests
|
|
521
|
+
|
|
522
|
+
```ruby
|
|
523
|
+
FactoryBot.create_list(:leg, 10)
|
|
524
|
+
|
|
525
|
+
Leg.last(10).each do |l|
|
|
526
|
+
l.chair
|
|
527
|
+
end
|
|
528
|
+
```
|
|
346
529
|
|
|
347
|
-
|
|
530
|
+
#### Not run by associations
|
|
348
531
|
|
|
349
|
-
|
|
532
|
+
```ruby
|
|
533
|
+
Leg.last(4).each do |l|
|
|
534
|
+
Chair.find(l.chair_id)
|
|
535
|
+
end
|
|
536
|
+
```
|
|
350
537
|
|
|
351
|
-
|
|
352
|
-
highlighted with red, or you want logs sent to a custom location.
|
|
538
|
+
#### First, last and pluck of collection associations
|
|
353
539
|
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
540
|
+
```ruby
|
|
541
|
+
Chair.last(20).each do |c|
|
|
542
|
+
c.legs.first
|
|
543
|
+
c.legs.last
|
|
544
|
+
c.legs.pluck(:id)
|
|
545
|
+
end
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
#### After changing the class of records with becomes
|
|
358
549
|
|
|
359
550
|
```ruby
|
|
360
|
-
|
|
361
|
-
|
|
551
|
+
Chair.last(20).map { |c| c.becomes(ArmChair) }.each do |ac|
|
|
552
|
+
ac.legs.map(&:id)
|
|
553
|
+
end
|
|
362
554
|
```
|
|
363
555
|
|
|
556
|
+
#### Of Mongoid models calling Active Record
|
|
557
|
+
|
|
364
558
|
```ruby
|
|
365
|
-
|
|
366
|
-
|
|
559
|
+
class Leg::Design
|
|
560
|
+
include Mongoid::Document
|
|
561
|
+
...
|
|
562
|
+
field :cid, as: :chair_id, type: Integer
|
|
563
|
+
...
|
|
564
|
+
def chair
|
|
565
|
+
@chair ||= Chair.where(id: chair_id).first!
|
|
566
|
+
end
|
|
567
|
+
end
|
|
568
|
+
|
|
569
|
+
Leg::Design.last(20).each do |l|
|
|
570
|
+
l.chair
|
|
571
|
+
end
|
|
367
572
|
```
|
|
368
573
|
|
|
574
|
+
On the other hand, bullet also detects eager loading that isn't used, and suggests counter caches for counting
|
|
575
|
+
associations. The first needs to track how associations are loaded and used, which queries alone can't show, and is
|
|
576
|
+
known for false positives. Prosopite suggests counter caches for N+1s of `COUNT` queries.
|
|
577
|
+
|
|
578
|
+
## Compared to strict_loading
|
|
579
|
+
|
|
580
|
+
Rails' [`strict_loading`](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-strict_loading)
|
|
581
|
+
prevents the lazy loading of associations, which is a different trade-off:
|
|
582
|
+
|
|
583
|
+
* It raises on every lazy load of a strict record, even a single one that isn't part of an N+1, so enabling it on an
|
|
584
|
+
existing app means fixing all of them first. Its `n_plus_one_only` mode raises only for records loaded through a
|
|
585
|
+
`has_many` association of a strict record, so it misses the N+1 queries on the records of a plain query.
|
|
586
|
+
* It doesn't see N+1 queries that don't lazy load an association, like calling `Chair.find(leg.chair_id)`,
|
|
587
|
+
`chair.legs.count` or `chair.legs.exists?` in a loop.
|
|
588
|
+
|
|
589
|
+
Prosopite reports only the queries that actually repeat, whatever runs them, without changing your models.
|
|
590
|
+
The two can also be combined, e.g. with `strict_loading` on the models you have already fixed.
|
|
591
|
+
|
|
369
592
|
## Contributing
|
|
370
593
|
|
|
371
594
|
Bug reports and pull requests are welcome on GitHub at https://github.com/charkost/prosopite.
|