rails-minify 0.1.3 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +16 -0
- data/README.md +262 -54
- data/exe/rails-minify-ab +56 -0
- data/lib/rails_minify/ab.rb +154 -0
- data/lib/rails_minify/async_queries.rb +218 -0
- data/lib/rails_minify/doctor.rb +220 -0
- data/lib/rails_minify/railtie.rb +96 -14
- data/lib/rails_minify/sql_profile.rb +236 -0
- data/lib/rails_minify/version.rb +1 -1
- data/lib/rails_minify.rb +24 -7
- metadata +20 -32
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: c682f7b5f509613697d35d78e163167b825525a03aae99fecb6ab430550f2429
|
|
4
|
+
data.tar.gz: dd258711680a2472583815a96af871f418d8fac5cf06f323c2dc7f102908e962
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 29a12e198162b3bd148f39d2f7be79e69aa32d471164120d1055a8c2960a97f7afee2a1dd2728b30dfe6e8f2d8f22b528a854420600f3ae1d3e325077265ce60
|
|
7
|
+
data.tar.gz: c2031ba5b96903be93f11d324afe736b18c476f420467e1a48503b4ac397ee24540812e7173819aa1ff6fd97e5025c7ac865616f7ad15d0db0498de29f2144eb
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,21 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.3.0 (2026-11-04)
|
|
4
|
+
|
|
5
|
+
- `config.rails_minify.sql_profile = 0.01` samples requests and logs each distinct query and call site with its calls, time and rows, plus the request's SQL and instantiation time; `bin/rails minify:sql:report LOG=...` ranks actions and queries by total time, and with `EXPLAIN=1` adds SQLite query plans and flags full table scans and sorts without an index.
|
|
6
|
+
- `config.rails_minify.async_queries = 0.01` samples requests for relations the controller leaves unloaded and the view then loads one after another, and logs one line per request; `bin/rails minify:async:report LOG=...` ranks actions by the estimated saving from starting those relations with `load_async`. `minify:doctor` also reports `load_async` without `config.active_record.async_query_executor`.
|
|
7
|
+
- A/B in production, under the same traffic: `rails-minify-ab COMMAND` runs the server as groups A and B on one port (`SO_REUSEPORT`) with half the workers each when `RAILS_MINIFY_AB` is set; `config.rails_minify.ab` changes group B's configuration; both groups' log lines are tagged; `minify:ab:report` compares the groups from a request log (with B−A and a 95% interval) and `minify:ab:memory` reports their memory from `/proc`.
|
|
8
|
+
- `bin/rails minify:doctor` reports production configuration that costs speed or memory, and never fails: a JIT turned on before boot (or JIT options without `--zjit-disable`/`--yjit-disable`), Falcon without `preload` or Puma with `preload_app! false`, a Falcon preload script without `Process.warmup` on async-service 0.25 and earlier, `WEB_CONCURRENCY` that `falcon.rb` doesn't read, and Sentry breadcrumbs on every notification. It reads the environment from `config/deploy.yml`, the `Dockerfile` and `bin/docker-entrypoint` too.
|
|
9
|
+
|
|
10
|
+
## 0.2.0 (2026-10-20)
|
|
11
|
+
|
|
12
|
+
Narrowed to what held up in production (Japan Voyage, each part on half the workers under the same traffic; README, "What paid off in production").
|
|
13
|
+
|
|
14
|
+
- `config.rails_minify.enabled` turns on only strict mode (`ar_minimizer.slim`), enums and ZJIT after boot. ZJIT only in production and from Ruby 4.1 (4.0's ZJIT isn't meant for production); set `config.rails_minify.zjit` yourself otherwise.
|
|
15
|
+
- No longer turned on by `enabled`: `ar_minimizer.views`, `.load_time`, `.small_records`, `as_minimizer.unroll_callbacks`, `ap_minimizer.route_helpers`, `.views`, and the patches. Each still works when set on its own and is documented as experimental. Patches: `config.rails_minify.patches = true` for all, or a list.
|
|
16
|
+
- actionpack-minimizer is no longer a dependency; rails-minify loads it, and `minify:census` covers it, when it's in the Gemfile.
|
|
17
|
+
- Requires activerecord-minimizer and activesupport-minimizer 0.2.
|
|
18
|
+
|
|
3
19
|
## 0.1.3 (2026-10-18)
|
|
4
20
|
|
|
5
21
|
- `config.rails_minify.zjit = true` turns ZJIT on after boot, as `config.yjit` does for YJIT, instead of `RUBY_ZJIT_ENABLE=1` compiling boot-only code too. Japan Voyage, per process: 24% fewer compiled methods, 18 MB less JIT memory, about 25 MB less RSS, the same speed.
|
data/README.md
CHANGED
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# rails-minify
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Fewer generated Active Record methods, ZJIT turned on after boot, and checks
|
|
4
|
+
for callback problems.
|
|
4
5
|
|
|
5
6
|
```ruby
|
|
6
7
|
# Gemfile
|
|
@@ -10,14 +11,19 @@ gem "rails-minify"
|
|
|
10
11
|
config.rails_minify.enabled = true
|
|
11
12
|
```
|
|
12
13
|
|
|
13
|
-
`enabled` turns on
|
|
14
|
-
|
|
14
|
+
`enabled` turns on what held up in production on a real app:
|
|
15
|
+
|
|
16
|
+
* strict mode (`config.ar_minimizer.slim`): only readers and writers per column;
|
|
17
|
+
* enum predicates and bang methods as plain methods (`config.ar_minimizer.enums`);
|
|
18
|
+
* ZJIT after boot (`config.rails_minify.zjit`), in production on Ruby 4.1+.
|
|
19
|
+
|
|
20
|
+
Each can still be set on its own (`config.ar_minimizer.enums = false`).
|
|
21
|
+
Everything else is [experimental](#experimental) and off unless you set it.
|
|
15
22
|
Ruby 3.4+, Rails 8.0, 8.1 and main.
|
|
16
23
|
|
|
17
24
|
Turn it on in tests too, so a call to a method strict mode removed fails in CI
|
|
18
25
|
rather than in production. In development, strict mode alone is enough: the
|
|
19
|
-
same errors show up while you work
|
|
20
|
-
follow code reloads, stay off.
|
|
26
|
+
same errors show up while you work.
|
|
21
27
|
|
|
22
28
|
```ruby
|
|
23
29
|
# config/environments/test.rb
|
|
@@ -27,7 +33,7 @@ config.rails_minify.enabled = true
|
|
|
27
33
|
config.ar_minimizer.slim = true # strict mode, without the rest
|
|
28
34
|
```
|
|
29
35
|
|
|
30
|
-
## Strict
|
|
36
|
+
## Strict mode
|
|
31
37
|
|
|
32
38
|
Active Record defines, for every column, a reader, a writer and 19 more
|
|
33
39
|
methods (`name?`, `name_changed?`, `saved_change_to_name?`, ...). With
|
|
@@ -46,50 +52,17 @@ rubocop-minify rewrites those calls in your code. For calls in gems
|
|
|
46
52
|
`config.ar_minimizer.strict = false` keeps the removed methods working
|
|
47
53
|
instead, generated the first time each is called.
|
|
48
54
|
|
|
49
|
-
## What's in it
|
|
50
|
-
|
|
51
|
-
| part | does | gem |
|
|
52
|
-
|---|---|---|
|
|
53
|
-
| attribute methods | readers and writers only (strict) | activerecord-minimizer |
|
|
54
|
-
| enums | predicates/bang methods as plain methods (1.3-1.6x) | activerecord-minimizer |
|
|
55
|
-
| small records | no Hashes for records that never fill them (-22% per record) | activerecord-minimizer |
|
|
56
|
-
| callbacks | chains as generated methods (`valid?` -33..45%) | activesupport-minimizer |
|
|
57
|
-
| route helpers | `post_path(post)` as interpolation (3-14x) | actionpack-minimizer |
|
|
58
|
-
| views | `render`/`link_to` without their machinery (2x per page of partials) | actionpack-minimizer |
|
|
59
|
-
| patches | below | rails-minify |
|
|
60
|
-
| rewrites | removed-method calls in Ruby and ERB, as RuboCop autocorrections | rubocop-minify (development) |
|
|
61
|
-
|
|
62
|
-
## Patches
|
|
63
|
-
|
|
64
|
-
For parts of Rails and common gems that redo work on every request.
|
|
65
|
-
`config.rails_minify.patches = [...]` picks them; all by default.
|
|
66
|
-
|
|
67
|
-
* `:action_text_plain_text`: `ActionText::Content#to_plain_text` parses the
|
|
68
|
-
HTML with Nokogiri, and looks up every attachment's record, on every call.
|
|
69
|
-
It's computed once per distinct HTML (cache capped at 8 MB of HTML).
|
|
70
|
-
Attachments that are Active Storage blobs, remote images or inline content
|
|
71
|
-
are cached too: a blob without a caption contributes its filename, which a
|
|
72
|
-
hit checks in one query (renamed or deleted: computed again). Other
|
|
73
|
-
attachables aren't cached. In [Japan Voyage](https://japanvoyage.jp), 13% of all allocations on its
|
|
74
|
-
pages came from it, and guides with images another 4.5%.
|
|
75
|
-
* `:request_host`: `ActionDispatch::Request#host` runs a regexp substitution
|
|
76
|
-
on every call, port or not, and Rails reads it several times per request.
|
|
77
|
-
rails/rails#59104 (Rails 8.2) for 8.1: no substitution without a port, and
|
|
78
|
-
the host kept while the raw host is the same String. Not installed on 8.2.
|
|
79
|
-
* `:omniauth`: every OmniAuth strategy middleware dups itself and its
|
|
80
|
-
options on every request before looking at the path. Requests that can't
|
|
81
|
-
be OmniAuth's go straight to the app. Only when that's certain up front:
|
|
82
|
-
a session is present, not in test mode, no `other_phase` or callable
|
|
83
|
-
paths, and the strategy doesn't override how paths are matched. Not
|
|
84
|
-
installed when OmniAuth already does this itself (it then has a public
|
|
85
|
-
`Strategy#passthrough?`, as in the version proposed upstream).
|
|
86
|
-
|
|
87
55
|
## ZJIT after boot
|
|
88
56
|
|
|
89
57
|
```ruby
|
|
90
58
|
config.rails_minify.zjit = true # instead of RUBY_ZJIT_ENABLE=1 / --zjit
|
|
91
59
|
```
|
|
92
60
|
|
|
61
|
+
`enabled` sets it in production on Ruby 4.1 and later; on 4.0, ZJIT isn't
|
|
62
|
+
meant for production yet, so set it yourself if you want it there. ZJIT
|
|
63
|
+
options still go in `RUBYOPT`, after `--zjit-disable` so they don't turn it
|
|
64
|
+
on at startup: `RUBYOPT="--zjit-disable --zjit-call-threshold=2"`.
|
|
65
|
+
|
|
93
66
|
`RUBY_ZJIT_ENABLE=1` turns ZJIT on before anything loads, so it also compiles
|
|
94
67
|
what runs only while booting: RubyGems, requiring and eager loading, and the
|
|
95
68
|
minimizers' own compiling. Rails turns YJIT on after boot (`config.yjit`) for
|
|
@@ -100,16 +73,33 @@ On Japan Voyage, per process after the same 132 pages: 12,612 → 9,557
|
|
|
100
73
|
compiled methods, JIT code 45.9 → 35.3 MB, ZJIT memory 82.9 → 64.6 MB,
|
|
101
74
|
RSS about 25 MB less, the same CPU time per page.
|
|
102
75
|
|
|
103
|
-
## Numbers
|
|
104
|
-
|
|
105
|
-
Japan Voyage, 132 production pages, with everything on (strict mode after
|
|
106
|
-
rubocop-minify's rewrites, all patches): identical HTML, and
|
|
107
|
-
1,561,283 → 1,224,141 allocations per pass (−21.6%). The machine was too
|
|
108
|
-
loaded for wall-clock times to mean much.
|
|
109
|
-
|
|
110
76
|
## Checks
|
|
111
77
|
|
|
112
|
-
* `bin/rails minify:
|
|
78
|
+
* `bin/rails minify:doctor`: production configuration that costs speed or
|
|
79
|
+
memory. It only reports, and never fails. Each finding says where it was
|
|
80
|
+
found, why it matters (with the number from Japan Voyage) and the fix:
|
|
81
|
+
* a JIT turned on before boot (`RUBY_ZJIT_ENABLE`, `RUBY_YJIT_ENABLE`,
|
|
82
|
+
`--zjit`/`--yjit` in `RUBYOPT`), or ZJIT/YJIT options without
|
|
83
|
+
`--zjit-disable`/`--yjit-disable`, which turn it on at startup too;
|
|
84
|
+
* Falcon with more than one worker and no `preload`, or a
|
|
85
|
+
`config/preload.rb` it doesn't load; Puma with `workers` and
|
|
86
|
+
`preload_app! false`;
|
|
87
|
+
* a Falcon preload script without `Process.warmup`, while async-service
|
|
88
|
+
(0.25 and earlier) warms the heap up only after forking the workers
|
|
89
|
+
(socketry/async-service#13);
|
|
90
|
+
* `WEB_CONCURRENCY` set while `falcon.rb` sets no `count`: Falcon starts
|
|
91
|
+
one worker per CPU;
|
|
92
|
+
* Sentry's `breadcrumbs_logger` with `:active_support_logger`;
|
|
93
|
+
* `load_async` in `app/` without `config.active_record.async_query_executor`:
|
|
94
|
+
the query then runs in the foreground, as if `load_async` weren't there.
|
|
95
|
+
|
|
96
|
+
Doctor usually runs on your machine rather than the server, so it reads the
|
|
97
|
+
environment from `config/deploy.yml` (Kamal's `env.clear`), the
|
|
98
|
+
`Dockerfile`'s `ENV` lines and `bin/docker-entrypoint`'s `export`s as well
|
|
99
|
+
as its own. Run on Japan Voyage's history, it finds the three problems
|
|
100
|
+
fixed there: ZJIT turned on in the entrypoint, the preload file Falcon
|
|
101
|
+
didn't load (and `WEB_CONCURRENCY` it didn't read), and the heap warmed up
|
|
102
|
+
after the fork.
|
|
113
103
|
* `bin/rails minify:check` (for CI; works whether or not the rest of
|
|
114
104
|
rails-minify is on) fails on:
|
|
115
105
|
* callbacks naming methods that don't exist;
|
|
@@ -121,7 +111,8 @@ loaded for wall-clock times to mean much.
|
|
|
121
111
|
reported once, with the classes that inherit it;
|
|
122
112
|
* model callbacks that call `save`, `update`, `touch`, ... on the record
|
|
123
113
|
they run for, which runs the chain again;
|
|
124
|
-
* an out-of-date manifest
|
|
114
|
+
* an out-of-date manifest, when there is one (only the experimental
|
|
115
|
+
compiling parts use it).
|
|
125
116
|
|
|
126
117
|
```ruby
|
|
127
118
|
config.rails_minify.callback_limit = 8
|
|
@@ -131,7 +122,224 @@ loaded for wall-clock times to mean much.
|
|
|
131
122
|
config.rails_minify.allow_callback_loops = ["Trip#recalculate"] # guarded on purpose
|
|
132
123
|
```
|
|
133
124
|
|
|
134
|
-
##
|
|
125
|
+
## Where Active Record time goes
|
|
126
|
+
|
|
127
|
+
```ruby
|
|
128
|
+
# config/environments/production.rb
|
|
129
|
+
config.rails_minify.sql_profile = 0.01 # 1% of requests; true for all
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
For each sampled request, one log line per distinct query and call site
|
|
133
|
+
(calls, time, rows, the first line in the app that ran it, the SQL with its
|
|
134
|
+
values taken out) and one with the request's totals, SQL time against the
|
|
135
|
+
time spent turning rows into records. `bin/rails minify:sql:report
|
|
136
|
+
LOG=production.log` ranks actions and queries by total time; with
|
|
137
|
+
`EXPLAIN=1` it loads the app and adds each SELECT's SQLite query plan,
|
|
138
|
+
flagging full table scans and sorts without an index:
|
|
139
|
+
|
|
140
|
+
```
|
|
141
|
+
5. 5.2% of SQL time TravelDestination Pluck 2.30 ms per call, 1.0 calls per request, 7.2 rows per call
|
|
142
|
+
SELECT "travel_destinations"."id", ... WHERE "travel_destinations"."published" = TRUE AND "travel_destinations"."latitude" BETWEEN ? AND ? ...
|
|
143
|
+
at app/models/concerns/nearby_venues.rb:39 (SkiResortsController#show 12x, GolfCoursesController#show 12x)
|
|
144
|
+
plan: SEARCH travel_destinations USING INDEX index_travel_destinations_on_published (published=?)
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Many calls per request from one line is an N+1; a query on every page
|
|
148
|
+
(a layout or footer) is worth caching. A sort without an index only matters
|
|
149
|
+
when the query is slow; on a few rows it isn't. Run EXPLAIN against a
|
|
150
|
+
database with production's schema and, ideally, its data: plans depend on
|
|
151
|
+
both.
|
|
152
|
+
|
|
153
|
+
## Queries load_async could start
|
|
154
|
+
|
|
155
|
+
A page's queries usually run one after another, each where the view first
|
|
156
|
+
needs it. But a relation the controller leaves unloaded in an instance
|
|
157
|
+
variable already has its SQL fixed when rendering starts, so all of them
|
|
158
|
+
could start at once with `load_async`, in the background, while the view
|
|
159
|
+
renders. This finds them in production traffic:
|
|
160
|
+
|
|
161
|
+
```ruby
|
|
162
|
+
# config/environments/production.rb
|
|
163
|
+
config.rails_minify.async_queries = 0.01 # 1% of requests; true for all
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
For each sampled request with such relations, one log line records which
|
|
167
|
+
the view loaded, how long each main query took and how far into rendering
|
|
168
|
+
it was first needed:
|
|
169
|
+
|
|
170
|
+
```
|
|
171
|
+
[rails-minify] load_async SpotsController#index sql=28.0 @spots=Spot:11.0@2.0 @areas=Area:4.0@14.0 @news=Article:-
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
`bin/rails minify:async:report LOG=production.log` adds them up per action,
|
|
175
|
+
with the estimated saving:
|
|
176
|
+
|
|
177
|
+
```
|
|
178
|
+
SpotsController#index n 2 saving mean 8.0 ms, p90 10.0 ms (foreground SQL mean 26.5 ms)
|
|
179
|
+
@spots Spot loaded in 100% query 10.0 ms first used 1.5 ms into rendering
|
|
180
|
+
@areas Area loaded in 100% query 3.5 ms first used 13.0 ms into rendering
|
|
181
|
+
@news Article loaded in 50% query 6.0 ms first used 20.0 ms into rendering
|
|
182
|
+
```
|
|
183
|
+
|
|
184
|
+
The fix is `@spots = Spot.where(...).load_async` in the controller. The
|
|
185
|
+
estimate assumes the database answers as fast in parallel; check the change
|
|
186
|
+
with the [A/B](#ab-in-production) before keeping it. `load_async` needs
|
|
187
|
+
`config.active_record.async_query_executor = :global_thread_pool` and a
|
|
188
|
+
database pool big enough for it (`minify:doctor` reports a missing
|
|
189
|
+
executor). Not counted: preloads (`includes`), which still run in the
|
|
190
|
+
foreground when the relation is used; relations the view only counts
|
|
191
|
+
(`size`, `any?`); and association proxies (`@user.posts`). A relation loaded
|
|
192
|
+
in only some requests costs a query nobody reads in the others.
|
|
193
|
+
|
|
194
|
+
Not for SQLite: the sqlite3 gem (2.10) holds the GVL while a query runs,
|
|
195
|
+
so a query in the background stops the view rendering in the foreground and
|
|
196
|
+
nothing overlaps (a 415 ms query and 136 ms of Ruby: 526 ms in two threads,
|
|
197
|
+
551 ms one after the other). It needs a database the process waits on over
|
|
198
|
+
a socket (PostgreSQL, MySQL).
|
|
199
|
+
|
|
200
|
+
No production result yet.
|
|
201
|
+
|
|
202
|
+
## What paid off in production
|
|
203
|
+
|
|
204
|
+
Each gem's README has benchmark numbers. In production, on
|
|
205
|
+
[Japan Voyage](https://japanvoyage.jp) (Falcon, 6 workers, ZJIT, SQLite),
|
|
206
|
+
each part ran for an hour on half the workers while the other half ran
|
|
207
|
+
without it, under the same traffic (about 5,000 requests per half):
|
|
208
|
+
|
|
209
|
+
| part | in production | in 0.2.0 |
|
|
210
|
+
|---|---|---|
|
|
211
|
+
| ZJIT after boot | HTML p50/p90/p99 75/131/231 → 70/126/208 ms, ~25 MB less per worker (an hour before against an hour after) | on with `enabled` (production, Ruby 4.1+) |
|
|
212
|
+
| enums | the same medians; that hour's lower mean and p99 came from a few slow requests, as it changes no allocations on these pages | on with `enabled`: harmless, no measured gain |
|
|
213
|
+
| strict | the same latency and memory; no calls to a removed method | on with `enabled`, so production runs what the tests run |
|
|
214
|
+
| small records | no difference (tens of records per request) | experimental |
|
|
215
|
+
| route helpers | no difference in view time | experimental |
|
|
216
|
+
| views (`render`/`link_to`) | no difference in view time | experimental |
|
|
217
|
+
| compiled call sites in views | ~22 MB more per worker, no faster | experimental |
|
|
218
|
+
| compiled call sites at load time | ~12% more CPU per request (not reproduced locally) | experimental |
|
|
219
|
+
| callbacks | no faster, ~20 MB more per worker | experimental |
|
|
220
|
+
| patches | no difference (view time 25.3 vs 26.8 ms) | experimental |
|
|
221
|
+
|
|
222
|
+
ZJIT after boot is the one part with a clear effect. What `enabled` turns on
|
|
223
|
+
is what Japan Voyage runs.
|
|
224
|
+
|
|
225
|
+
What the benchmarks speed up is per call: a method call, a Hash, a helper.
|
|
226
|
+
A page there spends ~30 ms in Active Record and ~30 ms in views, and those
|
|
227
|
+
savings were smaller than the difference between two halves of the same
|
|
228
|
+
workers. Parts that generate or compile code in each worker (callbacks,
|
|
229
|
+
compiled views) cost memory in every worker; most of these ran before the
|
|
230
|
+
app was preloaded, and would cost less with preloading.
|
|
231
|
+
|
|
232
|
+
Besides ZJIT after boot, what sped Japan Voyage up was outside this gem: a
|
|
233
|
+
preloading server warming the heap up before it forks, and one query fixed.
|
|
234
|
+
Together, HTML p50/p90/p99 went from 75/131/231 to 45/66/~105 ms.
|
|
235
|
+
|
|
236
|
+
## A/B in production
|
|
237
|
+
|
|
238
|
+
How the numbers above were measured, packaged: the server runs twice on one
|
|
239
|
+
port, half the workers each, as groups A and B. The kernel spreads
|
|
240
|
+
connections over both (`SO_REUSEPORT`), so the groups see the same traffic,
|
|
241
|
+
and group B runs with one change. Comparing an hour before with an hour after
|
|
242
|
+
mixes the change up with the traffic; this doesn't.
|
|
243
|
+
|
|
244
|
+
```ruby
|
|
245
|
+
# config/environments/production.rb: group B's change (any configuration)
|
|
246
|
+
config.rails_minify.ab = ->(config) { config.ar_minimizer.small_records = true }
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
```ruby
|
|
250
|
+
# config/falcon.rb: both groups bind the port, and count follows WEB_CONCURRENCY
|
|
251
|
+
service ["myapp", ENV["RAILS_MINIFY_GROUP"]].compact.join("-") do
|
|
252
|
+
include Falcon::Environment::Rack
|
|
253
|
+
preload "config/preload.rb"
|
|
254
|
+
count ENV.fetch("WEB_CONCURRENCY", Etc.nprocessors).to_i
|
|
255
|
+
endpoint do
|
|
256
|
+
Async::HTTP::Endpoint.parse("http://0.0.0.0:3000")
|
|
257
|
+
.with(protocol: Async::HTTP::Protocol::HTTP11, reuse_port: true)
|
|
258
|
+
end
|
|
259
|
+
end
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
```dockerfile
|
|
263
|
+
CMD ["bundle", "exec", "rails-minify-ab", "falcon", "host", "config/falcon.rb"]
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
* `rails-minify-ab COMMAND` runs the command as it is. With
|
|
267
|
+
`RAILS_MINIFY_AB=1` it runs it twice instead, with `RAILS_MINIFY_GROUP=A`
|
|
268
|
+
and `=B` and `WEB_CONCURRENCY` split in half. It stops both on TERM/INT,
|
|
269
|
+
exiting 0 for a clean stop, and stops the other when one exits.
|
|
270
|
+
* Each group's log lines get a tag, after the request id (which is added if
|
|
271
|
+
`config.log_tags` doesn't have it): `[<request id>] [minify:B] ...`.
|
|
272
|
+
* `bin/rails minify:ab:report LOG=file` (or `LOG=-` for stdin) compares the
|
|
273
|
+
groups: requests and 5xx, latency p50/p90/p99 and mean, view, Active Record
|
|
274
|
+
and GC time, for all requests, HTML and the busiest actions, with B−A and a
|
|
275
|
+
95% interval. For example,
|
|
276
|
+
`bin/kamal app logs --since 60m --skip-timestamps | bin/rails minify:ab:report LOG=-`.
|
|
277
|
+
* `bin/rails minify:ab:memory`, where the server runs
|
|
278
|
+
(`bin/kamal app exec --reuse 'bin/rails minify:ab:memory'`): RSS, PSS and
|
|
279
|
+
private memory per process and PSS per group, from `/proc`.
|
|
280
|
+
* `RailsMinify::AB.group` is the process's group (nil outside an A/B run),
|
|
281
|
+
for changes that aren't configuration.
|
|
282
|
+
|
|
283
|
+
Tried with Falcon only so far; Puma would need each cluster to bind with
|
|
284
|
+
`SO_REUSEPORT`. An hour shows a large difference; run another before believing a small
|
|
285
|
+
one.
|
|
286
|
+
|
|
287
|
+
## Experimental
|
|
288
|
+
|
|
289
|
+
Each was faster in benchmarks and made no measurable difference in
|
|
290
|
+
production (above). They still work, each turned on by its own setting:
|
|
291
|
+
|
|
292
|
+
| part | setting | gem |
|
|
293
|
+
|---|---|---|
|
|
294
|
+
| small records: no Hashes for records that never fill them (-22% per record) | `config.ar_minimizer.small_records` | activerecord-minimizer |
|
|
295
|
+
| call sites compiled to what removed methods stood for | `config.ar_minimizer.views`, `.load_time` | activerecord-minimizer |
|
|
296
|
+
| callback chains as generated methods (`valid?` -33..45%) | `config.as_minimizer.unroll_callbacks` | activesupport-minimizer |
|
|
297
|
+
| route helpers as interpolation (3-14x) | `config.ap_minimizer.route_helpers` | actionpack-minimizer |
|
|
298
|
+
| `render`/`link_to` without their machinery (2x per page of partials) | `config.ap_minimizer.views` | actionpack-minimizer |
|
|
299
|
+
| patches | `config.rails_minify.patches` | rails-minify |
|
|
300
|
+
|
|
301
|
+
actionpack-minimizer is no longer a dependency: add `gem "actionpack-minimizer"`
|
|
302
|
+
to use it. `bin/rails minify:census` shows how much of the app the compiling
|
|
303
|
+
parts cover, and why the rest isn't (actionpack-minimizer's part when it's
|
|
304
|
+
installed).
|
|
305
|
+
|
|
306
|
+
### Patches
|
|
307
|
+
|
|
308
|
+
For parts of Rails and common gems that redo work on every request. Off
|
|
309
|
+
unless `config.rails_minify.patches` is set: `true` for all of them, or a
|
|
310
|
+
list (`[:omniauth]`). In production they made no difference in view time;
|
|
311
|
+
each is on its way upstream.
|
|
312
|
+
|
|
313
|
+
* `:action_text_plain_text`: `ActionText::Content#to_plain_text` parses the
|
|
314
|
+
HTML with Nokogiri, and looks up every attachment's record, on every call.
|
|
315
|
+
It's computed once per distinct HTML (cache capped at 8 MB of HTML).
|
|
316
|
+
Attachments that are Active Storage blobs, remote images or inline content
|
|
317
|
+
are cached too: a blob without a caption contributes its filename, which a
|
|
318
|
+
hit checks in one query (renamed or deleted: computed again). Other
|
|
319
|
+
attachables aren't cached. In [Japan Voyage](https://japanvoyage.jp), 13% of all allocations on its
|
|
320
|
+
pages came from it, and guides with images another 4.5%.
|
|
321
|
+
* `:request_host`: `ActionDispatch::Request#host` runs a regexp substitution
|
|
322
|
+
on every call, port or not, and Rails reads it several times per request.
|
|
323
|
+
rails/rails#59104 (Rails 8.2) for 8.1: no substitution without a port, and
|
|
324
|
+
the host kept while the raw host is the same String. Not installed on 8.2.
|
|
325
|
+
* `:omniauth`: every OmniAuth strategy middleware dups itself and its
|
|
326
|
+
options on every request before looking at the path. Requests that can't
|
|
327
|
+
be OmniAuth's go straight to the app. Only when that's certain up front:
|
|
328
|
+
a session is present, not in test mode, no `other_phase` or callable
|
|
329
|
+
paths, and the strategy doesn't override how paths are matched. Not
|
|
330
|
+
installed when OmniAuth already does this itself (it then has a public
|
|
331
|
+
`Strategy#passthrough?`, as in the version proposed upstream).
|
|
332
|
+
|
|
333
|
+
### Allocations
|
|
334
|
+
|
|
335
|
+
Japan Voyage, 132 production pages, with everything on (strict mode after
|
|
336
|
+
rubocop-minify's rewrites, all patches): identical HTML, and
|
|
337
|
+
1,561,283 → 1,224,141 allocations per pass (−21.6%). The machine was too
|
|
338
|
+
loaded for wall-clock times to mean much.
|
|
339
|
+
|
|
340
|
+
## Things found that are app config
|
|
341
|
+
|
|
342
|
+
`minify:doctor` checks for these.
|
|
135
343
|
|
|
136
344
|
* sentry-rails' `breadcrumbs_logger = [:active_support_logger]` subscribes to
|
|
137
345
|
every notification (each SQL query, render, instantiation) and takes
|
data/exe/rails-minify-ab
ADDED
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
# frozen_string_literal: true
|
|
3
|
+
|
|
4
|
+
# rails-minify-ab COMMAND...
|
|
5
|
+
#
|
|
6
|
+
# Runs COMMAND (the server, e.g. `bundle exec falcon host config/falcon.rb`).
|
|
7
|
+
# With RAILS_MINIFY_AB set, runs it twice instead, as groups A and B:
|
|
8
|
+
# RAILS_MINIFY_GROUP=A and =B, WEB_CONCURRENCY split in half. Both bind the
|
|
9
|
+
# same port, so the server has to set SO_REUSEPORT (Falcon: an endpoint
|
|
10
|
+
# `.with(reuse_port: true)`); the kernel then spreads connections over both,
|
|
11
|
+
# and the groups see the same traffic. Stops both when told to (TERM, INT)
|
|
12
|
+
# or when either exits.
|
|
13
|
+
|
|
14
|
+
require "etc"
|
|
15
|
+
|
|
16
|
+
abort "usage: rails-minify-ab COMMAND..." if ARGV.empty?
|
|
17
|
+
exec(*ARGV) if ENV["RAILS_MINIFY_AB"].to_s.empty?
|
|
18
|
+
|
|
19
|
+
workers = Integer(ENV.fetch("WEB_CONCURRENCY", Etc.nprocessors))
|
|
20
|
+
half = workers / 2
|
|
21
|
+
abort "rails-minify-ab: WEB_CONCURRENCY=#{workers}, needs at least 2 for two groups" if half.zero?
|
|
22
|
+
|
|
23
|
+
pids = { "A" => half, "B" => workers - half }.to_h do |group, count|
|
|
24
|
+
[Process.spawn({ "RAILS_MINIFY_GROUP" => group, "WEB_CONCURRENCY" => count.to_s }, *ARGV), group]
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
stopping = false
|
|
28
|
+
%w[TERM INT].each do |signal|
|
|
29
|
+
Signal.trap(signal) do
|
|
30
|
+
stopping = true
|
|
31
|
+
pids.each_key { Process.kill(signal, it) rescue nil }
|
|
32
|
+
end
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
# The first to exit, then the other: told to stop, exit with their own
|
|
36
|
+
# status (0 for a clean stop); one exited by itself is never a clean stop.
|
|
37
|
+
statuses = {}
|
|
38
|
+
pid, status = Process.wait2
|
|
39
|
+
statuses[pid] = status
|
|
40
|
+
(pids.keys - [pid]).each do |other|
|
|
41
|
+
Process.kill("TERM", other)
|
|
42
|
+
rescue Errno::ESRCH
|
|
43
|
+
end
|
|
44
|
+
(pids.keys - [pid]).each do |other|
|
|
45
|
+
statuses[other] = Process.wait2(other).last
|
|
46
|
+
rescue Errno::ECHILD
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
codes = statuses.values.compact.map { it.exitstatus || 128 + it.termsig.to_i }
|
|
50
|
+
if stopping
|
|
51
|
+
exit codes.find(&:nonzero?) || 0
|
|
52
|
+
else
|
|
53
|
+
first = status.exitstatus || 128 + status.termsig.to_i
|
|
54
|
+
warn "rails-minify-ab: group #{pids[pid]} exited (#{first}), stopped the other"
|
|
55
|
+
exit first.zero? ? 1 : first
|
|
56
|
+
end
|
|
@@ -0,0 +1,154 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module RailsMinify
|
|
4
|
+
# Compares a change against the app without it, under the same traffic: the
|
|
5
|
+
# server runs twice on one port (SO_REUSEPORT), half the workers each, as
|
|
6
|
+
# groups A and B (exe/rails-minify-ab). RAILS_MINIFY_GROUP tells a process
|
|
7
|
+
# its group, config.rails_minify.ab changes group B's configuration, and a
|
|
8
|
+
# log tag per group lets AB::Report compare them. One hour is enough to see
|
|
9
|
+
# a large difference, not to rule one out: run it again before believing a
|
|
10
|
+
# small one.
|
|
11
|
+
module AB
|
|
12
|
+
GROUPS = %w[A B].freeze
|
|
13
|
+
|
|
14
|
+
def self.group
|
|
15
|
+
group = ENV["RAILS_MINIFY_GROUP"].to_s.upcase
|
|
16
|
+
group if GROUPS.include?(group)
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def self.log_tag(group = self.group) = "minify:#{group}"
|
|
20
|
+
|
|
21
|
+
# Request log lines, as Rails writes them with config.log_tags, from any
|
|
22
|
+
# log the tags survive in (kamal app logs, docker logs, a log file).
|
|
23
|
+
class Report
|
|
24
|
+
Request = Data.define(:group, :action, :status, :total, :views, :db, :gc)
|
|
25
|
+
|
|
26
|
+
TAGS = /((?:\[[^\]]*\] )+)/
|
|
27
|
+
PROCESSING = /#{TAGS}Processing by (\S+) as (\S+)/
|
|
28
|
+
COMPLETED = /#{TAGS}Completed (\d{3}) .*? in (\d+)ms(?: \((.*)\))?/
|
|
29
|
+
|
|
30
|
+
attr_reader :requests
|
|
31
|
+
|
|
32
|
+
def initialize(lines)
|
|
33
|
+
actions = {}
|
|
34
|
+
@requests = []
|
|
35
|
+
lines.each do |line|
|
|
36
|
+
line = line.scrub
|
|
37
|
+
if (m = PROCESSING.match(line))
|
|
38
|
+
group, key = tags(m[1])
|
|
39
|
+
actions[key] = "#{m[2]} #{m[3]}" if group
|
|
40
|
+
elsif (m = COMPLETED.match(line))
|
|
41
|
+
group, key = tags(m[1])
|
|
42
|
+
next unless group
|
|
43
|
+
|
|
44
|
+
details = m[4].to_s
|
|
45
|
+
ms = ->(name) { details[/#{name}: ([\d.]+)ms/, 1].to_f }
|
|
46
|
+
@requests << Request.new(group, actions.delete(key) || "?", m[2].to_i, m[3].to_i,
|
|
47
|
+
ms.("Views"), ms.("ActiveRecord"), ms.("GC"))
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def report(io = $stdout, actions: 10)
|
|
53
|
+
ok = @requests.select { it.status < 400 }
|
|
54
|
+
io.puts "requests: #{GROUPS.map { |g| "#{g} #{@requests.count { it.group == g }}" }.join(", ")}; " \
|
|
55
|
+
"5xx: #{GROUPS.map { |g| "#{g} #{@requests.count { it.group == g && it.status >= 500 }}" }.join(", ")}"
|
|
56
|
+
if @requests.empty?
|
|
57
|
+
io.puts "no tagged requests: is the log from processes started by rails-minify-ab, with RAILS_MINIFY_AB set?"
|
|
58
|
+
return
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
io.puts
|
|
62
|
+
section(io, "all", ok)
|
|
63
|
+
section(io, "html", ok.select { it.action.end_with?(" HTML") })
|
|
64
|
+
by_action = ok.select { it.action.end_with?(" HTML") }.group_by(&:action)
|
|
65
|
+
by_action.max_by(actions) { |_, rs| rs.size }.each { |action, rs| section(io, action, rs) }
|
|
66
|
+
io.puts
|
|
67
|
+
io.puts "B-A: difference of the means with a 95% interval. An interval that includes 0 is no difference " \
|
|
68
|
+
"this window can show; one that doesn't is worth another window (requests aren't independent, " \
|
|
69
|
+
"so the interval is optimistic)."
|
|
70
|
+
end
|
|
71
|
+
|
|
72
|
+
private
|
|
73
|
+
|
|
74
|
+
def tags(text)
|
|
75
|
+
list = text.scan(/\[([^\]]*)\]/).flatten
|
|
76
|
+
group = list.find { it.start_with?("minify:") }&.delete_prefix("minify:")
|
|
77
|
+
[group, (list - ["minify:#{group}"]).join(" ")] if GROUPS.include?(group)
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def section(io, title, requests)
|
|
81
|
+
a, b = GROUPS.map { |g| requests.select { it.group == g } }
|
|
82
|
+
io.puts title
|
|
83
|
+
{ "A" => a, "B" => b }.each { |g, rs| io.puts " #{g}: #{summary(rs)}" }
|
|
84
|
+
io.puts " B-A: #{difference(a, b)}" unless a.empty? || b.empty?
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def summary(rs)
|
|
88
|
+
return "-" if rs.empty?
|
|
89
|
+
|
|
90
|
+
t = rs.map(&:total)
|
|
91
|
+
format("n %5d p50 %4d p90 %4d p99 %4d mean %6.1f views %5.1f AR %5.1f GC %4.2f ms",
|
|
92
|
+
t.size, pct(t, 50), pct(t, 90), pct(t, 99), mean(t), mean(rs.map(&:views)), mean(rs.map(&:db)), mean(rs.map(&:gc)))
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
def difference(a, b)
|
|
96
|
+
%i[total views db gc].map do |field|
|
|
97
|
+
x, y = a.map(&field), b.map(&field)
|
|
98
|
+
d = mean(y) - mean(x)
|
|
99
|
+
half = 1.96 * Math.sqrt(variance(x) / x.size + variance(y) / y.size)
|
|
100
|
+
label = { total: "mean", views: "views", db: "AR", gc: "GC" }[field]
|
|
101
|
+
format("%s %+.1f (%+.1f..%+.1f)", label, d, d - half, d + half)
|
|
102
|
+
end.join(" ")
|
|
103
|
+
end
|
|
104
|
+
|
|
105
|
+
def pct(xs, p)
|
|
106
|
+
xs = xs.sort
|
|
107
|
+
xs[[xs.size - 1, ((p / 100.0) * (xs.size - 1)).round].min]
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
def mean(xs) = xs.empty? ? 0.0 : xs.sum.to_f / xs.size
|
|
111
|
+
|
|
112
|
+
def variance(xs)
|
|
113
|
+
return 0.0 if xs.size < 2
|
|
114
|
+
|
|
115
|
+
m = mean(xs)
|
|
116
|
+
xs.sum { (it - m)**2 } / (xs.size - 1)
|
|
117
|
+
end
|
|
118
|
+
end
|
|
119
|
+
|
|
120
|
+
# Memory of the processes in each group, from /proc (Linux; run it where
|
|
121
|
+
# the server runs, e.g. bin/kamal app exec). PSS counts pages shared with
|
|
122
|
+
# other processes (the preloaded parent's) in part, so the groups' totals
|
|
123
|
+
# compare what each costs.
|
|
124
|
+
module Memory
|
|
125
|
+
Entry = Data.define(:pid, :group, :title, :rss, :pss, :private_dirty)
|
|
126
|
+
|
|
127
|
+
def self.processes(proc_dir = "/proc")
|
|
128
|
+
Dir["#{proc_dir}/[0-9]*"].filter_map do |dir|
|
|
129
|
+
environ = File.binread("#{dir}/environ") rescue next
|
|
130
|
+
group = environ.split("\0").find { it.start_with?("RAILS_MINIFY_GROUP=") }&.delete_prefix("RAILS_MINIFY_GROUP=")
|
|
131
|
+
next unless GROUPS.include?(group)
|
|
132
|
+
|
|
133
|
+
rollup = File.read("#{dir}/smaps_rollup") rescue next
|
|
134
|
+
kb = ->(name) { rollup[/^#{name}:\s+(\d+) kB/, 1].to_i }
|
|
135
|
+
title = (File.binread("#{dir}/cmdline").tr("\0", " ").strip rescue "")
|
|
136
|
+
Entry.new(File.basename(dir).to_i, group, title, kb.("Rss"), kb.("Pss"), kb.("Private_Dirty"))
|
|
137
|
+
end.sort_by { [it.group, it.pid] }
|
|
138
|
+
end
|
|
139
|
+
|
|
140
|
+
def self.report(io = $stdout, proc_dir: "/proc")
|
|
141
|
+
list = processes(proc_dir)
|
|
142
|
+
if list.empty?
|
|
143
|
+
io.puts "no processes with RAILS_MINIFY_GROUP in #{proc_dir}: run this where the server runs, with RAILS_MINIFY_AB set"
|
|
144
|
+
return
|
|
145
|
+
end
|
|
146
|
+
|
|
147
|
+
list.each do |p|
|
|
148
|
+
io.puts format("%-2s %7d rss %5d pss %5d private %5d MB %s", p.group, p.pid, p.rss / 1024, p.pss / 1024, p.private_dirty / 1024, p.title[0, 50])
|
|
149
|
+
end
|
|
150
|
+
io.puts GROUPS.map { |g| "#{g}: PSS #{list.select { it.group == g }.sum(&:pss) / 1024} MB" }.join(", ")
|
|
151
|
+
end
|
|
152
|
+
end
|
|
153
|
+
end
|
|
154
|
+
end
|