schema_reaper 1.0.16 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +98 -0
- data/PRO.md +11 -6
- data/README.md +368 -71
- data/lib/generators/schema_reaper/install/install_generator.rb +194 -0
- data/lib/generators/schema_reaper/install/templates/initializer.rb.tt +18 -0
- data/lib/generators/schema_reaper/install/templates/schema_reaper_controller.rb.tt +69 -0
- data/lib/schema_reaper/alert_config.rb +55 -0
- data/lib/schema_reaper/analyzers/dead_column.rb +10 -4
- data/lib/schema_reaper/introspect/postgres.rb +22 -4
- data/lib/schema_reaper/mailer.rb +31 -0
- data/lib/schema_reaper/notifier.rb +106 -0
- data/lib/schema_reaper/scan_job.rb +39 -0
- data/lib/schema_reaper/tasks/schema_reaper.rake +5 -0
- data/lib/schema_reaper/version.rb +1 -1
- data/lib/schema_reaper.rb +14 -0
- data/schema_reaper.gemspec +5 -1
- metadata +83 -2
data/README.md
CHANGED
|
@@ -1,7 +1,18 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
<img src="docs/assets/hero.svg" width="100%" alt="schema_reaper — find and safely remove the dead columns, tables and indexes your Rails + PostgreSQL app no longer uses" />
|
|
4
|
+
|
|
5
|
+
[](https://rubygems.org/gems/schema_reaper)
|
|
6
|
+
[](https://rubygems.org/gems/schema_reaper)
|
|
7
|
+
[](https://github.com/aksshatt/schema_reaper/actions/workflows/main.yml)
|
|
8
|
+
[](LICENSE.txt)
|
|
9
|
+
[](schema_reaper.gemspec)
|
|
10
|
+
[-336791?logo=postgresql&logoColor=white)](#install)
|
|
11
|
+
|
|
12
|
+
| ⚡ [Quickstart](#quickstart) | ⚙️ [Usage](#usage) | 🤖 [Production automation](#production-automation) | 🔍 [Analyzers](#analyzers) | 🛡️ [Safety model](#safety-model) | ✅ [CI](#ci) | 🧩 [Configuration](#configuration) | 💼 [Pro](#pro-for-teams) |
|
|
13
|
+
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
|
14
|
+
|
|
15
|
+
</div>
|
|
5
16
|
|
|
6
17
|
`schema_reaper` reads your **live PostgreSQL schema and planner statistics** and
|
|
7
18
|
cross-references them against a **static scan of your codebase** (Ruby via the
|
|
@@ -10,129 +21,408 @@ Prism AST, plus views and SQL string literals). Optionally it also fuses in a
|
|
|
10
21
|
production. Every finding is scored by confidence and severity, carries an
|
|
11
22
|
estimate of the disk it reclaims, and comes with a concrete fix.
|
|
12
23
|
|
|
13
|
-
|
|
24
|
+
It finds three kinds of schema debt:
|
|
25
|
+
|
|
26
|
+
<picture>
|
|
27
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/debt-dark.svg">
|
|
28
|
+
<img src="docs/assets/debt-light.svg" width="100%" alt="Dead weight — columns and tables nothing references any more (dead_column, dead_table). Index trouble — indexes nobody queries, indexes a wider one already covers, and foreign keys with no index (unused_index, duplicate_index, missing_fk_index). Degenerate data — columns that are always NULL or hold one value in every row (always_null_column, single_value_column).">
|
|
29
|
+
</picture>
|
|
30
|
+
|
|
31
|
+
## Quickstart
|
|
14
32
|
|
|
15
33
|
```ruby
|
|
16
34
|
# Gemfile
|
|
17
|
-
gem "schema_reaper"
|
|
35
|
+
gem "schema_reaper"
|
|
18
36
|
```
|
|
19
37
|
|
|
20
|
-
```
|
|
38
|
+
```sh
|
|
21
39
|
bundle install
|
|
40
|
+
bundle exec schema_reaper scan
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
In a Rails app that's it — the connection comes from `config/database.yml`.
|
|
44
|
+
You get a report like this:
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<img src="docs/assets/terminal.svg" width="100%" alt="Terminal output of bundle exec schema_reaper scan: 3 findings across 2 tables — users.team_id has no covering index (90%, medium, fix: add_index :users, :team_id); users.api_key is NULL in every row (85%, high, 46.9 KB reclaimable); table stale_exports has no model or query reference and holds 0 rows (85%, high, fix: drop_table :stale_exports after confirming no external consumer)." />
|
|
48
|
+
</p>
|
|
49
|
+
|
|
50
|
+
> [!TIP]
|
|
51
|
+
> Scan a database that has served real traffic — a production read-replica or a
|
|
52
|
+
> recent snapshot. `unused_index` needs query history (it skips itself on a
|
|
53
|
+
> fresh database and tells you why), and the data analyzers read `pg_stats`,
|
|
54
|
+
> which PostgreSQL only fills in after `ANALYZE`.
|
|
55
|
+
|
|
56
|
+
## How it works
|
|
57
|
+
|
|
58
|
+
<picture>
|
|
59
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/pipeline-dark.svg">
|
|
60
|
+
<img src="docs/assets/pipeline-light.svg" width="100%" alt="Three inputs — the live PostgreSQL schema and statistics, a static scan of your codebase, and an optional runtime signal — feed seven analyzers. Their scored findings become reports in four formats, a CI baseline gate, scheduled webhook and email reports, and staged removal migrations.">
|
|
61
|
+
</picture>
|
|
62
|
+
|
|
63
|
+
Findings that say the same thing about different tables are rolled up into one
|
|
64
|
+
entry; when several analyzers flag the same column, the strongest finding wins
|
|
65
|
+
and notes the others that agreed. `schema_reaper` never changes your database —
|
|
66
|
+
it only reads, and hands you migrations to review.
|
|
67
|
+
|
|
68
|
+
## Install
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
# Gemfile
|
|
72
|
+
gem "schema_reaper"
|
|
22
73
|
```
|
|
23
74
|
|
|
24
|
-
|
|
25
|
-
|
|
75
|
+
| requirement | notes |
|
|
76
|
+
|---|---|
|
|
77
|
+
| Ruby >= 2.7 | CI-tested on 2.7 – 3.3 |
|
|
78
|
+
| PostgreSQL + the `pg` gem | uses your app's own `pg` (every Rails + PostgreSQL app already bundles it); add `gem "pg"` for standalone CLI use. Tables in the `public` schema are scanned. MySQL is on the [roadmap](#roadmap). |
|
|
79
|
+
| Rails | optional for the CLI; the rake tasks and production automation need it, and the runtime tracker needs ActiveRecord |
|
|
80
|
+
|
|
81
|
+
> [!IMPORTANT]
|
|
82
|
+
> **Only scanning locally or in CI?** Scope it out of your production bundle:
|
|
83
|
+
> `gem "schema_reaper", group: :development`.
|
|
84
|
+
>
|
|
85
|
+
> **Planning to use [production automation](#production-automation)?** Leave it
|
|
86
|
+
> unscoped. Most deploys exclude the development and test groups
|
|
87
|
+
> (`BUNDLE_WITHOUT=development:test`), so a dev-scoped gem is simply absent when
|
|
88
|
+
> the scheduled scan tries to run. The install generator warns you about this.
|
|
89
|
+
|
|
90
|
+
<details>
|
|
91
|
+
<summary><b>How the database connection is resolved</b></summary>
|
|
92
|
+
|
|
93
|
+
<br>
|
|
26
94
|
|
|
27
95
|
1. `database_url:` in `.schema_reaper.yml`
|
|
28
96
|
2. `ENV["DATABASE_URL"]`
|
|
29
|
-
3. `config/database.yml` for the current environment (`SCHEMA_REAPER_ENV
|
|
30
|
-
`RAILS_ENV`, default `development`) — ERB and YAML aliases
|
|
31
|
-
are Rails 6+ multi-database sections
|
|
97
|
+
3. `config/database.yml` for the current environment (`SCHEMA_REAPER_ENV`, then
|
|
98
|
+
`RAILS_ENV`, then `RACK_ENV`, default `development`) — ERB and YAML aliases
|
|
99
|
+
are handled, as are Rails 6+ multi-database sections (the `primary` entry is
|
|
100
|
+
used)
|
|
32
101
|
|
|
33
|
-
|
|
102
|
+
</details>
|
|
34
103
|
|
|
35
104
|
## Usage
|
|
36
105
|
|
|
37
|
-
```
|
|
106
|
+
```sh
|
|
38
107
|
bundle exec schema_reaper scan # grouped terminal report
|
|
39
108
|
bundle exec schema_reaper scan --format markdown # PR-comment table
|
|
40
109
|
bundle exec schema_reaper scan --format sarif # GitHub code scanning
|
|
41
110
|
bundle exec schema_reaper scan --format json
|
|
42
|
-
bundle exec schema_reaper scan --ci # exit 1 on
|
|
111
|
+
bundle exec schema_reaper scan --ci # exit 1 on findings not in the baseline
|
|
43
112
|
bundle exec schema_reaper scan --min-confidence 0.8
|
|
44
|
-
bundle exec schema_reaper scan --no-color # force plain output
|
|
45
113
|
bundle exec schema_reaper baseline # accept current findings
|
|
46
114
|
bundle exec schema_reaper trend # snapshot + progress delta
|
|
47
115
|
bundle exec schema_reaper generate-migration users legacy_api_token
|
|
48
116
|
```
|
|
49
117
|
|
|
50
|
-
|
|
51
|
-
|
|
118
|
+
<details>
|
|
119
|
+
<summary><b>All <code>scan</code> options</b></summary>
|
|
120
|
+
|
|
121
|
+
<br>
|
|
122
|
+
|
|
123
|
+
| option | default | what it does |
|
|
124
|
+
|---|---|---|
|
|
125
|
+
| `--format` | `table` | `table`, `json`, `markdown` or `sarif` |
|
|
126
|
+
| `--ci` | off | exit 1 when a finding isn't in `.schema_reaper/baseline.json` |
|
|
127
|
+
| `--min-confidence N` | `0.0` | hide findings below this confidence (0.0 – 1.0) |
|
|
128
|
+
| `--record` | off | also append this run to the history log used by `trend` |
|
|
129
|
+
| `--color` / `--no-color` | auto | force colour on or off for the table report |
|
|
130
|
+
| `--config PATH` | `.schema_reaper.yml` | config file to load (works on every command) |
|
|
131
|
+
|
|
132
|
+
Colour is automatic on a terminal, and off when output is piped or `NO_COLOR` is set.
|
|
133
|
+
|
|
134
|
+
</details>
|
|
135
|
+
|
|
136
|
+
<details>
|
|
137
|
+
<summary><b>Rake tasks (Rails)</b></summary>
|
|
138
|
+
|
|
139
|
+
<br>
|
|
140
|
+
|
|
141
|
+
The railtie adds these to any Rails app with the gem in its bundle:
|
|
142
|
+
|
|
143
|
+
| task | what it does |
|
|
144
|
+
|---|---|
|
|
145
|
+
| `rake schema_reaper:scan` | same as `schema_reaper scan`; pick a format with `FORMAT=json` etc. |
|
|
146
|
+
| `rake schema_reaper:baseline` | write the current findings to the baseline file |
|
|
147
|
+
| `rake schema_reaper:trend` | record a snapshot and print the trend |
|
|
148
|
+
| `rake schema_reaper:alert` | run one production scan and send the report — what the [schedule](#production-automation) calls |
|
|
149
|
+
|
|
150
|
+
</details>
|
|
151
|
+
|
|
152
|
+
## Production automation
|
|
153
|
+
|
|
154
|
+
*New in v2.0.* Run `schema_reaper` unattended in production and have the report
|
|
155
|
+
land in your team's chat and inbox — no DevOps ticket, and no server access
|
|
156
|
+
beyond a normal deploy.
|
|
157
|
+
|
|
158
|
+
```sh
|
|
159
|
+
bin/rails generate schema_reaper:install
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
<picture>
|
|
163
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/automation-dark.svg">
|
|
164
|
+
<img src="docs/assets/automation-light.svg" width="100%" alt="whenever or cron runs rake schema_reaper:alert, sidekiq-cron enqueues the job directly, and the token-protected manual trigger POST /internal/schema_scan enqueues it on demand. All three run SchemaReaper::ScanJob, which scans the live database and your code; the Notifier sends the report to a Slack-compatible webhook and by email through your ApplicationMailer.">
|
|
165
|
+
</picture>
|
|
166
|
+
|
|
167
|
+
| generated | purpose |
|
|
168
|
+
|---|---|
|
|
169
|
+
| `config/initializers/schema_reaper.rb` | where reports go (webhook and/or email) |
|
|
170
|
+
| an entry in `config/schedule.rb` **or** `config/schedule.yml` | the quarterly scheduled scan (whenever or sidekiq-cron) |
|
|
171
|
+
| `app/controllers/schema_reaper_controller.rb` + a route | a token-protected "scan now" endpoint |
|
|
172
|
+
| a trigger token, printed once | you paste it into your encrypted credentials |
|
|
173
|
+
|
|
174
|
+
Re-running the generator is safe: every step checks for its own marker and
|
|
175
|
+
skips itself rather than duplicating anything. Commit the files, deploy, and
|
|
176
|
+
check it works end to end with one manual run in production:
|
|
177
|
+
|
|
178
|
+
```sh
|
|
179
|
+
bin/rails schema_reaper:alert
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
### 1. Where reports go
|
|
52
183
|
|
|
184
|
+
The generated initializer is commented out, so an unfilled config is a safe
|
|
185
|
+
no-op, not an error. Uncomment one or both:
|
|
186
|
+
|
|
187
|
+
```ruby
|
|
188
|
+
SchemaReaper::AlertConfig.configure do |config|
|
|
189
|
+
# config.emails = %w[dev1@example.com dev2@example.com]
|
|
190
|
+
# config.webhook_url = "https://hooks.slack.com/services/T000/B000/XXXX"
|
|
191
|
+
end
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Both channels fire when both are set — not a fallback chain, but two audiences:
|
|
195
|
+
a dev-facing chat channel, and an inbox record for people who don't watch chat.
|
|
196
|
+
|
|
197
|
+
- **Webhook** — a Slack-compatible `{"text": "..."}` JSON POST carrying the
|
|
198
|
+
Markdown report, with a 10-second timeout.
|
|
199
|
+
- **Email** — sent by `SchemaReaper::Mailer`, which inherits your app's
|
|
200
|
+
`ApplicationMailer` (its `default from:`, delivery settings and so on), or
|
|
201
|
+
`ActionMailer::Base` if you don't have one. Delivered with `deliver_later`.
|
|
202
|
+
|
|
203
|
+
Delivery failures are logged with a `[schema_reaper]` prefix and never raised,
|
|
204
|
+
so a flaky endpoint can't fail or retry-storm the scan.
|
|
205
|
+
|
|
206
|
+
### 2. The schedule
|
|
207
|
+
|
|
208
|
+
The generator looks at your `Gemfile.lock` and uses whichever scheduler you
|
|
209
|
+
already run. The default cadence is **quarterly**; edit the generated entry to
|
|
210
|
+
change it.
|
|
211
|
+
|
|
212
|
+
| you use | the generator adds | also needed |
|
|
213
|
+
|---|---|---|
|
|
214
|
+
| [whenever](https://github.com/javan/whenever) | an `every 3.months` block running `rake "schema_reaper:alert"` in `config/schedule.rb` | whenever only writes your crontab when `whenever --update-crontab` runs on the server — usually via its Capistrano recipe. Platforms without a crontab (Heroku, most container hosts) need one of the other rows. |
|
|
215
|
+
| [sidekiq-cron](https://github.com/sidekiq-cron/sidekiq-cron) | a `schema_reaper_scan` entry in `config/schedule.yml` (`0 4 1 */3 *` — 04:00 on the 1st, every 3 months) that enqueues `SchemaReaper::ScanJob` | nothing — sidekiq-cron ≥ 1.6 loads `config/schedule.yml` automatically |
|
|
216
|
+
| neither | nothing — it prints the command instead of guessing | point any scheduler (cron, Heroku Scheduler, a Kubernetes CronJob) at `bin/rails schema_reaper:alert` |
|
|
217
|
+
|
|
218
|
+
`rake schema_reaper:alert` enqueues the scan on your ActiveJob backend. If that
|
|
219
|
+
backend is Rails' in-process `:async` adapter (the default before Rails 8 unless
|
|
220
|
+
you've set one up), the job would die with the short-lived rake process — so
|
|
221
|
+
the task runs the scan inline instead, email included.
|
|
222
|
+
|
|
223
|
+
### 3. The manual trigger
|
|
224
|
+
|
|
225
|
+
For "scan now" without shelling into a box:
|
|
226
|
+
|
|
227
|
+
```sh
|
|
228
|
+
curl -X POST https://your-app.example.com/internal/schema_scan \
|
|
229
|
+
-H "Authorization: Bearer <token>"
|
|
53
230
|
```
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
████░ 85% high dead_table
|
|
68
|
-
no model or query reference · table holds ~0 row(s)
|
|
69
|
-
→ confirm no external consumer, then `drop_table :stale_exports`
|
|
70
|
-
|
|
71
|
-
high 2 medium 1 low 2
|
|
231
|
+
|
|
232
|
+
| response | meaning |
|
|
233
|
+
|---|---|
|
|
234
|
+
| `202 Accepted` | scan enqueued — the report arrives through the channels above |
|
|
235
|
+
| `401 Unauthorized` | missing or wrong token, or no token configured in this environment |
|
|
236
|
+
| `429 Too Many Requests` | a scan was already triggered in the last 5 minutes |
|
|
237
|
+
|
|
238
|
+
The generator prints the token once, with the exact snippet to paste into
|
|
239
|
+
`bin/rails credentials:edit`:
|
|
240
|
+
|
|
241
|
+
```yaml
|
|
242
|
+
schema_reaper:
|
|
243
|
+
trigger_token: <token>
|
|
72
244
|
```
|
|
73
245
|
|
|
74
|
-
|
|
246
|
+
Credentials are decrypted with the `RAILS_MASTER_KEY` your app already has, so
|
|
247
|
+
this adds no new production configuration. The token is checked with a
|
|
248
|
+
constant-time comparison. The controller deliberately doesn't inherit your
|
|
249
|
+
`ApplicationController` and skips CSRF (it's a token-authenticated API endpoint,
|
|
250
|
+
not a form), so your own auth filters don't apply to it.
|
|
251
|
+
|
|
252
|
+
> [!NOTE]
|
|
253
|
+
> The 5-minute cooldown lives in `Rails.cache`, so it's only as shared as your
|
|
254
|
+
> cache store. Redis, Memcached or Solid Cache give one cooldown for the whole
|
|
255
|
+
> app; a file or memory store gives one per host or process; `:null_store`
|
|
256
|
+
> turns it off.
|
|
257
|
+
|
|
258
|
+
### 4. Install-time warnings
|
|
259
|
+
|
|
260
|
+
Two misconfigurations would otherwise fail silently much later, so the
|
|
261
|
+
generator flags them when it runs:
|
|
75
262
|
|
|
76
|
-
|
|
77
|
-
|
|
263
|
+
- the gem is scoped to `group: :development` — the scheduled scan would never
|
|
264
|
+
run in production
|
|
265
|
+
- neither `ApplicationMailer` nor `config.action_mailer.default_options` sets a
|
|
266
|
+
`from:` address — the mailer would fall back to a placeholder sender that most
|
|
267
|
+
SMTP relays reject, so email reports would never arrive
|
|
268
|
+
|
|
269
|
+
> [!WARNING]
|
|
270
|
+
> **Still on 2.0.0?** Two production-automation bugs are fixed on `main` and
|
|
271
|
+
> ship in the next release:
|
|
272
|
+
> - Report emails ignore `ApplicationMailer` and are sent from
|
|
273
|
+
> `schema_reaper@localhost`. Work around it by adding
|
|
274
|
+
> `SchemaReaper::Mailer.default from: "you@your-domain.com"` to
|
|
275
|
+
> `config/initializers/schema_reaper.rb`.
|
|
276
|
+
> - With the `:async` ActiveJob adapter, `rake schema_reaper:alert` enqueues a
|
|
277
|
+
> job that never runs. Use a persistent backend (Sidekiq, GoodJob, Solid
|
|
278
|
+
> Queue, …) for the whenever and plain-cron paths.
|
|
279
|
+
|
|
280
|
+
<details>
|
|
281
|
+
<summary><b>Why not just run it in CI?</b></summary>
|
|
282
|
+
|
|
283
|
+
<br>
|
|
284
|
+
|
|
285
|
+
[`--format sarif` in CI](#ci) is for PR- and staging-level checks. GitHub-hosted
|
|
286
|
+
runners have no network path to your production database by default, and a CI
|
|
287
|
+
database has no production traffic statistics. Production automation is the path
|
|
288
|
+
for watching the real, live database.
|
|
289
|
+
|
|
290
|
+
</details>
|
|
78
291
|
|
|
79
292
|
## Analyzers
|
|
80
293
|
|
|
81
294
|
| type | what it flags | main signal |
|
|
82
295
|
|---|---|---|
|
|
83
296
|
| `dead_column` | column no code path references | static scan (+ runtime) |
|
|
84
|
-
| `dead_table` | table with no model
|
|
85
|
-
| `unused_index` | non-unique index
|
|
86
|
-
| `duplicate_index` | index that is a prefix of a wider one | schema shape |
|
|
87
|
-
| `missing_fk_index` | `*_id` /
|
|
297
|
+
| `dead_table` | table with no model or query reference | static scan + row count |
|
|
298
|
+
| `unused_index` | non-unique index with `idx_scan = 0` (needs query history) | `pg_stat_user_indexes` |
|
|
299
|
+
| `duplicate_index` | index that exactly duplicates another, or is a prefix of a wider one | schema shape |
|
|
300
|
+
| `missing_fk_index` | `*_id` / foreign-key column with no index (polymorphic pairs need a `(type, id)` index) | schema shape |
|
|
88
301
|
| `always_null_column` | `null_frac = 1.0` — no data at all | `pg_stats` |
|
|
89
|
-
| `single_value_column` | one distinct value on a
|
|
302
|
+
| `single_value_column` | one distinct value on a table of 500+ rows | `pg_stats` |
|
|
90
303
|
|
|
91
|
-
Columns owned by common gems
|
|
92
|
-
|
|
93
|
-
|
|
304
|
+
Columns and tables owned by common gems are whitelisted automatically when the
|
|
305
|
+
gem is in your bundle: **devise**, **devise-api**, **paper_trail**,
|
|
306
|
+
**audited**, **friendly_id**, **paranoia** / **acts_as_paranoid**,
|
|
307
|
+
**activestorage**, **actiontext**, **pg_search**, **ahoy_matey** and
|
|
308
|
+
**activeadmin**.
|
|
94
309
|
|
|
95
310
|
## Runtime signal (optional, raises confidence)
|
|
96
311
|
|
|
97
312
|
Static analysis alone can't see metaprogrammed access, so `dead_column`
|
|
98
|
-
confidence is capped at **0.6** without runtime data. To lift the cap
|
|
313
|
+
confidence is capped at **0.6** without runtime data. To lift the cap, sample
|
|
314
|
+
real column reads:
|
|
99
315
|
|
|
100
316
|
```ruby
|
|
101
|
-
# config/initializers
|
|
317
|
+
# config/initializers/schema_reaper_tracker.rb
|
|
102
318
|
SchemaReaper::Runtime::Tracker.install!(
|
|
103
319
|
store: SchemaReaper::Runtime::Store.new(path: ".schema_reaper/runtime.jsonl"),
|
|
104
320
|
sample_rate: 0.05
|
|
105
321
|
)
|
|
106
322
|
```
|
|
107
323
|
|
|
108
|
-
or, in Rails, boot with `SCHEMA_REAPER_TRACK=1
|
|
324
|
+
or, in Rails, boot with `SCHEMA_REAPER_TRACK=1` (and optionally
|
|
325
|
+
`SCHEMA_REAPER_SAMPLE=0.05` for the sample rate). Let it run in staging or
|
|
109
326
|
production for a couple of weeks. A column unseen in **both** code and
|
|
110
|
-
|
|
327
|
+
≥ 14 observed days of runtime data (`min_age_days` in
|
|
328
|
+
[`.schema_reaper.yml`](#configuration)) reaches **0.9** confidence (0.8 if it's
|
|
329
|
+
`NOT NULL`).
|
|
330
|
+
|
|
331
|
+
> [!NOTE]
|
|
332
|
+
> This is separate from [production automation](#production-automation): the
|
|
333
|
+
> tracker raises confidence in the findings a scan already makes, while
|
|
334
|
+
> production automation runs the scan on a schedule and delivers the report.
|
|
335
|
+
> Most teams want both.
|
|
111
336
|
|
|
112
337
|
## Safety model
|
|
113
338
|
|
|
114
|
-
|
|
339
|
+
<picture>
|
|
340
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/safety-dark.svg">
|
|
341
|
+
<img src="docs/assets/safety-light.svg" width="100%" alt="generate-migration writes two migrations. Step 1 adds the column to ignored_columns and is deployed; nothing is dropped. Once it has soaked in production and nothing reads the column, step 2 runs remove_column, which is irreversible.">
|
|
342
|
+
</picture>
|
|
115
343
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
2. **Drop** — run only after step 1 has soaked in production and nothing broke.
|
|
344
|
+
`schema_reaper` never drops anything itself. For a dead column,
|
|
345
|
+
`generate-migration users legacy_api_token` writes a pair:
|
|
119
346
|
|
|
120
|
-
|
|
121
|
-
`
|
|
347
|
+
1. **`…_ignore_users_legacy_api_token.rb`** — a no-op migration that reminds
|
|
348
|
+
you to add `self.ignored_columns += %w[legacy_api_token]` to the model. Deploy
|
|
349
|
+
that. Nothing is dropped; ActiveRecord just stops selecting the column.
|
|
350
|
+
2. **`…_drop_users_legacy_api_token.rb`** — the `remove_column`. Run it only
|
|
351
|
+
after step 1 has soaked in production and nothing broke. Its `down` raises
|
|
352
|
+
`ActiveRecord::IrreversibleMigration` on purpose.
|
|
353
|
+
|
|
354
|
+
The data-driven fixes are cautious too: `always_null_column` asks you to
|
|
355
|
+
confirm with a `SELECT count(...)` first, `single_value_column` to check the
|
|
356
|
+
value isn't a meaningful default, and `dead_table` to rule out external
|
|
357
|
+
consumers.
|
|
122
358
|
|
|
123
359
|
## CI
|
|
124
360
|
|
|
361
|
+
<picture>
|
|
362
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/ci-dark.svg">
|
|
363
|
+
<img src="docs/assets/ci-light.svg" width="100%" alt="On each pull request, schema_reaper scan --ci compares the findings with the committed baseline.json. It passes when every finding is already in the baseline and exits 1 when the change adds new dead weight.">
|
|
364
|
+
</picture>
|
|
365
|
+
|
|
125
366
|
```yaml
|
|
126
|
-
# .github/workflows/schema_reaper.yml
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
367
|
+
# .github/workflows/schema_reaper.yml (the relevant parts)
|
|
368
|
+
permissions:
|
|
369
|
+
contents: read
|
|
370
|
+
security-events: write # required by upload-sarif
|
|
371
|
+
|
|
372
|
+
# ...job setup: Ruby, a PostgreSQL service, DATABASE_URL...
|
|
373
|
+
steps:
|
|
374
|
+
- run: bin/rails db:schema:load
|
|
375
|
+
- run: bundle exec schema_reaper scan --ci --format sarif > reaper.sarif
|
|
376
|
+
- uses: github/codeql-action/upload-sarif@v4
|
|
377
|
+
if: always() # upload even when --ci fails the step above
|
|
378
|
+
with: { sarif_file: reaper.sarif }
|
|
130
379
|
```
|
|
131
380
|
|
|
132
|
-
Commit `.schema_reaper/baseline.json` so the job
|
|
133
|
-
*new* dead weight.
|
|
381
|
+
Commit `.schema_reaper/baseline.json` (from `schema_reaper baseline`) so the job
|
|
382
|
+
fails only when a change adds *new* dead weight.
|
|
383
|
+
|
|
384
|
+
> [!NOTE]
|
|
385
|
+
> A freshly loaded CI database has no rows, statistics or query history, so CI
|
|
386
|
+
> catches the schema- and code-shaped findings (`dead_column`, `dead_table`,
|
|
387
|
+
> `duplicate_index`, `missing_fk_index`). The data-driven ones need a real
|
|
388
|
+
> database — that's what [production automation](#production-automation) is for.
|
|
389
|
+
|
|
390
|
+
## Configuration
|
|
391
|
+
|
|
392
|
+
Every key is optional. Drop a `.schema_reaper.yml` in the project root to
|
|
393
|
+
override any of these defaults.
|
|
134
394
|
|
|
135
|
-
|
|
395
|
+
> [!CAUTION]
|
|
396
|
+
> A list you set **replaces** the default list rather than adding to it. Setting
|
|
397
|
+
> `ignore: { tables: [legacy_audit] }` stops ignoring `schema_migrations` and
|
|
398
|
+
> `ar_internal_metadata`, and `always_keep_columns: [uuid]` stops protecting
|
|
399
|
+
> `created_at`, `updated_at` and `type`. Repeat the defaults you still want.
|
|
400
|
+
|
|
401
|
+
<details>
|
|
402
|
+
<summary><b>.schema_reaper.yml — all keys with their defaults</b></summary>
|
|
403
|
+
|
|
404
|
+
<br>
|
|
405
|
+
|
|
406
|
+
```yaml
|
|
407
|
+
database_url: # falls back to DATABASE_URL, then config/database.yml
|
|
408
|
+
database_yml: config/database.yml
|
|
409
|
+
scan_paths: [app, lib, config]
|
|
410
|
+
view_globs: ["app/**/*.erb", "app/**/*.haml", "app/**/*.slim", "app/**/*.jbuilder"]
|
|
411
|
+
ignore:
|
|
412
|
+
tables: [schema_migrations, ar_internal_metadata]
|
|
413
|
+
columns: [] # exact names, or "/regex/" patterns
|
|
414
|
+
always_keep_columns: [id, created_at, updated_at, type]
|
|
415
|
+
gem_awareness: true # auto-whitelist columns owned by known gems
|
|
416
|
+
min_age_days: 14 # days of runtime data before dead_column trusts it
|
|
417
|
+
runtime_log: .schema_reaper/runtime.jsonl
|
|
418
|
+
history_log: .schema_reaper/history.jsonl
|
|
419
|
+
baseline: .schema_reaper/baseline.json
|
|
420
|
+
require: [] # extra files to load, e.g. custom analyzers
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
</details>
|
|
424
|
+
|
|
425
|
+
### Custom analyzers
|
|
136
426
|
|
|
137
427
|
```ruby
|
|
138
428
|
# lib/schema_reaper/analyzers/my_check.rb
|
|
@@ -153,6 +443,10 @@ require:
|
|
|
153
443
|
|
|
154
444
|
## Roadmap
|
|
155
445
|
|
|
446
|
+
> [!TIP]
|
|
447
|
+
> ✅ Self-hosted scheduled scans + alerts shipped in **v2.0** — see
|
|
448
|
+
> [Production automation](#production-automation).
|
|
449
|
+
|
|
156
450
|
- Runtime verdict fusion for index and table findings
|
|
157
451
|
- Orphan-row and `schema.rb`↔DB drift analyzers
|
|
158
452
|
- Disk/$ reclaim from real `pg_total_relation_size`
|
|
@@ -161,12 +455,12 @@ require:
|
|
|
161
455
|
|
|
162
456
|
## Pro (for teams)
|
|
163
457
|
|
|
164
|
-
The gem is free and complete for a single app
|
|
165
|
-
team-scale layer: MySQL
|
|
166
|
-
|
|
167
|
-
`pg_total_relation_size` + $ estimates,
|
|
168
|
-
mountable dashboard engine. See
|
|
169
|
-
open an issue tagged `pro`.
|
|
458
|
+
The gem is free and complete for a single app — including the scheduled scans
|
|
459
|
+
and alerts above. **schema_reaper Pro** adds the team-scale layer: MySQL
|
|
460
|
+
adapter, multi-database fan-out, orphan-row and schema-drift analyzers, native
|
|
461
|
+
Slack/Jira/PR-comment reporters, real `pg_total_relation_size` + $ estimates, a
|
|
462
|
+
hosted zero-install scan runner, and a mountable dashboard engine. See
|
|
463
|
+
[PRO.md](PRO.md). Waitlist / early access: open an issue tagged `pro`.
|
|
170
464
|
|
|
171
465
|
## Sponsor
|
|
172
466
|
|
|
@@ -176,11 +470,14 @@ repo.
|
|
|
176
470
|
|
|
177
471
|
## Development
|
|
178
472
|
|
|
179
|
-
```
|
|
473
|
+
```sh
|
|
180
474
|
bin/setup
|
|
181
475
|
bundle exec rake # rspec + rubocop
|
|
476
|
+
|
|
477
|
+
# opt-in: live-database introspection specs
|
|
478
|
+
SCHEMA_REAPER_TEST_DATABASE_URL=postgres://localhost/schema_reaper_test bundle exec rspec
|
|
182
479
|
```
|
|
183
480
|
|
|
184
481
|
## License
|
|
185
482
|
|
|
186
|
-
MIT.
|
|
483
|
+
[MIT](LICENSE.txt).
|