schema_reaper 2.0.0 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/PRO.md +11 -6
- data/README.md +368 -71
- data/lib/generators/schema_reaper/install/install_generator.rb +11 -11
- data/lib/schema_reaper/analyzers/dead_column.rb +10 -4
- data/lib/schema_reaper/introspect/postgres.rb +22 -4
- data/lib/schema_reaper/notifier.rb +12 -3
- data/lib/schema_reaper/scan_job.rb +24 -2
- data/lib/schema_reaper/tasks/schema_reaper.rake +2 -2
- data/lib/schema_reaper/version.rb +1 -1
- data/lib/schema_reaper.rb +11 -1
- data/schema_reaper.gemspec +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 89431a69d7cd997b743e0a7996ce1bae64244946dbea4d0683f8beb477ad1498
|
|
4
|
+
data.tar.gz: d43309fe35f2f6b4c669e21413c300997fdabf03f3d77f3295a6742de5f3cea2
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 6d2c527fc15a64ce62680a96a989c1ba0fbda445fd216afe590dd95e3344b1a8c2ef271a102eed01626b1cac1b0e9400874c8cc177f71d1d27efa32fda79e6fb
|
|
7
|
+
data.tar.gz: 777a1561b89af7d5795a326d9c30d6764190b14f6a3dbb1af5f22d2fb525f3959d73e2f3beaa25262a9508f94958a1c1cad42d908f879855f1643f0206bc1208
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,47 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [2.0.1] - 2026-09-23
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- **Report emails now use the app's `ApplicationMailer`.** `SchemaReaper::Mailer`
|
|
7
|
+
was defined while the gem loaded (`Bundler.require`), before the app's
|
|
8
|
+
autoloader could see `app/mailers/application_mailer.rb`, so it always
|
|
9
|
+
inherited `ActionMailer::Base` and sent from the `schema_reaper@localhost`
|
|
10
|
+
placeholder -- which most SMTP relays reject. It is now autoloaded on first
|
|
11
|
+
use, after boot. The install generator's `from:` check now also accepts
|
|
12
|
+
`config.action_mailer.default_options` for apps without an
|
|
13
|
+
`ApplicationMailer`. 2.0.0 workaround: add
|
|
14
|
+
`SchemaReaper::Mailer.default from: "..."` to the initializer.
|
|
15
|
+
- **`rake schema_reaper:alert` no longer silently drops the scan on the
|
|
16
|
+
`:async` ActiveJob adapter** (the default before Rails 8). The job ran on a
|
|
17
|
+
thread inside the rake process, which exited first -- so the whenever and
|
|
18
|
+
plain-cron schedules never scanned. The task now runs the scan inline (with
|
|
19
|
+
email delivered immediately) when the adapter is `:async`, and enqueues as
|
|
20
|
+
before otherwise.
|
|
21
|
+
- **A missing `pg` gem now gives a clear error.** `pg` is deliberately not a
|
|
22
|
+
runtime dependency (the gem uses the host app's own), but without it the
|
|
23
|
+
scan failed with `NameError: uninitialized constant ...::PG` from the
|
|
24
|
+
connection rescue. It now says to add `gem "pg"`.
|
|
25
|
+
- **`min_age_days` in `.schema_reaper.yml` now takes effect.** It was defined
|
|
26
|
+
and documented since 1.0.0 but never read -- `dead_column` used its own
|
|
27
|
+
hardcoded 14-day threshold for trusting runtime data. The analyzer now reads
|
|
28
|
+
the config key (default still 14, so nothing changes unless you set it).
|
|
29
|
+
|
|
30
|
+
### Changed
|
|
31
|
+
- README rewritten around a quickstart, with animated SVG illustrations
|
|
32
|
+
(`docs/assets/`, excluded from the packaged gem) and corrected details on
|
|
33
|
+
scheduling, the manual trigger's cooldown, CI and gem awareness. Diagrams
|
|
34
|
+
for each major section; animations loop with a long hold on the finished
|
|
35
|
+
frame, since GitHub can't start a README image's animation on scroll.
|
|
36
|
+
Documents the `pg` requirement, that only the `public` schema is scanned, and
|
|
37
|
+
that lists in `.schema_reaper.yml` replace their defaults, and what
|
|
38
|
+
`min_age_days` controls.
|
|
39
|
+
- The `automation`, `safety` and `ci` section diagrams carry label text meant
|
|
40
|
+
to be read, so they no longer loop a fade in/out -- the reveal plays once
|
|
41
|
+
and holds the finished frame, instead of pulling attention away from the
|
|
42
|
+
text every 12s. The safety timeline's progress line keeps a continuous
|
|
43
|
+
moving-dash accent once grown in, since that's motion, not text.
|
|
44
|
+
|
|
3
45
|
## [2.0.0] - 2026-09-23
|
|
4
46
|
|
|
5
47
|
### Added
|
data/PRO.md
CHANGED
|
@@ -24,7 +24,7 @@ work that only pays off at scale and that individuals rarely need.
|
|
|
24
24
|
| History / trend | local JSONL + `trend` | hosted timeline, burndown charts, per-team dashboards |
|
|
25
25
|
| Baseline gate | one file | per-branch baselines, ownership routing (CODEOWNERS) |
|
|
26
26
|
| Multi-database / shards | one DB per run | fan-out across N databases, divergence report |
|
|
27
|
-
| Scheduled scans + alerts |
|
|
27
|
+
| Scheduled scans + alerts | ✅ self-hosted (`rails g schema_reaper:install`: whenever/sidekiq-cron + webhook/email) | ✅ hosted, zero-install + native Slack/Jira routing + per-team alert rules |
|
|
28
28
|
| Mountable dashboard engine | — | ✅ `mount SchemaReaper::Pro::Engine` |
|
|
29
29
|
| SSO, audit log, RBAC | — | ✅ (Enterprise) |
|
|
30
30
|
| Support | GitHub issues, best effort | private issues, SLA |
|
|
@@ -59,7 +59,7 @@ team-scale.
|
|
|
59
59
|
| Tier | Who | Price | Limits |
|
|
60
60
|
|---|---|---|---|
|
|
61
61
|
| **Solo Pro** | 1 dev, unlimited personal/side projects | $9 / mo ($90 / yr) | 3 databases |
|
|
62
|
-
| **Team** | up to 10 devs | $49 / mo ($490 / yr) | 15 databases, Slack/Jira, scheduled scans |
|
|
62
|
+
| **Team** | up to 10 devs | $49 / mo ($490 / yr) | 15 databases, Slack/Jira, hosted scheduled scans |
|
|
63
63
|
| **Business** | up to 50 devs | $199 / mo | unlimited databases, dashboard engine, priority support |
|
|
64
64
|
| **Enterprise** | 50+ / regulated | custom (from ~$12k / yr) | SSO, audit log, RBAC, on-prem license server, SLA |
|
|
65
65
|
|
|
@@ -82,10 +82,15 @@ scheduler + a web app.
|
|
|
82
82
|
1. **Now**: Sponsor button, `PRO.md`, "Pro" section in README (waitlist link).
|
|
83
83
|
2. **v1.1 (OSS)**: MySQL adapter groundwork, real size math — proves demand,
|
|
84
84
|
some lands free.
|
|
85
|
-
3. **
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
85
|
+
3. **v2.0 (OSS, shipped)**: self-hosted scheduled scans + webhook/email alerts
|
|
86
|
+
(`rails g schema_reaper:install`) — this landed free rather than Pro; it's
|
|
87
|
+
the kind of single-app automation the "anything a single app genuinely
|
|
88
|
+
needs stays free" principle below is about.
|
|
89
|
+
4. **Pro 0.1**: private gem = MySQL + Slack/Jira reporters + hosted scan
|
|
90
|
+
runner + license key. Sell to the first 10 teams at half price for
|
|
91
|
+
feedback.
|
|
92
|
+
5. **Pro 0.2**: dashboard engine, multi-DB fan-out, per-branch baselines.
|
|
93
|
+
6. **SaaS beta**: once ~25 paying teams self-host.
|
|
89
94
|
|
|
90
95
|
---
|
|
91
96
|
|
data/README.md
CHANGED
|
@@ -1,7 +1,18 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
<img src="docs/assets/hero.svg" width="100%" alt="schema_reaper — find and safely remove the dead columns, tables and indexes your Rails + PostgreSQL app no longer uses" />
|
|
4
|
+
|
|
5
|
+
[](https://rubygems.org/gems/schema_reaper)
|
|
6
|
+
[](https://rubygems.org/gems/schema_reaper)
|
|
7
|
+
[](https://github.com/aksshatt/schema_reaper/actions/workflows/main.yml)
|
|
8
|
+
[](LICENSE.txt)
|
|
9
|
+
[](schema_reaper.gemspec)
|
|
10
|
+
[-336791?logo=postgresql&logoColor=white)](#install)
|
|
11
|
+
|
|
12
|
+
| ⚡ [Quickstart](#quickstart) | ⚙️ [Usage](#usage) | 🤖 [Production automation](#production-automation) | 🔍 [Analyzers](#analyzers) | 🛡️ [Safety model](#safety-model) | ✅ [CI](#ci) | 🧩 [Configuration](#configuration) | 💼 [Pro](#pro-for-teams) |
|
|
13
|
+
|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
|
14
|
+
|
|
15
|
+
</div>
|
|
5
16
|
|
|
6
17
|
`schema_reaper` reads your **live PostgreSQL schema and planner statistics** and
|
|
7
18
|
cross-references them against a **static scan of your codebase** (Ruby via the
|
|
@@ -10,129 +21,408 @@ Prism AST, plus views and SQL string literals). Optionally it also fuses in a
|
|
|
10
21
|
production. Every finding is scored by confidence and severity, carries an
|
|
11
22
|
estimate of the disk it reclaims, and comes with a concrete fix.
|
|
12
23
|
|
|
13
|
-
|
|
24
|
+
It finds three kinds of schema debt:
|
|
25
|
+
|
|
26
|
+
<picture>
|
|
27
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/debt-dark.svg">
|
|
28
|
+
<img src="docs/assets/debt-light.svg" width="100%" alt="Dead weight — columns and tables nothing references any more (dead_column, dead_table). Index trouble — indexes nobody queries, indexes a wider one already covers, and foreign keys with no index (unused_index, duplicate_index, missing_fk_index). Degenerate data — columns that are always NULL or hold one value in every row (always_null_column, single_value_column).">
|
|
29
|
+
</picture>
|
|
30
|
+
|
|
31
|
+
## Quickstart
|
|
14
32
|
|
|
15
33
|
```ruby
|
|
16
34
|
# Gemfile
|
|
17
|
-
gem "schema_reaper"
|
|
35
|
+
gem "schema_reaper"
|
|
18
36
|
```
|
|
19
37
|
|
|
20
|
-
```
|
|
38
|
+
```sh
|
|
21
39
|
bundle install
|
|
40
|
+
bundle exec schema_reaper scan
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
In a Rails app that's it — the connection comes from `config/database.yml`.
|
|
44
|
+
You get a report like this:
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<img src="docs/assets/terminal.svg" width="100%" alt="Terminal output of bundle exec schema_reaper scan: 3 findings across 2 tables — users.team_id has no covering index (90%, medium, fix: add_index :users, :team_id); users.api_key is NULL in every row (85%, high, 46.9 KB reclaimable); table stale_exports has no model or query reference and holds 0 rows (85%, high, fix: drop_table :stale_exports after confirming no external consumer)." />
|
|
48
|
+
</p>
|
|
49
|
+
|
|
50
|
+
> [!TIP]
|
|
51
|
+
> Scan a database that has served real traffic — a production read-replica or a
|
|
52
|
+
> recent snapshot. `unused_index` needs query history (it skips itself on a
|
|
53
|
+
> fresh database and tells you why), and the data analyzers read `pg_stats`,
|
|
54
|
+
> which PostgreSQL only fills in after `ANALYZE`.
|
|
55
|
+
|
|
56
|
+
## How it works
|
|
57
|
+
|
|
58
|
+
<picture>
|
|
59
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/pipeline-dark.svg">
|
|
60
|
+
<img src="docs/assets/pipeline-light.svg" width="100%" alt="Three inputs — the live PostgreSQL schema and statistics, a static scan of your codebase, and an optional runtime signal — feed seven analyzers. Their scored findings become reports in four formats, a CI baseline gate, scheduled webhook and email reports, and staged removal migrations.">
|
|
61
|
+
</picture>
|
|
62
|
+
|
|
63
|
+
Findings that say the same thing about different tables are rolled up into one
|
|
64
|
+
entry; when several analyzers flag the same column, the strongest finding wins
|
|
65
|
+
and notes the others that agreed. `schema_reaper` never changes your database —
|
|
66
|
+
it only reads, and hands you migrations to review.
|
|
67
|
+
|
|
68
|
+
## Install
|
|
69
|
+
|
|
70
|
+
```ruby
|
|
71
|
+
# Gemfile
|
|
72
|
+
gem "schema_reaper"
|
|
22
73
|
```
|
|
23
74
|
|
|
24
|
-
|
|
25
|
-
|
|
75
|
+
| requirement | notes |
|
|
76
|
+
|---|---|
|
|
77
|
+
| Ruby >= 2.7 | CI-tested on 2.7 – 3.3 |
|
|
78
|
+
| PostgreSQL + the `pg` gem | uses your app's own `pg` (every Rails + PostgreSQL app already bundles it); add `gem "pg"` for standalone CLI use. Tables in the `public` schema are scanned. MySQL is on the [roadmap](#roadmap). |
|
|
79
|
+
| Rails | optional for the CLI; the rake tasks and production automation need it, and the runtime tracker needs ActiveRecord |
|
|
80
|
+
|
|
81
|
+
> [!IMPORTANT]
|
|
82
|
+
> **Only scanning locally or in CI?** Scope it out of your production bundle:
|
|
83
|
+
> `gem "schema_reaper", group: :development`.
|
|
84
|
+
>
|
|
85
|
+
> **Planning to use [production automation](#production-automation)?** Leave it
|
|
86
|
+
> unscoped. Most deploys exclude the development and test groups
|
|
87
|
+
> (`BUNDLE_WITHOUT=development:test`), so a dev-scoped gem is simply absent when
|
|
88
|
+
> the scheduled scan tries to run. The install generator warns you about this.
|
|
89
|
+
|
|
90
|
+
<details>
|
|
91
|
+
<summary><b>How the database connection is resolved</b></summary>
|
|
92
|
+
|
|
93
|
+
<br>
|
|
26
94
|
|
|
27
95
|
1. `database_url:` in `.schema_reaper.yml`
|
|
28
96
|
2. `ENV["DATABASE_URL"]`
|
|
29
|
-
3. `config/database.yml` for the current environment (`SCHEMA_REAPER_ENV
|
|
30
|
-
`RAILS_ENV`, default `development`) — ERB and YAML aliases
|
|
31
|
-
are Rails 6+ multi-database sections
|
|
97
|
+
3. `config/database.yml` for the current environment (`SCHEMA_REAPER_ENV`, then
|
|
98
|
+
`RAILS_ENV`, then `RACK_ENV`, default `development`) — ERB and YAML aliases
|
|
99
|
+
are handled, as are Rails 6+ multi-database sections (the `primary` entry is
|
|
100
|
+
used)
|
|
32
101
|
|
|
33
|
-
|
|
102
|
+
</details>
|
|
34
103
|
|
|
35
104
|
## Usage
|
|
36
105
|
|
|
37
|
-
```
|
|
106
|
+
```sh
|
|
38
107
|
bundle exec schema_reaper scan # grouped terminal report
|
|
39
108
|
bundle exec schema_reaper scan --format markdown # PR-comment table
|
|
40
109
|
bundle exec schema_reaper scan --format sarif # GitHub code scanning
|
|
41
110
|
bundle exec schema_reaper scan --format json
|
|
42
|
-
bundle exec schema_reaper scan --ci # exit 1 on
|
|
111
|
+
bundle exec schema_reaper scan --ci # exit 1 on findings not in the baseline
|
|
43
112
|
bundle exec schema_reaper scan --min-confidence 0.8
|
|
44
|
-
bundle exec schema_reaper scan --no-color # force plain output
|
|
45
113
|
bundle exec schema_reaper baseline # accept current findings
|
|
46
114
|
bundle exec schema_reaper trend # snapshot + progress delta
|
|
47
115
|
bundle exec schema_reaper generate-migration users legacy_api_token
|
|
48
116
|
```
|
|
49
117
|
|
|
50
|
-
|
|
51
|
-
|
|
118
|
+
<details>
|
|
119
|
+
<summary><b>All <code>scan</code> options</b></summary>
|
|
120
|
+
|
|
121
|
+
<br>
|
|
122
|
+
|
|
123
|
+
| option | default | what it does |
|
|
124
|
+
|---|---|---|
|
|
125
|
+
| `--format` | `table` | `table`, `json`, `markdown` or `sarif` |
|
|
126
|
+
| `--ci` | off | exit 1 when a finding isn't in `.schema_reaper/baseline.json` |
|
|
127
|
+
| `--min-confidence N` | `0.0` | hide findings below this confidence (0.0 – 1.0) |
|
|
128
|
+
| `--record` | off | also append this run to the history log used by `trend` |
|
|
129
|
+
| `--color` / `--no-color` | auto | force colour on or off for the table report |
|
|
130
|
+
| `--config PATH` | `.schema_reaper.yml` | config file to load (works on every command) |
|
|
131
|
+
|
|
132
|
+
Colour is automatic on a terminal, and off when output is piped or `NO_COLOR` is set.
|
|
133
|
+
|
|
134
|
+
</details>
|
|
135
|
+
|
|
136
|
+
<details>
|
|
137
|
+
<summary><b>Rake tasks (Rails)</b></summary>
|
|
138
|
+
|
|
139
|
+
<br>
|
|
140
|
+
|
|
141
|
+
The railtie adds these to any Rails app with the gem in its bundle:
|
|
142
|
+
|
|
143
|
+
| task | what it does |
|
|
144
|
+
|---|---|
|
|
145
|
+
| `rake schema_reaper:scan` | same as `schema_reaper scan`; pick a format with `FORMAT=json` etc. |
|
|
146
|
+
| `rake schema_reaper:baseline` | write the current findings to the baseline file |
|
|
147
|
+
| `rake schema_reaper:trend` | record a snapshot and print the trend |
|
|
148
|
+
| `rake schema_reaper:alert` | run one production scan and send the report — what the [schedule](#production-automation) calls |
|
|
149
|
+
|
|
150
|
+
</details>
|
|
151
|
+
|
|
152
|
+
## Production automation
|
|
153
|
+
|
|
154
|
+
*New in v2.0.* Run `schema_reaper` unattended in production and have the report
|
|
155
|
+
land in your team's chat and inbox — no DevOps ticket, and no server access
|
|
156
|
+
beyond a normal deploy.
|
|
157
|
+
|
|
158
|
+
```sh
|
|
159
|
+
bin/rails generate schema_reaper:install
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
<picture>
|
|
163
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/automation-dark.svg">
|
|
164
|
+
<img src="docs/assets/automation-light.svg" width="100%" alt="whenever or cron runs rake schema_reaper:alert, sidekiq-cron enqueues the job directly, and the token-protected manual trigger POST /internal/schema_scan enqueues it on demand. All three run SchemaReaper::ScanJob, which scans the live database and your code; the Notifier sends the report to a Slack-compatible webhook and by email through your ApplicationMailer.">
|
|
165
|
+
</picture>
|
|
166
|
+
|
|
167
|
+
| generated | purpose |
|
|
168
|
+
|---|---|
|
|
169
|
+
| `config/initializers/schema_reaper.rb` | where reports go (webhook and/or email) |
|
|
170
|
+
| an entry in `config/schedule.rb` **or** `config/schedule.yml` | the quarterly scheduled scan (whenever or sidekiq-cron) |
|
|
171
|
+
| `app/controllers/schema_reaper_controller.rb` + a route | a token-protected "scan now" endpoint |
|
|
172
|
+
| a trigger token, printed once | you paste it into your encrypted credentials |
|
|
173
|
+
|
|
174
|
+
Re-running the generator is safe: every step checks for its own marker and
|
|
175
|
+
skips itself rather than duplicating anything. Commit the files, deploy, and
|
|
176
|
+
check it works end to end with one manual run in production:
|
|
177
|
+
|
|
178
|
+
```sh
|
|
179
|
+
bin/rails schema_reaper:alert
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
### 1. Where reports go
|
|
52
183
|
|
|
184
|
+
The generated initializer is commented out, so an unfilled config is a safe
|
|
185
|
+
no-op, not an error. Uncomment one or both:
|
|
186
|
+
|
|
187
|
+
```ruby
|
|
188
|
+
SchemaReaper::AlertConfig.configure do |config|
|
|
189
|
+
# config.emails = %w[dev1@example.com dev2@example.com]
|
|
190
|
+
# config.webhook_url = "https://hooks.slack.com/services/T000/B000/XXXX"
|
|
191
|
+
end
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Both channels fire when both are set — not a fallback chain, but two audiences:
|
|
195
|
+
a dev-facing chat channel, and an inbox record for people who don't watch chat.
|
|
196
|
+
|
|
197
|
+
- **Webhook** — a Slack-compatible `{"text": "..."}` JSON POST carrying the
|
|
198
|
+
Markdown report, with a 10-second timeout.
|
|
199
|
+
- **Email** — sent by `SchemaReaper::Mailer`, which inherits your app's
|
|
200
|
+
`ApplicationMailer` (its `default from:`, delivery settings and so on), or
|
|
201
|
+
`ActionMailer::Base` if you don't have one. Delivered with `deliver_later`.
|
|
202
|
+
|
|
203
|
+
Delivery failures are logged with a `[schema_reaper]` prefix and never raised,
|
|
204
|
+
so a flaky endpoint can't fail or retry-storm the scan.
|
|
205
|
+
|
|
206
|
+
### 2. The schedule
|
|
207
|
+
|
|
208
|
+
The generator looks at your `Gemfile.lock` and uses whichever scheduler you
|
|
209
|
+
already run. The default cadence is **quarterly**; edit the generated entry to
|
|
210
|
+
change it.
|
|
211
|
+
|
|
212
|
+
| you use | the generator adds | also needed |
|
|
213
|
+
|---|---|---|
|
|
214
|
+
| [whenever](https://github.com/javan/whenever) | an `every 3.months` block running `rake "schema_reaper:alert"` in `config/schedule.rb` | whenever only writes your crontab when `whenever --update-crontab` runs on the server — usually via its Capistrano recipe. Platforms without a crontab (Heroku, most container hosts) need one of the other rows. |
|
|
215
|
+
| [sidekiq-cron](https://github.com/sidekiq-cron/sidekiq-cron) | a `schema_reaper_scan` entry in `config/schedule.yml` (`0 4 1 */3 *` — 04:00 on the 1st, every 3 months) that enqueues `SchemaReaper::ScanJob` | nothing — sidekiq-cron ≥ 1.6 loads `config/schedule.yml` automatically |
|
|
216
|
+
| neither | nothing — it prints the command instead of guessing | point any scheduler (cron, Heroku Scheduler, a Kubernetes CronJob) at `bin/rails schema_reaper:alert` |
|
|
217
|
+
|
|
218
|
+
`rake schema_reaper:alert` enqueues the scan on your ActiveJob backend. If that
|
|
219
|
+
backend is Rails' in-process `:async` adapter (the default before Rails 8 unless
|
|
220
|
+
you've set one up), the job would die with the short-lived rake process — so
|
|
221
|
+
the task runs the scan inline instead, email included.
|
|
222
|
+
|
|
223
|
+
### 3. The manual trigger
|
|
224
|
+
|
|
225
|
+
For "scan now" without shelling into a box:
|
|
226
|
+
|
|
227
|
+
```sh
|
|
228
|
+
curl -X POST https://your-app.example.com/internal/schema_scan \
|
|
229
|
+
-H "Authorization: Bearer <token>"
|
|
53
230
|
```
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
████░ 85% high dead_table
|
|
68
|
-
no model or query reference · table holds ~0 row(s)
|
|
69
|
-
→ confirm no external consumer, then `drop_table :stale_exports`
|
|
70
|
-
|
|
71
|
-
high 2 medium 1 low 2
|
|
231
|
+
|
|
232
|
+
| response | meaning |
|
|
233
|
+
|---|---|
|
|
234
|
+
| `202 Accepted` | scan enqueued — the report arrives through the channels above |
|
|
235
|
+
| `401 Unauthorized` | missing or wrong token, or no token configured in this environment |
|
|
236
|
+
| `429 Too Many Requests` | a scan was already triggered in the last 5 minutes |
|
|
237
|
+
|
|
238
|
+
The generator prints the token once, with the exact snippet to paste into
|
|
239
|
+
`bin/rails credentials:edit`:
|
|
240
|
+
|
|
241
|
+
```yaml
|
|
242
|
+
schema_reaper:
|
|
243
|
+
trigger_token: <token>
|
|
72
244
|
```
|
|
73
245
|
|
|
74
|
-
|
|
246
|
+
Credentials are decrypted with the `RAILS_MASTER_KEY` your app already has, so
|
|
247
|
+
this adds no new production configuration. The token is checked with a
|
|
248
|
+
constant-time comparison. The controller deliberately doesn't inherit your
|
|
249
|
+
`ApplicationController` and skips CSRF (it's a token-authenticated API endpoint,
|
|
250
|
+
not a form), so your own auth filters don't apply to it.
|
|
251
|
+
|
|
252
|
+
> [!NOTE]
|
|
253
|
+
> The 5-minute cooldown lives in `Rails.cache`, so it's only as shared as your
|
|
254
|
+
> cache store. Redis, Memcached or Solid Cache give one cooldown for the whole
|
|
255
|
+
> app; a file or memory store gives one per host or process; `:null_store`
|
|
256
|
+
> turns it off.
|
|
257
|
+
|
|
258
|
+
### 4. Install-time warnings
|
|
259
|
+
|
|
260
|
+
Two misconfigurations would otherwise fail silently much later, so the
|
|
261
|
+
generator flags them when it runs:
|
|
75
262
|
|
|
76
|
-
|
|
77
|
-
|
|
263
|
+
- the gem is scoped to `group: :development` — the scheduled scan would never
|
|
264
|
+
run in production
|
|
265
|
+
- neither `ApplicationMailer` nor `config.action_mailer.default_options` sets a
|
|
266
|
+
`from:` address — the mailer would fall back to a placeholder sender that most
|
|
267
|
+
SMTP relays reject, so email reports would never arrive
|
|
268
|
+
|
|
269
|
+
> [!WARNING]
|
|
270
|
+
> **Still on 2.0.0?** Two production-automation bugs are fixed on `main` and
|
|
271
|
+
> ship in the next release:
|
|
272
|
+
> - Report emails ignore `ApplicationMailer` and are sent from
|
|
273
|
+
> `schema_reaper@localhost`. Work around it by adding
|
|
274
|
+
> `SchemaReaper::Mailer.default from: "you@your-domain.com"` to
|
|
275
|
+
> `config/initializers/schema_reaper.rb`.
|
|
276
|
+
> - With the `:async` ActiveJob adapter, `rake schema_reaper:alert` enqueues a
|
|
277
|
+
> job that never runs. Use a persistent backend (Sidekiq, GoodJob, Solid
|
|
278
|
+
> Queue, …) for the whenever and plain-cron paths.
|
|
279
|
+
|
|
280
|
+
<details>
|
|
281
|
+
<summary><b>Why not just run it in CI?</b></summary>
|
|
282
|
+
|
|
283
|
+
<br>
|
|
284
|
+
|
|
285
|
+
[`--format sarif` in CI](#ci) is for PR- and staging-level checks. GitHub-hosted
|
|
286
|
+
runners have no network path to your production database by default, and a CI
|
|
287
|
+
database has no production traffic statistics. Production automation is the path
|
|
288
|
+
for watching the real, live database.
|
|
289
|
+
|
|
290
|
+
</details>
|
|
78
291
|
|
|
79
292
|
## Analyzers
|
|
80
293
|
|
|
81
294
|
| type | what it flags | main signal |
|
|
82
295
|
|---|---|---|
|
|
83
296
|
| `dead_column` | column no code path references | static scan (+ runtime) |
|
|
84
|
-
| `dead_table` | table with no model
|
|
85
|
-
| `unused_index` | non-unique index
|
|
86
|
-
| `duplicate_index` | index that is a prefix of a wider one | schema shape |
|
|
87
|
-
| `missing_fk_index` | `*_id` /
|
|
297
|
+
| `dead_table` | table with no model or query reference | static scan + row count |
|
|
298
|
+
| `unused_index` | non-unique index with `idx_scan = 0` (needs query history) | `pg_stat_user_indexes` |
|
|
299
|
+
| `duplicate_index` | index that exactly duplicates another, or is a prefix of a wider one | schema shape |
|
|
300
|
+
| `missing_fk_index` | `*_id` / foreign-key column with no index (polymorphic pairs need a `(type, id)` index) | schema shape |
|
|
88
301
|
| `always_null_column` | `null_frac = 1.0` — no data at all | `pg_stats` |
|
|
89
|
-
| `single_value_column` | one distinct value on a
|
|
302
|
+
| `single_value_column` | one distinct value on a table of 500+ rows | `pg_stats` |
|
|
90
303
|
|
|
91
|
-
Columns owned by common gems
|
|
92
|
-
|
|
93
|
-
|
|
304
|
+
Columns and tables owned by common gems are whitelisted automatically when the
|
|
305
|
+
gem is in your bundle: **devise**, **devise-api**, **paper_trail**,
|
|
306
|
+
**audited**, **friendly_id**, **paranoia** / **acts_as_paranoid**,
|
|
307
|
+
**activestorage**, **actiontext**, **pg_search**, **ahoy_matey** and
|
|
308
|
+
**activeadmin**.
|
|
94
309
|
|
|
95
310
|
## Runtime signal (optional, raises confidence)
|
|
96
311
|
|
|
97
312
|
Static analysis alone can't see metaprogrammed access, so `dead_column`
|
|
98
|
-
confidence is capped at **0.6** without runtime data. To lift the cap
|
|
313
|
+
confidence is capped at **0.6** without runtime data. To lift the cap, sample
|
|
314
|
+
real column reads:
|
|
99
315
|
|
|
100
316
|
```ruby
|
|
101
|
-
# config/initializers
|
|
317
|
+
# config/initializers/schema_reaper_tracker.rb
|
|
102
318
|
SchemaReaper::Runtime::Tracker.install!(
|
|
103
319
|
store: SchemaReaper::Runtime::Store.new(path: ".schema_reaper/runtime.jsonl"),
|
|
104
320
|
sample_rate: 0.05
|
|
105
321
|
)
|
|
106
322
|
```
|
|
107
323
|
|
|
108
|
-
or, in Rails, boot with `SCHEMA_REAPER_TRACK=1
|
|
324
|
+
or, in Rails, boot with `SCHEMA_REAPER_TRACK=1` (and optionally
|
|
325
|
+
`SCHEMA_REAPER_SAMPLE=0.05` for the sample rate). Let it run in staging or
|
|
109
326
|
production for a couple of weeks. A column unseen in **both** code and
|
|
110
|
-
|
|
327
|
+
≥ 14 observed days of runtime data (`min_age_days` in
|
|
328
|
+
[`.schema_reaper.yml`](#configuration)) reaches **0.9** confidence (0.8 if it's
|
|
329
|
+
`NOT NULL`).
|
|
330
|
+
|
|
331
|
+
> [!NOTE]
|
|
332
|
+
> This is separate from [production automation](#production-automation): the
|
|
333
|
+
> tracker raises confidence in the findings a scan already makes, while
|
|
334
|
+
> production automation runs the scan on a schedule and delivers the report.
|
|
335
|
+
> Most teams want both.
|
|
111
336
|
|
|
112
337
|
## Safety model
|
|
113
338
|
|
|
114
|
-
|
|
339
|
+
<picture>
|
|
340
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/safety-dark.svg">
|
|
341
|
+
<img src="docs/assets/safety-light.svg" width="100%" alt="generate-migration writes two migrations. Step 1 adds the column to ignored_columns and is deployed; nothing is dropped. Once it has soaked in production and nothing reads the column, step 2 runs remove_column, which is irreversible.">
|
|
342
|
+
</picture>
|
|
115
343
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
2. **Drop** — run only after step 1 has soaked in production and nothing broke.
|
|
344
|
+
`schema_reaper` never drops anything itself. For a dead column,
|
|
345
|
+
`generate-migration users legacy_api_token` writes a pair:
|
|
119
346
|
|
|
120
|
-
|
|
121
|
-
`
|
|
347
|
+
1. **`…_ignore_users_legacy_api_token.rb`** — a no-op migration that reminds
|
|
348
|
+
you to add `self.ignored_columns += %w[legacy_api_token]` to the model. Deploy
|
|
349
|
+
that. Nothing is dropped; ActiveRecord just stops selecting the column.
|
|
350
|
+
2. **`…_drop_users_legacy_api_token.rb`** — the `remove_column`. Run it only
|
|
351
|
+
after step 1 has soaked in production and nothing broke. Its `down` raises
|
|
352
|
+
`ActiveRecord::IrreversibleMigration` on purpose.
|
|
353
|
+
|
|
354
|
+
The data-driven fixes are cautious too: `always_null_column` asks you to
|
|
355
|
+
confirm with a `SELECT count(...)` first, `single_value_column` to check the
|
|
356
|
+
value isn't a meaningful default, and `dead_table` to rule out external
|
|
357
|
+
consumers.
|
|
122
358
|
|
|
123
359
|
## CI
|
|
124
360
|
|
|
361
|
+
<picture>
|
|
362
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/ci-dark.svg">
|
|
363
|
+
<img src="docs/assets/ci-light.svg" width="100%" alt="On each pull request, schema_reaper scan --ci compares the findings with the committed baseline.json. It passes when every finding is already in the baseline and exits 1 when the change adds new dead weight.">
|
|
364
|
+
</picture>
|
|
365
|
+
|
|
125
366
|
```yaml
|
|
126
|
-
# .github/workflows/schema_reaper.yml
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
367
|
+
# .github/workflows/schema_reaper.yml (the relevant parts)
|
|
368
|
+
permissions:
|
|
369
|
+
contents: read
|
|
370
|
+
security-events: write # required by upload-sarif
|
|
371
|
+
|
|
372
|
+
# ...job setup: Ruby, a PostgreSQL service, DATABASE_URL...
|
|
373
|
+
steps:
|
|
374
|
+
- run: bin/rails db:schema:load
|
|
375
|
+
- run: bundle exec schema_reaper scan --ci --format sarif > reaper.sarif
|
|
376
|
+
- uses: github/codeql-action/upload-sarif@v4
|
|
377
|
+
if: always() # upload even when --ci fails the step above
|
|
378
|
+
with: { sarif_file: reaper.sarif }
|
|
130
379
|
```
|
|
131
380
|
|
|
132
|
-
Commit `.schema_reaper/baseline.json` so the job
|
|
133
|
-
*new* dead weight.
|
|
381
|
+
Commit `.schema_reaper/baseline.json` (from `schema_reaper baseline`) so the job
|
|
382
|
+
fails only when a change adds *new* dead weight.
|
|
383
|
+
|
|
384
|
+
> [!NOTE]
|
|
385
|
+
> A freshly loaded CI database has no rows, statistics or query history, so CI
|
|
386
|
+
> catches the schema- and code-shaped findings (`dead_column`, `dead_table`,
|
|
387
|
+
> `duplicate_index`, `missing_fk_index`). The data-driven ones need a real
|
|
388
|
+
> database — that's what [production automation](#production-automation) is for.
|
|
389
|
+
|
|
390
|
+
## Configuration
|
|
391
|
+
|
|
392
|
+
Every key is optional. Drop a `.schema_reaper.yml` in the project root to
|
|
393
|
+
override any of these defaults.
|
|
134
394
|
|
|
135
|
-
|
|
395
|
+
> [!CAUTION]
|
|
396
|
+
> A list you set **replaces** the default list rather than adding to it. Setting
|
|
397
|
+
> `ignore: { tables: [legacy_audit] }` stops ignoring `schema_migrations` and
|
|
398
|
+
> `ar_internal_metadata`, and `always_keep_columns: [uuid]` stops protecting
|
|
399
|
+
> `created_at`, `updated_at` and `type`. Repeat the defaults you still want.
|
|
400
|
+
|
|
401
|
+
<details>
|
|
402
|
+
<summary><b>.schema_reaper.yml — all keys with their defaults</b></summary>
|
|
403
|
+
|
|
404
|
+
<br>
|
|
405
|
+
|
|
406
|
+
```yaml
|
|
407
|
+
database_url: # falls back to DATABASE_URL, then config/database.yml
|
|
408
|
+
database_yml: config/database.yml
|
|
409
|
+
scan_paths: [app, lib, config]
|
|
410
|
+
view_globs: ["app/**/*.erb", "app/**/*.haml", "app/**/*.slim", "app/**/*.jbuilder"]
|
|
411
|
+
ignore:
|
|
412
|
+
tables: [schema_migrations, ar_internal_metadata]
|
|
413
|
+
columns: [] # exact names, or "/regex/" patterns
|
|
414
|
+
always_keep_columns: [id, created_at, updated_at, type]
|
|
415
|
+
gem_awareness: true # auto-whitelist columns owned by known gems
|
|
416
|
+
min_age_days: 14 # days of runtime data before dead_column trusts it
|
|
417
|
+
runtime_log: .schema_reaper/runtime.jsonl
|
|
418
|
+
history_log: .schema_reaper/history.jsonl
|
|
419
|
+
baseline: .schema_reaper/baseline.json
|
|
420
|
+
require: [] # extra files to load, e.g. custom analyzers
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
</details>
|
|
424
|
+
|
|
425
|
+
### Custom analyzers
|
|
136
426
|
|
|
137
427
|
```ruby
|
|
138
428
|
# lib/schema_reaper/analyzers/my_check.rb
|
|
@@ -153,6 +443,10 @@ require:
|
|
|
153
443
|
|
|
154
444
|
## Roadmap
|
|
155
445
|
|
|
446
|
+
> [!TIP]
|
|
447
|
+
> ✅ Self-hosted scheduled scans + alerts shipped in **v2.0** — see
|
|
448
|
+
> [Production automation](#production-automation).
|
|
449
|
+
|
|
156
450
|
- Runtime verdict fusion for index and table findings
|
|
157
451
|
- Orphan-row and `schema.rb`↔DB drift analyzers
|
|
158
452
|
- Disk/$ reclaim from real `pg_total_relation_size`
|
|
@@ -161,12 +455,12 @@ require:
|
|
|
161
455
|
|
|
162
456
|
## Pro (for teams)
|
|
163
457
|
|
|
164
|
-
The gem is free and complete for a single app
|
|
165
|
-
team-scale layer: MySQL
|
|
166
|
-
|
|
167
|
-
`pg_total_relation_size` + $ estimates,
|
|
168
|
-
mountable dashboard engine. See
|
|
169
|
-
open an issue tagged `pro`.
|
|
458
|
+
The gem is free and complete for a single app — including the scheduled scans
|
|
459
|
+
and alerts above. **schema_reaper Pro** adds the team-scale layer: MySQL
|
|
460
|
+
adapter, multi-database fan-out, orphan-row and schema-drift analyzers, native
|
|
461
|
+
Slack/Jira/PR-comment reporters, real `pg_total_relation_size` + $ estimates, a
|
|
462
|
+
hosted zero-install scan runner, and a mountable dashboard engine. See
|
|
463
|
+
[PRO.md](PRO.md). Waitlist / early access: open an issue tagged `pro`.
|
|
170
464
|
|
|
171
465
|
## Sponsor
|
|
172
466
|
|
|
@@ -176,11 +470,14 @@ repo.
|
|
|
176
470
|
|
|
177
471
|
## Development
|
|
178
472
|
|
|
179
|
-
```
|
|
473
|
+
```sh
|
|
180
474
|
bin/setup
|
|
181
475
|
bundle exec rake # rspec + rubocop
|
|
476
|
+
|
|
477
|
+
# opt-in: live-database introspection specs
|
|
478
|
+
SCHEMA_REAPER_TEST_DATABASE_URL=postgres://localhost/schema_reaper_test bundle exec rspec
|
|
182
479
|
```
|
|
183
480
|
|
|
184
481
|
## License
|
|
185
482
|
|
|
186
|
-
MIT.
|
|
483
|
+
[MIT](LICENSE.txt).
|
|
@@ -48,7 +48,7 @@ module SchemaReaper
|
|
|
48
48
|
|
|
49
49
|
say_status :warning,
|
|
50
50
|
"schema_reaper is Gemfile-scoped to group: :development. Most production deploy " \
|
|
51
|
-
"pipelines strip dev/test groups (`
|
|
51
|
+
"pipelines strip dev/test groups (`BUNDLE_WITHOUT=development:test`), so " \
|
|
52
52
|
"the scheduled scan will not run in production until you remove that restriction.",
|
|
53
53
|
:red
|
|
54
54
|
end
|
|
@@ -57,7 +57,8 @@ module SchemaReaper
|
|
|
57
57
|
return if mailer_from_configured?
|
|
58
58
|
|
|
59
59
|
say_status :warning,
|
|
60
|
-
"no ApplicationMailer default `from:` detected
|
|
60
|
+
"no ApplicationMailer default `from:` detected (nor config.action_mailer.default_options). " \
|
|
61
|
+
"SchemaReaper::Mailer falls back to a " \
|
|
61
62
|
"placeholder sender in that case, which most real SMTP relays reject or spam-flag -- " \
|
|
62
63
|
"email reports would silently fail to arrive. If you plan to use the email channel " \
|
|
63
64
|
"(config.emails in the initializer this generator just wrote), set " \
|
|
@@ -152,7 +153,7 @@ module SchemaReaper
|
|
|
152
153
|
# `gem "schema_reaper", group: :development` (or the equivalent
|
|
153
154
|
# `group :development do ... end` block form) in the app's own Gemfile
|
|
154
155
|
# -- see README's current install snippet. A dev-scoped gem is absent
|
|
155
|
-
# from `
|
|
156
|
+
# from `BUNDLE_WITHOUT=development:test` installs, which most
|
|
156
157
|
# production deploy pipelines run, so the scheduled scan silently
|
|
157
158
|
# never runs.
|
|
158
159
|
#
|
|
@@ -171,15 +172,14 @@ module SchemaReaper
|
|
|
171
172
|
false # a Gemfile we can't parse shouldn't block the rest of the generator
|
|
172
173
|
end
|
|
173
174
|
|
|
174
|
-
# Checked against
|
|
175
|
-
#
|
|
176
|
-
#
|
|
177
|
-
#
|
|
178
|
-
#
|
|
175
|
+
# Checked against the class SchemaReaper::Mailer will inherit from --
|
|
176
|
+
# ApplicationMailer when the app has one, ActionMailer::Base otherwise
|
|
177
|
+
# (where `config.action_mailer.default_options = { from: ... }` lands).
|
|
178
|
+
# At generate time the app is already booted, so this reads the same
|
|
179
|
+
# `default_params[:from]` chain Mailer itself will see.
|
|
179
180
|
def mailer_from_configured?
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
::ApplicationMailer.default_params[:from].present?
|
|
181
|
+
parent = defined?(::ApplicationMailer) ? ::ApplicationMailer : ::ActionMailer::Base
|
|
182
|
+
parent.default_params[:from].present?
|
|
183
183
|
rescue StandardError
|
|
184
184
|
true # can't determine -- don't nag over something we're unsure about
|
|
185
185
|
end
|
|
@@ -4,12 +4,12 @@ module SchemaReaper
|
|
|
4
4
|
module Analyzers
|
|
5
5
|
# Flags columns present in the schema but never referenced in code. When a
|
|
6
6
|
# runtime usage log is supplied, its signal is fused in: a column unseen in
|
|
7
|
-
# BOTH code and
|
|
7
|
+
# BOTH code and at least `min_age_days` (default 14) observed days of
|
|
8
|
+
# runtime reaches high confidence.
|
|
8
9
|
class DeadColumn < Base
|
|
9
10
|
Registry.register(self)
|
|
10
11
|
|
|
11
|
-
STATIC_ONLY_CAP
|
|
12
|
-
RUNTIME_MIN_DAYS = 14
|
|
12
|
+
STATIC_ONLY_CAP = 0.6
|
|
13
13
|
|
|
14
14
|
def call
|
|
15
15
|
schema.tables.reject { |t| config.ignore_tables.include?(t.name) }
|
|
@@ -48,7 +48,7 @@ module SchemaReaper
|
|
|
48
48
|
end
|
|
49
49
|
|
|
50
50
|
def confidence_for(col)
|
|
51
|
-
if runtime.present? && runtime.observed_days >=
|
|
51
|
+
if runtime.present? && runtime.observed_days >= min_runtime_days
|
|
52
52
|
col.null ? 0.9 : 0.8
|
|
53
53
|
else
|
|
54
54
|
base = col.null ? 0.5 : 0.4
|
|
@@ -56,6 +56,12 @@ module SchemaReaper
|
|
|
56
56
|
end
|
|
57
57
|
end
|
|
58
58
|
|
|
59
|
+
# How many days of runtime data it takes before "never read at runtime"
|
|
60
|
+
# is trusted. A config built without the key (nil) keeps the default.
|
|
61
|
+
def min_runtime_days
|
|
62
|
+
config.min_age_days || Config::DEFAULTS["min_age_days"]
|
|
63
|
+
end
|
|
64
|
+
|
|
59
65
|
def evidence_for(table, col)
|
|
60
66
|
ev = ["no `#{col.name}` reference found in scanned code"]
|
|
61
67
|
ev << "column is nullable" if col.null
|
|
@@ -1,7 +1,18 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module SchemaReaper
|
|
4
|
+
# Reads a live database's schema and statistics. PostgreSQL only for now.
|
|
4
5
|
module Introspect
|
|
6
|
+
# `pg` is deliberately not a runtime dependency -- the host app already
|
|
7
|
+
# bundles its own (every Rails + PostgreSQL app does), and pinning a second
|
|
8
|
+
# version here would fight it. Without this rescue a missing gem surfaces
|
|
9
|
+
# as a baffling NameError instead.
|
|
10
|
+
def self.require_pg!
|
|
11
|
+
require "pg"
|
|
12
|
+
rescue LoadError
|
|
13
|
+
raise Error, "schema_reaper needs the `pg` gem to talk to PostgreSQL -- add `gem \"pg\"` to your Gemfile"
|
|
14
|
+
end
|
|
15
|
+
|
|
5
16
|
# Reads live schema + planner statistics from PostgreSQL using the `pg` gem
|
|
6
17
|
# directly, so the host app does not need to boot Rails.
|
|
7
18
|
class Postgres
|
|
@@ -22,12 +33,10 @@ module SchemaReaper
|
|
|
22
33
|
COLUMN_SEPARATOR = 31.chr
|
|
23
34
|
|
|
24
35
|
def initialize(url)
|
|
25
|
-
|
|
36
|
+
Introspect.require_pg!
|
|
26
37
|
raise Error, NO_URL if url.nil? || url.empty?
|
|
27
38
|
|
|
28
|
-
@conn =
|
|
29
|
-
rescue PG::Error => e
|
|
30
|
-
raise Error, "could not connect to the database: #{e.message.strip}"
|
|
39
|
+
@conn = connect(url)
|
|
31
40
|
end
|
|
32
41
|
|
|
33
42
|
def call
|
|
@@ -39,6 +48,15 @@ module SchemaReaper
|
|
|
39
48
|
|
|
40
49
|
private
|
|
41
50
|
|
|
51
|
+
# Kept out of #initialize: a `rescue PG::Error` there is evaluated for
|
|
52
|
+
# *any* exception, including require_pg!'s, and would itself raise NameError
|
|
53
|
+
# when pg is the thing that failed to load.
|
|
54
|
+
def connect(url)
|
|
55
|
+
PG.connect(url)
|
|
56
|
+
rescue PG::Error => e
|
|
57
|
+
raise Error, "could not connect to the database: #{e.message.strip}"
|
|
58
|
+
end
|
|
59
|
+
|
|
42
60
|
# Cluster-wide cumulative index scans. Lets the unused-index analyzer tell
|
|
43
61
|
# "this index is never used" apart from "this database has no query
|
|
44
62
|
# history", which look identical at the level of a single idx_scan = 0.
|
|
@@ -22,11 +22,17 @@ module SchemaReaper
|
|
|
22
22
|
# or because a caller genuinely wants email delivery skipped) is
|
|
23
23
|
# respected rather than silently falling back to the auto-detected
|
|
24
24
|
# SchemaReaper::Mailer via `||`.
|
|
25
|
-
|
|
25
|
+
#
|
|
26
|
+
# `mail_delivery:` is :later (enqueue via ActiveJob, the normal case) or
|
|
27
|
+
# :now -- for a caller that is itself running inline in a short-lived
|
|
28
|
+
# process, where an in-process queue would be torn down before the mail
|
|
29
|
+
# job ever ran (see the schema_reaper:alert rake task).
|
|
30
|
+
def initialize(findings, config: AlertConfig.instance, http: Net::HTTP, mailer: :auto, mail_delivery: :later)
|
|
26
31
|
@findings = findings
|
|
27
32
|
@config = config
|
|
28
33
|
@http = http
|
|
29
34
|
@mailer = mailer == :auto ? default_mailer : mailer
|
|
35
|
+
@mail_delivery = mail_delivery
|
|
30
36
|
end
|
|
31
37
|
|
|
32
38
|
def deliver
|
|
@@ -38,8 +44,10 @@ module SchemaReaper
|
|
|
38
44
|
|
|
39
45
|
private
|
|
40
46
|
|
|
47
|
+
# Mailer is autoloaded (see lib/schema_reaper.rb), and it can only be
|
|
48
|
+
# defined when ActionMailer is present -- check that, not Mailer itself.
|
|
41
49
|
def default_mailer
|
|
42
|
-
defined?(
|
|
50
|
+
defined?(::ActionMailer::Base) ? Mailer : nil
|
|
43
51
|
end
|
|
44
52
|
|
|
45
53
|
def report_text
|
|
@@ -80,7 +88,8 @@ module SchemaReaper
|
|
|
80
88
|
return
|
|
81
89
|
end
|
|
82
90
|
|
|
83
|
-
@mailer.report_email(to: @config.emails, report: report_text)
|
|
91
|
+
message = @mailer.report_email(to: @config.emails, report: report_text)
|
|
92
|
+
@mail_delivery == :now ? message.deliver_now : message.deliver_later
|
|
84
93
|
rescue StandardError => e
|
|
85
94
|
log_error("email delivery failed", e)
|
|
86
95
|
end
|
|
@@ -9,9 +9,31 @@ module SchemaReaper
|
|
|
9
9
|
class ScanJob < ActiveJob::Base
|
|
10
10
|
queue_as :default
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
# Rails' :async adapter (the default until Rails 8 unless an app sets up
|
|
13
|
+
# a real backend) runs jobs on a thread pool inside the enqueueing
|
|
14
|
+
# process. Fine in a web server; fatal in a one-shot `rake` process,
|
|
15
|
+
# which exits the moment the task returns and takes the queued job with
|
|
16
|
+
# it -- the scheduled scan would silently never run.
|
|
17
|
+
def self.in_process_queue?
|
|
18
|
+
queue_adapter.instance_of?(::ActiveJob::QueueAdapters::AsyncAdapter)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
# What the scheduled cron entry (`rake schema_reaper:alert`) calls:
|
|
22
|
+
# enqueue normally, but on an in-process queue run inline instead --
|
|
23
|
+
# email included -- so the scan isn't lost when the process exits.
|
|
24
|
+
def self.run_scheduled
|
|
25
|
+
return perform_later unless in_process_queue?
|
|
26
|
+
|
|
27
|
+
puts "[schema_reaper] ActiveJob adapter is :async -- running the scan inline"
|
|
28
|
+
perform_now(deliver_mail_now: true)
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
# `deliver_mail_now:` is for running inline from a process about to exit
|
|
32
|
+
# (see the schema_reaper:alert rake task) -- a deliver_later there would
|
|
33
|
+
# be lost for the same reason as above.
|
|
34
|
+
def perform(deliver_mail_now: false)
|
|
13
35
|
findings = Runner.new.run
|
|
14
|
-
Notifier.new(findings).deliver
|
|
36
|
+
Notifier.new(findings, mail_delivery: deliver_mail_now ? :now : :later).deliver
|
|
15
37
|
end
|
|
16
38
|
end
|
|
17
39
|
end
|
|
@@ -24,8 +24,8 @@ namespace :schema_reaper do
|
|
|
24
24
|
pp history.trend
|
|
25
25
|
end
|
|
26
26
|
|
|
27
|
-
desc "
|
|
27
|
+
desc "Run the production scan+alert job (what the scheduled cron entry calls; runs inline on :async)"
|
|
28
28
|
task alert: :environment do
|
|
29
|
-
SchemaReaper::ScanJob.
|
|
29
|
+
SchemaReaper::ScanJob.run_scheduled
|
|
30
30
|
end
|
|
31
31
|
end
|
data/lib/schema_reaper.rb
CHANGED
|
@@ -31,7 +31,6 @@ require_relative "schema_reaper/migration_generator"
|
|
|
31
31
|
require_relative "schema_reaper/runner"
|
|
32
32
|
require_relative "schema_reaper/alert_config"
|
|
33
33
|
require_relative "schema_reaper/notifier"
|
|
34
|
-
require_relative "schema_reaper/mailer" if defined?(ActionMailer::Base)
|
|
35
34
|
require_relative "schema_reaper/scan_job" if defined?(ActiveJob::Base)
|
|
36
35
|
require_relative "schema_reaper/railtie" if defined?(Rails::Railtie)
|
|
37
36
|
|
|
@@ -40,6 +39,17 @@ require_relative "schema_reaper/railtie" if defined?(Rails::Railtie)
|
|
|
40
39
|
module SchemaReaper
|
|
41
40
|
class Error < StandardError; end
|
|
42
41
|
|
|
42
|
+
# Autoloaded, not required: Mailer picks its superclass (the host app's
|
|
43
|
+
# ApplicationMailer, when there is one) at the moment it is defined. At
|
|
44
|
+
# gem-require time -- Bundler.require, early in boot -- the app's own
|
|
45
|
+
# autoloader isn't set up yet, so ApplicationMailer can never be seen and
|
|
46
|
+
# Mailer would always fall back to ActionMailer::Base, silently losing the
|
|
47
|
+
# app's `default from:`. Deferring to first reference (when a report is
|
|
48
|
+
# actually sent, long after boot) fixes that, and still lets a worker
|
|
49
|
+
# process resolve "SchemaReaper::Mailer" by name when it deserializes the
|
|
50
|
+
# mail delivery job.
|
|
51
|
+
autoload :Mailer, File.expand_path("schema_reaper/mailer", __dir__)
|
|
52
|
+
|
|
43
53
|
REPORTERS = {
|
|
44
54
|
"table" => Reporters::Table,
|
|
45
55
|
"json" => Reporters::Json,
|
data/schema_reaper.gemspec
CHANGED
|
@@ -37,7 +37,7 @@ Gem::Specification.new do |spec|
|
|
|
37
37
|
spec.files = Dir.chdir(__dir__) do
|
|
38
38
|
`git ls-files -z`.split("\x0").reject do |f|
|
|
39
39
|
(File.expand_path(f) == __FILE__) ||
|
|
40
|
-
f.start_with?(*%w[bin/ test/ spec/ features/ .git .github Gemfile])
|
|
40
|
+
f.start_with?(*%w[bin/ test/ spec/ features/ docs/ .git .github Gemfile])
|
|
41
41
|
end
|
|
42
42
|
end
|
|
43
43
|
spec.bindir = "exe"
|