schema_reaper 2.0.0 → 2.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 27824319f6c1b0a06c833b1de1e6c9f627f042964f12920083899aebea81c3ee
-  data.tar.gz: 4a5a7b872774fbb6a4fad373206baa54878d2a1bf77e31f8d24232a45e8e65cb
+  metadata.gz: 65c5a4cc1857d1fbf940b5907cbb7d8e48de74cebb7b69eb6dd1b8e2a0c26bfa
+  data.tar.gz: 679e01d41a797c37dfd94858464127b1ce8789bda1d92cb905f011a2369fe1c8
 SHA512:
-  metadata.gz: 95bfcabf3a810c80deab66508b5f75c04df8149f15e5d134c2977749a893b01e1ba5f471c9774dddad2d431dbe26e6178bb348d973c370b99bdb42f5e1249c18
-  data.tar.gz: 95f99230df34ab0f09f2ccf713552cf02c513d6c67d27715224283d2af107c60d0929c01a102232c468774766f644f1c831ed97db39d61675591a1aa4032244d
+  metadata.gz: ef8476dfea36578ef5c97c43fc97c7334371ce38aa14f0d736028d57e42d370b43363922511676af417d76f5fd9b9b35536e98168253efc9c6a663230c56d8c6
+  data.tar.gz: cf4f22deed425352ac3d0553deefb34495e4c234170f402167866fa445b4b4bf707fc3d078a3a612fc99477ed39b9c152f96f4cdfb45555ecfa7a0eaaab9cd82

data/.rubocop.yml

@@ -31,6 +31,7 @@ Metrics/BlockLength:
     - "*.gemspec"
 
 Metrics/ParameterLists:
+  CountKeywordArgs: false # keyword args name themselves at every call site
   Exclude:
     - "spec/**/*"
 
@@ -45,6 +46,8 @@ Metrics/MethodLength:
 
 Metrics/ClassLength:
   Max: 135
+  Exclude:
+    - "lib/schema_reaper/introspect/postgres.rb" # mostly SQL heredocs, one per catalog query
 
 Metrics/CyclomaticComplexity:
   Max: 12

data/.schema_reaper.yml.example

@@ -2,6 +2,13 @@
 ---
 # database_url: postgres://localhost/myapp_development   # else ENV["DATABASE_URL"]
 
+# PostgreSQL schemas to scan. Tables outside public are reported as schema.table.
+schemas:
+  - public
+
+# Hide findings below this confidence on every command (the CLI flag overrides).
+min_confidence: 0.0
+
 scan_paths:
   - app
   - lib
@@ -33,11 +40,21 @@ gem_awareness: true
 
 # Runtime usage log written by the tracker (see README > Runtime signal).
 runtime_log: .schema_reaper/runtime.jsonl
+# Rotate it to runtime.jsonl.1 past this many bytes; both are read. null: never.
+runtime_log_max_bytes: 52428800
 
 # Append-only snapshot log for `schema_reaper trend`.
 history_log: .schema_reaper/history.jsonl
 
 baseline: .schema_reaper/baseline.json
 
+# Per-file cache of scan results; repeat scans re-parse only changed files.
+scan_cache: .schema_reaper/scan_cache.json
+
+# global: a column name anywhere in code counts as use on every table.
+# model:  a name only in an app/models file counts for that model's table and
+#         the tables it mentions. Stricter; review its findings before acting.
+column_scope: global
+
 # Extra Ruby files to load before analysis -- register custom analyzers here.
 require: []

data/CHANGELOG.md

@@ -1,5 +1,149 @@
 # Changelog
 
+## [2.1.0] - 2026-10-14
+
+### Fixed
+- **Introspection no longer mixes up same-named objects across schemas.**
+  Row counts were read from `pg_class` by name alone, foreign keys were joined
+  on constraint name alone (constraint names are only unique per table), and
+  the primary-key lookup resolved the table through `search_path`. All three
+  are now qualified by schema. Foreign keys are read from `pg_constraint`, so
+  composite keys keep their columns together.
+- **`ignore.columns` now applies to every column analyzer.** It was only
+  honoured by `dead_column`; `always_null_column`, `single_value_column` and
+  `missing_fk_index` ignored it.
+- **Partitioned tables are reported once, as the parent.** Each partition used
+  to appear as its own table -- and, unreferenced by name, got a `dead_table`
+  finding. The parent's row count is now summed from its partitions, and its
+  indexes' usage from the partitions' indexes, so `unused_index` no longer
+  flags every index on a partitioned table (PostgreSQL 12+).
+- **Stats-based findings say when they are estimates.** On tables larger than
+  ANALYZE's 30,000-row sample, `always_null_column` (0.85 -> 0.7) and
+  `single_value_column` (0.6 -> 0.5) lower their confidence and say so, and
+  `single_value_column` no longer claims "the same value in every row" for a
+  column that also holds NULLs.
+- **`missing_fk_index`** finds the referenced table in a non-public table's own
+  schema.
+- **Runtime `observed_days` no longer depends on line order.** Several
+  processes append to one log, so the span is now min..max timestamp rather
+  than first..last line.
+
+### Added
+- `schemas:` config key to scan PostgreSQL schemas besides `public`. Tables
+  outside `public` are reported as `schema.table`; `public` tables keep their
+  bare names, so existing baselines and ignore lists still match.
+- `sequence_exhaustion` analyzer (CYCLE and descending sequences excluded): a serial/IDENTITY sequence past 50% (medium)
+  or 75% (high) of its column type's range -- e.g. an `integer` key fed by a
+  bigint sequence.
+- `duplicate_foreign_key` analyzer: two FK constraints on the same columns
+  referencing the same key, with the same ON UPDATE / ON DELETE, MATCH and
+  deferrability (anything else changes behaviour when dropped). A validated
+  copy is kept over a NOT VALID one.
+- `--only` / `--except` to choose analyzers (custom ones included), and a
+  `min_confidence` config key used by every command.
+- `generate-migration TABLE --index NAME` (concurrent `remove_index` with
+  `disable_ddl_transaction!`; the migration checks at run time whether the
+  index is partitioned, which PostgreSQL can't drop concurrently, and drops it
+  plainly then) and `generate-migration TABLE` (`drop_table`). The
+  `unused_index` / `duplicate_index` fixes say the same for partitioned
+  indexes.
+- The runtime tracker also samples tables read by executed SQL (`track_sql:`,
+  on by default; FROM / JOIN / UPDATE, not INSERT targets, so a write-only
+  table can still be found), so a table read only through raw SQL isn't
+  reported dead.
+- The runtime log rotates to `<path>.1` past `runtime_log_max_bytes` (default
+  50 MB), under a file lock so concurrent workers can't rotate twice; both
+  generations are read.
+- `schema_reaper doctor`: checks the connection, statistics age, tables never
+  `ANALYZE`d, query history, replica status, runtime log length, expired
+  baseline entries, the schema file and alert config.
+- `schema_reaper explain ID` (full or partial id): evidence, confidence,
+  reclaim, schema line, baseline status and the `generate-migration` command.
+- `# schema_reaper:keep TARGET -- reason` comments in code silence findings.
+- `column_scope: model` (opt-in): a column name seen only in a model file
+  counts for that model's table and the tables it mentions, not every table.
+  Models are identified by their full class path (`Admin::User` is not
+  `User`); STI subclasses map to their parent's table (resolved the way Ruby
+  resolves the constant), namespaced models to both possible defaults, and
+  abstract bases, POROs and concerns stay shared.
+- Baseline entries carry an optional reason and expiry (`baseline --reason
+  --expires YYYY-MM-DD`); `scan --ci` fails on expired ones again, and passing
+  `--reason` / `--expires` renews them. `baseline --prune` drops resolved
+  entries without accepting new findings. With `--only` / `--except`,
+  `baseline` replaces or prunes only those analyzers' entries. The 2.0
+  `{"ids": [...]}` file is still read, and the file keeps an `ids` key
+  (unexpired entries) so a 2.0.x gem sharing it still works -- though 2.0.x
+  ignores reasons and expiry, and rewrites the file in the old format if it
+  runs `baseline`.
+- Scheduled reports say what changed since the last one (via `Rails.cache`);
+  `config.alert_on = :changes` sends only then. A report only counts as sent
+  once a channel accepted it -- a non-2xx webhook response is a failure -- so
+  a failed delivery is retried next run instead of suppressed.
+- Slack webhooks get a Block Kit summary instead of a Markdown table Slack
+  can't render; other receivers keep the full `{"text": ...}` report
+  (`config.webhook_format`).
+- Gem awareness covers solid_queue, solid_cache, solid_cable, good_job,
+  delayed_job_active_record, flipper-active_record, acts-as-taggable-on,
+  doorkeeper, noticed, activerecord-session_store, discard, acts_as_list and
+  ancestry. Every analyzer now leaves a gem-owned table's columns alone.
+- Per-file scan cache (`scan_cache:`), so repeat scans only re-parse changed
+  files. Keyed by a SHA-256 digest, so it survives across processes.
+- SARIF results point at the defining line in `db/schema.rb` or
+  `db/structure.sql` instead of the top of `db/schema.rb`.
+- `--color` / `--no-color` works on every command.
+
+### Changed
+- Column byte estimates use `pg_stats.avg_width` scaled by the non-NULL share
+  when the table has been analysed, instead of a fixed per-type guess (16
+  bytes for text/jsonb) that also counted NULLs as full width. An all-NULL
+  column is estimated at ~0 bytes.
+- `unused_index` evidence notes that `idx_scan` is per server, so indexes used
+  only on read replicas look unused on the primary.
+- CI tests Ruby 3.4 and 4.0 too, and runs the live introspection specs against
+  a PostgreSQL 16 service.
+
+## [2.0.1] - 2026-09-23
+
+### Fixed
+- **Report emails now use the app's `ApplicationMailer`.** `SchemaReaper::Mailer`
+  was defined while the gem loaded (`Bundler.require`), before the app's
+  autoloader could see `app/mailers/application_mailer.rb`, so it always
+  inherited `ActionMailer::Base` and sent from the `schema_reaper@localhost`
+  placeholder -- which most SMTP relays reject. It is now autoloaded on first
+  use, after boot. The install generator's `from:` check now also accepts
+  `config.action_mailer.default_options` for apps without an
+  `ApplicationMailer`. 2.0.0 workaround: add
+  `SchemaReaper::Mailer.default from: "..."` to the initializer.
+- **`rake schema_reaper:alert` no longer silently drops the scan on the
+  `:async` ActiveJob adapter** (the default before Rails 8). The job ran on a
+  thread inside the rake process, which exited first -- so the whenever and
+  plain-cron schedules never scanned. The task now runs the scan inline (with
+  email delivered immediately) when the adapter is `:async`, and enqueues as
+  before otherwise.
+- **A missing `pg` gem now gives a clear error.** `pg` is deliberately not a
+  runtime dependency (the gem uses the host app's own), but without it the
+  scan failed with `NameError: uninitialized constant ...::PG` from the
+  connection rescue. It now says to add `gem "pg"`.
+- **`min_age_days` in `.schema_reaper.yml` now takes effect.** It was defined
+  and documented since 1.0.0 but never read -- `dead_column` used its own
+  hardcoded 14-day threshold for trusting runtime data. The analyzer now reads
+  the config key (default still 14, so nothing changes unless you set it).
+
+### Changed
+- README rewritten around a quickstart, with animated SVG illustrations
+  (`docs/assets/`, excluded from the packaged gem) and corrected details on
+  scheduling, the manual trigger's cooldown, CI and gem awareness. Diagrams
+  for each major section; animations loop with a long hold on the finished
+  frame, since GitHub can't start a README image's animation on scroll.
+  Documents the `pg` requirement, that only the `public` schema is scanned, and
+  that lists in `.schema_reaper.yml` replace their defaults, and what
+  `min_age_days` controls.
+- The `automation`, `safety` and `ci` section diagrams carry label text meant
+  to be read, so they no longer loop a fade in/out -- the reveal plays once
+  and holds the finished frame, instead of pulling attention away from the
+  text every 12s. The safety timeline's progress line keeps a continuous
+  moving-dash accent once grown in, since that's motion, not text.
+
 ## [2.0.0] - 2026-09-23
 
 ### Added

data/PRO.md

@@ -24,7 +24,7 @@ work that only pays off at scale and that individuals rarely need.
 | History / trend | local JSONL + `trend` | hosted timeline, burndown charts, per-team dashboards |
 | Baseline gate | one file | per-branch baselines, ownership routing (CODEOWNERS) |
 | Multi-database / shards | one DB per run | fan-out across N databases, divergence report |
-| Scheduled scans + alerts | — | ✅ (cron runner, "queue breached", "new dead column in `billing`") |
+| Scheduled scans + alerts | ✅ self-hosted (`rails g schema_reaper:install`: whenever/sidekiq-cron + webhook/email) | ✅ hosted, zero-install + native Slack/Jira routing + per-team alert rules |
 | Mountable dashboard engine | — | ✅ `mount SchemaReaper::Pro::Engine` |
 | SSO, audit log, RBAC | — | ✅ (Enterprise) |
 | Support | GitHub issues, best effort | private issues, SLA |
@@ -59,7 +59,7 @@ team-scale.
 | Tier | Who | Price | Limits |
 |---|---|---|---|
 | **Solo Pro** | 1 dev, unlimited personal/side projects | $9 / mo ($90 / yr) | 3 databases |
-| **Team** | up to 10 devs | $49 / mo ($490 / yr) | 15 databases, Slack/Jira, scheduled scans |
+| **Team** | up to 10 devs | $49 / mo ($490 / yr) | 15 databases, Slack/Jira, hosted scheduled scans |
 | **Business** | up to 50 devs | $199 / mo | unlimited databases, dashboard engine, priority support |
 | **Enterprise** | 50+ / regulated | custom (from ~$12k / yr) | SSO, audit log, RBAC, on-prem license server, SLA |
 
@@ -82,10 +82,15 @@ scheduler + a web app.
 1. **Now**: Sponsor button, `PRO.md`, "Pro" section in README (waitlist link).
 2. **v1.1 (OSS)**: MySQL adapter groundwork, real size math — proves demand,
    some lands free.
-3. **Pro 0.1**: private gem = MySQL + Slack reporter + scheduled scans + license
-   key. Sell to the first 10 teams at half price for feedback.
-4. **Pro 0.2**: dashboard engine, multi-DB fan-out, per-branch baselines.
-5. **SaaS beta**: once ~25 paying teams self-host.
+3. **v2.0 (OSS, shipped)**: self-hosted scheduled scans + webhook/email alerts
+   (`rails g schema_reaper:install`) — this landed free rather than Pro; it's
+   the kind of single-app automation the "anything a single app genuinely
+   needs stays free" principle below is about.
+4. **Pro 0.1**: private gem = MySQL + Slack/Jira reporters + hosted scan
+   runner + license key. Sell to the first 10 teams at half price for
+   feedback.
+5. **Pro 0.2**: dashboard engine, multi-DB fan-out, per-branch baselines.
+6. **SaaS beta**: once ~25 paying teams self-host.
 
 ---