schema_reaper 1.0.10 → 1.0.11

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f8051cffdf927a39467d51f207093a09be0cc89d5641da5f28f1fc3768b47544
-  data.tar.gz: d79042e789a967b678afdd6808131c16577706fb2726da94cc3b6e95eea5c945
+  metadata.gz: 5039780ddae77f5851cbc16afb53bea80df2eb7328253e18b89c2495feb66218
+  data.tar.gz: e88e86f0cb4d723256505d9a4d7e89010c7cc532df86b57a1190713d71339a90
 SHA512:
-  metadata.gz: a8526fa89f8f9c449ea3638c196c3e9c042a68f27b547b3a471166942e636154205f506ff19504cb45b8aa55070f6d176e5fb0cab1e772ae2202f8d6e6454a7c
-  data.tar.gz: fb9d147aacb72e4c44a25ab57388005d7d17e7779430ade83e098f6c2ddf4878dafd441b3d211341fd71382ebaafa46a0ef26698cdb0f604d8e6d157ea914f95
+  metadata.gz: c4b095e2cc73c98f8415136810c3c00218b741cce186d3ff1d2d4e0063cd1658963aa60d296595bc4a2adc8dffaee982be49b750a8f7280993cc4e68a85fa4e0
+  data.tar.gz: d7437c7dbeb2e0c96f4671c89b0e07e89e0e327bdd741cf6b6ee9c128eedf8c2bf96f58c3d925efe742b94de4632c6204ff363e5a23ddc1d6130aa38eeace44c

data/CHANGELOG.md

@@ -1,5 +1,79 @@
 # Changelog
 
+## [1.0.11] - 2026-09-15
+
+Everything below landed after 1.0.10 was cut, so 1.0.10 on RubyGems contains
+none of it.
+
+### Fixed
+- **Composite index columns were read in table order, not index-key order.**
+  `indexes_for` aggregated with `ORDER BY a.attnum` — a column's position in
+  the *table* — so `(company_id, email)` came back as `email,company_id`.
+  Since `Index#covers?` is a prefix test, this broke `duplicate_index` in both
+  directions: it recommended `remove_index` on indexes that are not redundant
+  (dropping one would degrade queries on its leading column), and missed
+  genuinely redundant ones. `missing_fk_index` read the same corrupted order
+  through `indexed?`. Now joins `unnest(indkey) WITH ORDINALITY`. (#1, mitkush)
+- **`primary_key_for` returned one arbitrary column of a composite key.** It
+  matched `attnum = ANY(indkey)` with no `ORDER BY` and took `.first`, so the
+  other key columns looked like ordinary columns to the analyzers and could be
+  reported as `dead_column` or `single_value_column` — suggesting you drop part
+  of a primary key. Now returns every key column in order; `Table#primary_key?`
+  accepts a name or a list. (#1, mitkush)
+- **Polymorphic associations were reported as unindexed foreign keys.** A
+  `*_id` paired with a `*_type` is only ever queried with the type, so the
+  index that matters is the composite `(type, id)` — but `indexed?` only asked
+  whether the column led *some* index. Rails apps were told to add an index the
+  planner would never use. (#1, mitkush)
+- **`dead_table` missed models behind `-ies` tables and route helpers.**
+  `referenced?` singularised with `sub(/s\z/, "")`, turning `crm_activities`
+  into `crm_activitie`, so `CrmActivity` never matched. Because
+  `drop_findings_on_dead_tables` discards every column- and index-level finding
+  for a table called dead, one bad `dead_table` silently suppressed everything
+  else about that table. Inflection now handles `-ies`/`-sses`/`-xes`, and a
+  table name is also matched inside longer identifiers such as
+  `admin_crm_activities_path`. (#1, mitkush)
+- **Ruby 2.7 was broken despite being the declared floor.** `Config.load`
+  called `YAML.safe_load_file`, which arrived in Psych 3.3 (Ruby 3.0), so every
+  command crashed for any 2.7 user with a `.schema_reaper.yml`.
+  `Console#title` appended to an interpolated string, which is frozen on
+  Ruby <= 2.7. (#2, mitkush)
+- **`config/database.yml` credentials were not URL-escaped.** A password
+  containing `@` made libpq read the rest as the host, reporting
+  `could not translate host name "ss@localhost"` — a hostname the user never
+  configured. Also, when ERB failed to render (commonly
+  `Rails.application.credentials` outside a booted Rails), the resolver fell
+  back to parsing the file *un-rendered* and built a connection URL out of
+  template text; it now resolves nothing so the real error surfaces. (#1,
+  mitkush)
+- **An unknown reclaim estimate was printed as `0 B`.** A byte estimate needs a
+  row count, and `reltuples = -1` means unknown; `Base#finding` turned that into
+  zero, so reports opened with `~0.0 B reclaimable` — "nothing to gain" rather
+  than "cannot say". (#4, mitkush)
+
+### Changed
+- **`unused_index` is skipped when the database has no query history.**
+  `idx_scan = 0` means either "never used" or "this database has answered no
+  queries". On a freshly loaded schema the analyzer reported every non-unique
+  index — 197 of 266 findings on one app. It now compares cluster-wide
+  `idx_scan` against the index count and explains the skip on stderr. (#2,
+  mitkush)
+- **`missing_fk_index` confidence is tiered by evidence.** A declared FK
+  constraint scores 0.9; a `*_id` name with a table it plausibly references
+  scores 0.7; a `*_id` name with nothing to reference scores 0.5 and drops to
+  `:low`. Previously `voter_id` and `upi_id` were reported as confidently as a
+  real constraint, which made `--min-confidence` useless for this analyzer.
+  Nothing is dropped. (#3, mitkush)
+- **The report says what it contains.** The header now carries the finding
+  count, how many tables are affected, and the type breakdown — previously only
+  a tally in the footer, 245 lines below on a large report. (#4, mitkush)
+- **Findings that differ only in the names they mention are rolled up.** 37
+  missing foreign-key indexes spent 111 lines repeating one sentence. They now
+  collapse into a single entry that keeps the confidence bar, states the fix as
+  a template, and lists every target. Nothing is summarised away, and findings
+  whose evidence genuinely differs — every `duplicate_index` names a different
+  index pair — stay itemised. 246 lines to 124 on one app. (#5, mitkush)
+
 ## [1.0.10] - 2026-09-09
 
 Supersedes 1.0.9, which was published to RubyGems from an incomplete cut and

data/README.md

@@ -47,10 +47,13 @@ bundle exec schema_reaper trend                   # snapshot + progress delta
 bundle exec schema_reaper generate-migration users legacy_api_token
 ```
 
-The `scan` report groups findings by table and sorts by confidence:
+The `scan` report rolls up findings that say the same thing about different
+tables, then groups the rest by table and sorts by confidence:
 
 ```
-  schema_reaper  5 findings  ~93.8 KB reclaimable
+  schema_reaper  5 findings across 2 tables
+  missing_fk_index 3 · always_null_column 1 · dead_column 1
+  ~93.8 KB reclaimable
 
   users
     █████  90%  medium  missing_fk_index     team_id

data/lib/schema_reaper/analyzers/base.rb

@@ -58,12 +58,15 @@ module SchemaReaper
       end
 
       # Builds a Finding, filling in reclaimable_bytes from the row count.
+      # An unknown row count leaves reclaimable_bytes nil: "we cannot say" is
+      # not the same claim as "zero bytes", and reporters need to tell them
+      # apart. Finding#reclaimable_bytes still reads 0 for arithmetic.
       def finding(table:, bytes_per_row: 0, row_count: nil, **rest)
-        rows = row_count || schema.table(table)&.row_count || 0
+        rows = row_count || schema.table(table)&.row_count
         Finding.new(
           table: table,
           bytes_per_row: bytes_per_row,
-          reclaimable_bytes: bytes_per_row * rows,
+          reclaimable_bytes: rows && (bytes_per_row * rows),
           **rest
         )
       end

data/lib/schema_reaper/analyzers/missing_fk_index.rb

@@ -1,5 +1,7 @@
 # frozen_string_literal: true
 
+require "set"
+
 module SchemaReaper
   module Analyzers
     # A foreign-key column with no index: every parent delete/update scans the
@@ -12,6 +14,15 @@ module SchemaReaper
     class MissingFkIndex < Base
       Registry.register(self)
 
+      # How sure we are that the column really is a foreign key:
+      #   a declared constraint is proof;
+      #   a *_id name plus a table it plausibly points at is good evidence;
+      #   a *_id name alone is a guess -- voter_id and upi_id are not foreign
+      #   keys, they just end in _id.
+      CONSTRAINT_CONFIDENCE = 0.9
+      REFERENT_CONFIDENCE = 0.7
+      NAME_ONLY_CONFIDENCE = 0.5
+
       def call
         schema.tables.reject { |t| config.ignore_tables.include?(t.name) }
               .flat_map { |t| missing_in(t) }
@@ -27,7 +38,7 @@ module SchemaReaper
           type_col = polymorphic_type_for(table, col)
           next if type_col && polymorphic_indexed?(table, type_col, col)
 
-          finding_for(table, col, type_col)
+          finding_for(table, col, type_col, declared: table.foreign_keys.include?(col))
         end
       end
 
@@ -40,23 +51,45 @@ module SchemaReaper
         table.column(col)&.always_null? || false
       end
 
-      def finding_for(table, col, type_col)
+      def finding_for(table, col, type_col, declared:)
+        referent = declared ? nil : referent_table(col)
+        confidence = confidence_for(declared, referent)
+
         finding(
           type: :missing_fk_index,
           table: table.name,
           column: col,
-          severity: :medium,
-          confidence: 0.9,
+          severity: confidence >= REFERENT_CONFIDENCE ? :medium : :low,
+          confidence: confidence,
           bytes_per_row: 0,
-          evidence: [evidence_for(col, type_col)],
+          evidence: [evidence_for(col, type_col, declared, referent)],
           suggested_fix: fix_for(table, col, type_col)
         )
       end
 
-      def evidence_for(col, type_col)
-        return "#{col} is a foreign key with no covering index" unless type_col
+      def confidence_for(declared, referent)
+        return CONSTRAINT_CONFIDENCE if declared
+        return REFERENT_CONFIDENCE if referent
+
+        NAME_ONLY_CONFIDENCE
+      end
+
+      def evidence_for(col, type_col, declared, referent)
+        subject =
+          if type_col
+            "#{col} is a polymorphic association with #{type_col} and has no (#{type_col}, #{col}) index"
+          else
+            "#{col} has no covering index"
+          end
+
+        "#{subject} (#{provenance(declared, referent)})"
+      end
+
+      def provenance(declared, referent)
+        return "declared foreign key constraint" if declared
+        return "no FK constraint; named like a reference to `#{referent}`" if referent
 
-        "#{col} is a polymorphic association with #{type_col} and has no (#{type_col}, #{col}) index"
+        "no FK constraint and no table it plausibly references -- matched on the _id suffix alone"
       end
 
       def fix_for(table, col, type_col)
@@ -69,6 +102,24 @@ module SchemaReaper
         (table.foreign_keys + table.column_names.grep(/_id\z/)).uniq
       end
 
+      # A table the column plausibly points at: account_id -> accounts,
+      # company_id -> companies, address_id -> addresses. Absence is the signal
+      # that a *_id name is not really a foreign key.
+      def referent_table(col)
+        return nil unless col.end_with?("_id")
+
+        base = col[0..-4]
+        return nil if base.empty?
+
+        candidates = [base, "#{base}s", "#{base}es"]
+        candidates << "#{base[0..-2]}ies" if base.end_with?("y")
+        candidates.find { |name| table_names.include?(name) }
+      end
+
+      def table_names
+        @table_names ||= schema.tables.to_set(&:name)
+      end
+
       def indexed?(table, col)
         table.indexes.any? { |ix| ix.columns.first == col }
       end

data/lib/schema_reaper/config.rb

@@ -28,10 +28,19 @@ module SchemaReaper
     }.freeze
 
     def self.load(path = ".schema_reaper.yml")
-      raw = File.exist?(path) ? (YAML.safe_load_file(path) || {}) : {}
+      raw = File.exist?(path) ? (safe_load_file(path) || {}) : {}
       new(deep_merge(DEFAULTS, raw), root: File.dirname(File.expand_path(path)))
     end
 
+    # Psych.safe_load_file arrived in Psych 3.3 (Ruby 3.0), but the gem supports
+    # Ruby 2.7. Reading the file ourselves is equivalent -- safe_load_file
+    # defaults to aliases: false, which is what safe_load does too.
+    def self.safe_load_file(path)
+      return YAML.safe_load_file(path) if YAML.respond_to?(:safe_load_file)
+
+      YAML.safe_load(File.read(path))
+    end
+
     def self.deep_merge(base, override)
       base.merge(override) do |_key, a, b|
         a.is_a?(Hash) && b.is_a?(Hash) ? deep_merge(a, b) : b

data/lib/schema_reaper/database_url.rb

@@ -40,7 +40,11 @@ module SchemaReaper
         YAML.safe_load(rendered, [], [], true)
       end
     rescue StandardError
-      YAML.load_file(file)
+      # ERB that reaches into Rails -- Rails.application.credentials is the
+      # common one -- cannot render outside a booted Rails process. Resolving
+      # nothing yields a clear "no database_url configured"; parsing the file
+      # un-rendered would build a connection URL out of template text.
+      nil
     end
 
     # Rails 6+ allows `development: { primary: {...}, replica: {...} }`. Pick the
@@ -67,12 +71,19 @@ module SchemaReaper
       db = section["database"]
       return nil if db.to_s.empty?
 
-      userinfo = [section["username"], section["password"]].compact.join(":")
+      userinfo = [section["username"], section["password"]].compact.map { |v| escape(v) }.join(":")
       host = section["host"].to_s
       hostport = host.empty? ? "" : "#{host}#{":#{section["port"]}" if section["port"]}"
       auth = userinfo.empty? ? "" : "#{userinfo}@"
 
       "postgresql://#{auth}#{hostport}/#{db}"
     end
+
+    # Passwords routinely contain characters that are structural in a URL.
+    # An unescaped "@" makes libpq read the rest as the host, so it reports
+    # a bogus hostname rather than a credential problem.
+    def escape(value)
+      ERB::Util.url_encode(value.to_s)
+    end
   end
 end

data/lib/schema_reaper/finding.rb

@@ -36,6 +36,12 @@ module SchemaReaper
       self[:reclaimable_bytes] || 0
     end
 
+    # False when the row count was unavailable, so the byte estimate is
+    # unknown rather than zero.
+    def reclaim_known?
+      !self[:reclaimable_bytes].nil?
+    end
+
     # Column and/or index this finding points at, without the table name
     # (callers that group by table already show it). nil for a whole-table
     # finding.

data/lib/schema_reaper/reporters/console.rb

@@ -21,7 +21,9 @@ module SchemaReaper
 
       # "  schema_reaper trend  ·  3 snapshots"
       def title(command, meta = nil)
-        line = "  #{@a.paint("schema_reaper", :bold)} #{@a.paint(command, :bold)}"
+        # +"" : on Ruby <= 2.7 an interpolated string whose parts are all frozen
+        # literals is itself frozen, so the append below raises FrozenError.
+        line = +"  #{@a.paint("schema_reaper", :bold)} #{@a.paint(command, :bold)}"
         line << "  #{@a.paint("·  #{meta}", :dim)}" if meta
         @out.puts
         @out.puts line

data/lib/schema_reaper/reporters/rollup.rb

@@ -0,0 +1,55 @@
+# frozen_string_literal: true
+
+module SchemaReaper
+  module Reporters
+    # Groups findings that say the same thing about different targets.
+    #
+    # A report of 37 missing foreign-key indexes repeats one sentence 37 times,
+    # varying only the table and column. Collapsing those into a single entry
+    # with its targets listed keeps every fact and drops the repetition.
+    module Rollup
+      # Findings that differ only in which table and column they name carry no
+      # new information per line. At or above this many, roll them up.
+      THRESHOLD = 4
+
+      module_function
+
+      # @return [Array(Array<[key, Array<Finding>]>, Array<Finding>)]
+      #   the groups worth rolling up (largest first), and the findings to
+      #   spell out individually
+      def partition(findings)
+        by_shape = findings.group_by { |f| key_for(f) }
+        big, small = by_shape.partition { |_key, group| group.size >= THRESHOLD }
+        [big.sort_by { |_key, group| -group.size }, small.flat_map { |_key, group| group }]
+      end
+
+      def key_for(finding)
+        [finding.type, shape(evidence_line(finding), finding), shape(finding.suggested_fix, finding)]
+      end
+
+      def evidence_line(finding)
+        finding.evidence.join(" · ")
+      end
+
+      # Masks the names a finding is about, each with its own placeholder, so
+      # two findings saying the same thing about different targets compare
+      # equal -- and so the rolled-up fix still reads as a template.
+      #
+      # The lookarounds matter: without them, masking
+      # index_users_on_team_id would also eat it out of
+      # index_users_on_team_id_and_state, making two findings about different
+      # covering indexes look identical.
+      def shape(text, finding)
+        replacements(finding).reduce(text) do |acc, (name, token)|
+          acc.gsub(/(?<!\w)#{Regexp.escape(name)}(?!\w)/, token)
+        end
+      end
+
+      def replacements(finding)
+        [[finding.table, "<table>"], [finding.column, "<column>"], [finding.index, "<index>"]]
+          .reject { |name, _| name.nil? || name.empty? }
+          .sort_by { |name, _| -name.length }
+      end
+    end
+  end
+end

data/lib/schema_reaper/reporters/table.rb

@@ -2,15 +2,20 @@
 
 require_relative "bytes"
 require_relative "ansi"
+require_relative "rollup"
 
 module SchemaReaper
   module Reporters
-    # Human-readable terminal report: a summary line, findings grouped by table
-    # and sorted by confidence, then a severity/type tally. Colour is used only
-    # on an interactive terminal (see Ansi).
+    # Human-readable terminal report: a summary line, then findings. Findings
+    # that say the same thing about different tables are rolled up into one
+    # entry; the rest are grouped by table and sorted by confidence. Colour is
+    # used only on an interactive terminal (see Ansi).
     class Table
       SEV_ORDER = { high: 0, medium: 1, low: 2 }.freeze
 
+      # Width to wrap the target list of a rolled-up entry.
+      TARGET_LINE_WIDTH = 92
+
       def initialize(findings, io: $stdout, color: nil)
         @findings = findings
         @io = io
@@ -21,7 +26,8 @@ module SchemaReaper
         return render_clean if @findings.empty?
 
         header
-        grouped.each { |table, group| render_table(table, group) }
+        rolled_up.each { |key, group| render_rollup(key, group) }
+        grouped(itemised).each { |table, group| render_table(table, group) }
         footer
       end
 
@@ -33,14 +39,81 @@ module SchemaReaper
 
       def header
         @io.puts
-        @io.puts "  #{@a.paint("schema_reaper", :bold)}  " \
-                 "#{@a.paint("#{@findings.size} finding#{"s" unless @findings.size == 1}", :bold)}  " \
-                 "#{@a.paint("~#{Bytes.human(total_reclaimable)} reclaimable", :dim)}"
+        @io.puts "  #{@a.paint("schema_reaper", :bold)}  #{@a.paint(scope_text, :bold)}"
+        @io.puts "  #{@a.paint(type_breakdown, :dim)}"
+        @io.puts "  #{@a.paint(reclaim_text, :dim)}" if reclaim_text
+        @io.puts
+      end
+
+      def scope_text
+        tables = @findings.map(&:table).uniq.size
+        "#{@findings.size} finding#{"s" unless @findings.size == 1} across " \
+          "#{tables} table#{"s" unless tables == 1}"
+      end
+
+      def type_breakdown
+        @findings.group_by(&:type).transform_values(&:size)
+                 .sort_by { |_t, n| -n }
+                 .map { |t, n| "#{t} #{n}" }.join(" · ")
+      end
+
+      # A reclaim estimate needs a row count, and pg reports reltuples = -1 for
+      # a table it has never analysed. Printing 0.0 B for an unknown reads as
+      # "nothing to gain here", which is a different claim entirely.
+      def reclaim_text
+        return "~#{Bytes.human(total_reclaimable)} reclaimable" if total_reclaimable.positive?
+        return nil unless unmeasured?
+
+        "reclaim estimate unavailable — run ANALYZE to populate table statistics"
+      end
+
+      def unmeasured?
+        @findings.any? { |f| f.bytes_per_row.to_i.positive? && !f.reclaim_known? }
+      end
+
+      def rolled_up
+        partitioned.first
+      end
+
+      def itemised
+        partitioned.last
+      end
+
+      def partitioned
+        @partitioned ||= Rollup.partition(@findings)
+      end
+
+      def render_rollup(key, group)
+        type, evidence, fix = key
+        first = group.first
+        @io.puts rollup_head(type, first, group.size)
+        @io.puts "           #{@a.paint(evidence, :dim)}"
+        @io.puts "           #{@a.paint("→ #{fix}", :green)}"
+        target_lines(group).each { |line| @io.puts "           #{@a.paint(line, :dim)}" }
         @io.puts
       end
 
-      def grouped
-        @findings
+      def rollup_head(type, first, count)
+        pct = (first.confidence * 100).round
+        bar = @a.confidence_bar(first.confidence, first.severity)
+        sev = @a.severity(first.severity, format("%-6s", first.severity))
+        format("    %s %3d%%  %s  %s  %s", bar, pct, sev,
+               @a.paint(format("%-19s", type), :bold),
+               @a.paint("#{count} targets", :bold, :magenta))
+      end
+
+      # "users.team_id · orders.buyer_id · ..." wrapped to a readable width.
+      def target_lines(group)
+        labels = group.map { |f| [f.table, f.target_label].compact.join(".") }.sort
+        labels.each_with_object([+""]) do |label, lines|
+          lines << +"" if !lines.last.empty? && lines.last.length + label.length + 3 > TARGET_LINE_WIDTH
+          lines.last << " · " unless lines.last.empty?
+          lines.last << label
+        end
+      end
+
+      def grouped(findings)
+        findings
           .sort_by { |f| [-f.confidence, SEV_ORDER.fetch(f.severity, 9)] }
           .group_by(&:table)
       end
@@ -89,12 +162,8 @@ module SchemaReaper
           @a.severity(sev, "#{sev} #{n}") if n
         end.join("   ")
 
-        by_type = @findings.group_by(&:type).transform_values(&:size)
-                           .sort_by { |_t, n| -n }
-                           .map { |t, n| "#{t} #{n}" }.join(" · ")
-
         @io.puts "  #{tally}"
-        @io.puts "  #{@a.paint(by_type, :dim)}"
+        @io.puts "  #{@a.paint(type_breakdown, :dim)}"
       end
 
       def total_reclaimable

data/lib/schema_reaper/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SchemaReaper
-  VERSION = "1.0.10"
+  VERSION = "1.0.11"
 end

metadata

@@ -1,15 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: schema_reaper
 version: !ruby/object:Gem::Version
-  version: 1.0.10
+  version: 1.0.11
 platform: ruby
 authors:
 - aksshatt
 - mitkush
-autorequire: 
 bindir: exe
 cert_chain: []
-date: 2026-09-15 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: prism
@@ -131,6 +130,7 @@ files:
 - lib/schema_reaper/reporters/console.rb
 - lib/schema_reaper/reporters/json.rb
 - lib/schema_reaper/reporters/markdown.rb
+- lib/schema_reaper/reporters/rollup.rb
 - lib/schema_reaper/reporters/sarif.rb
 - lib/schema_reaper/reporters/table.rb
 - lib/schema_reaper/reporters/trend.rb
@@ -155,7 +155,6 @@ metadata:
   wiki_uri: https://github.com/aksshatt/schema_reaper/wiki
   funding_uri: https://github.com/sponsors/aksshatt
   rubygems_mfa_required: 'true'
-post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -170,8 +169,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.10
-signing_key: 
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Find and safely remove schema dead-weight in Rails + PostgreSQL apps.
 test_files: []