schema_reaper 1.0.10 → 1.0.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +91 -0
- data/README.md +5 -2
- data/lib/schema_reaper/analyzers/base.rb +5 -2
- data/lib/schema_reaper/analyzers/missing_fk_index.rb +59 -8
- data/lib/schema_reaper/config.rb +10 -1
- data/lib/schema_reaper/database_url.rb +13 -2
- data/lib/schema_reaper/finding.rb +6 -0
- data/lib/schema_reaper/reporters/console.rb +3 -1
- data/lib/schema_reaper/reporters/markdown.rb +13 -8
- data/lib/schema_reaper/reporters/reclaim.rb +47 -0
- data/lib/schema_reaper/reporters/rollup.rb +55 -0
- data/lib/schema_reaper/reporters/table.rb +74 -14
- data/lib/schema_reaper/version.rb +1 -1
- metadata +5 -6
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b7e70e2066753282ad6757f3d73175a57b73b7026aa8879314f693dafc8547e7
|
|
4
|
+
data.tar.gz: d4b2ac7a4e6934e068d79ac32f694a5bffcd329bed689c23d0a080266b8bdb6b
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: '04943ff8915c5a809a9f81741f14e98712a6453dcf046441b7ae10d726031a994045e9490b5a8a62f948210c0e030825429bde387f35ffc15176a6b419e5f57a'
|
|
7
|
+
data.tar.gz: da080808761005f66096458968fc7c68d3c6f3251d2c97aa50a56020a37abb0d673129782244e880473ffb6287df191418f1142e1bc7632e4da05f49def17cc2
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,96 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.0.12] - 2026-09-15
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- **`--format markdown` claimed `0 B reclaimable` when the row count was
|
|
7
|
+
unknown.** `Table` learned in 1.0.11 that `reclaimable_bytes` is `nil`
|
|
8
|
+
both when the true value is zero and when the row count has never been
|
|
9
|
+
measured, and that a report should say which. `Markdown` summed the raw
|
|
10
|
+
bytes directly and always printed a total, so on an unanalysed database
|
|
11
|
+
it stated there was nothing to reclaim — the exact claim the 1.0.11 fix
|
|
12
|
+
was written to stop, just in the one format meant for an unattended PR
|
|
13
|
+
comment or CI job summary. The per-row `Reclaims` column had the same
|
|
14
|
+
bug: an unmeasured finding showed `0.0 B`, indistinguishable from one
|
|
15
|
+
that genuinely frees nothing. The summary/unmeasured logic now lives in
|
|
16
|
+
one shared `Reporters::Reclaim` module used by both reporters, so the
|
|
17
|
+
two formats cannot answer the same question differently again. (#7,
|
|
18
|
+
mitkush)
|
|
19
|
+
|
|
20
|
+
## [1.0.11] - 2026-09-15
|
|
21
|
+
|
|
22
|
+
Everything below landed after 1.0.10 was cut, so 1.0.10 on RubyGems contains
|
|
23
|
+
none of it.
|
|
24
|
+
|
|
25
|
+
### Fixed
|
|
26
|
+
- **Composite index columns were read in table order, not index-key order.**
|
|
27
|
+
`indexes_for` aggregated with `ORDER BY a.attnum` — a column's position in
|
|
28
|
+
the *table* — so `(company_id, email)` came back as `email,company_id`.
|
|
29
|
+
Since `Index#covers?` is a prefix test, this broke `duplicate_index` in both
|
|
30
|
+
directions: it recommended `remove_index` on indexes that are not redundant
|
|
31
|
+
(dropping one would degrade queries on its leading column), and missed
|
|
32
|
+
genuinely redundant ones. `missing_fk_index` read the same corrupted order
|
|
33
|
+
through `indexed?`. Now joins `unnest(indkey) WITH ORDINALITY`. (#1, mitkush)
|
|
34
|
+
- **`primary_key_for` returned one arbitrary column of a composite key.** It
|
|
35
|
+
matched `attnum = ANY(indkey)` with no `ORDER BY` and took `.first`, so the
|
|
36
|
+
other key columns looked like ordinary columns to the analyzers and could be
|
|
37
|
+
reported as `dead_column` or `single_value_column` — suggesting you drop part
|
|
38
|
+
of a primary key. Now returns every key column in order; `Table#primary_key?`
|
|
39
|
+
accepts a name or a list. (#1, mitkush)
|
|
40
|
+
- **Polymorphic associations were reported as unindexed foreign keys.** A
|
|
41
|
+
`*_id` paired with a `*_type` is only ever queried with the type, so the
|
|
42
|
+
index that matters is the composite `(type, id)` — but `indexed?` only asked
|
|
43
|
+
whether the column led *some* index. Rails apps were told to add an index the
|
|
44
|
+
planner would never use. (#1, mitkush)
|
|
45
|
+
- **`dead_table` missed models behind `-ies` tables and route helpers.**
|
|
46
|
+
`referenced?` singularised with `sub(/s\z/, "")`, turning `crm_activities`
|
|
47
|
+
into `crm_activitie`, so `CrmActivity` never matched. Because
|
|
48
|
+
`drop_findings_on_dead_tables` discards every column- and index-level finding
|
|
49
|
+
for a table called dead, one bad `dead_table` silently suppressed everything
|
|
50
|
+
else about that table. Inflection now handles `-ies`/`-sses`/`-xes`, and a
|
|
51
|
+
table name is also matched inside longer identifiers such as
|
|
52
|
+
`admin_crm_activities_path`. (#1, mitkush)
|
|
53
|
+
- **Ruby 2.7 was broken despite being the declared floor.** `Config.load`
|
|
54
|
+
called `YAML.safe_load_file`, which arrived in Psych 3.3 (Ruby 3.0), so every
|
|
55
|
+
command crashed for any 2.7 user with a `.schema_reaper.yml`.
|
|
56
|
+
`Console#title` appended to an interpolated string, which is frozen on
|
|
57
|
+
Ruby <= 2.7. (#2, mitkush)
|
|
58
|
+
- **`config/database.yml` credentials were not URL-escaped.** A password
|
|
59
|
+
containing `@` made libpq read the rest as the host, reporting
|
|
60
|
+
`could not translate host name "ss@localhost"` — a hostname the user never
|
|
61
|
+
configured. Also, when ERB failed to render (commonly
|
|
62
|
+
`Rails.application.credentials` outside a booted Rails), the resolver fell
|
|
63
|
+
back to parsing the file *un-rendered* and built a connection URL out of
|
|
64
|
+
template text; it now resolves nothing so the real error surfaces. (#1,
|
|
65
|
+
mitkush)
|
|
66
|
+
- **An unknown reclaim estimate was printed as `0 B`.** A byte estimate needs a
|
|
67
|
+
row count, and `reltuples = -1` means unknown; `Base#finding` turned that into
|
|
68
|
+
zero, so reports opened with `~0.0 B reclaimable` — "nothing to gain" rather
|
|
69
|
+
than "cannot say". (#4, mitkush)
|
|
70
|
+
|
|
71
|
+
### Changed
|
|
72
|
+
- **`unused_index` is skipped when the database has no query history.**
|
|
73
|
+
`idx_scan = 0` means either "never used" or "this database has answered no
|
|
74
|
+
queries". On a freshly loaded schema the analyzer reported every non-unique
|
|
75
|
+
index — 197 of 266 findings on one app. It now compares cluster-wide
|
|
76
|
+
`idx_scan` against the index count and explains the skip on stderr. (#2,
|
|
77
|
+
mitkush)
|
|
78
|
+
- **`missing_fk_index` confidence is tiered by evidence.** A declared FK
|
|
79
|
+
constraint scores 0.9; a `*_id` name with a table it plausibly references
|
|
80
|
+
scores 0.7; a `*_id` name with nothing to reference scores 0.5 and drops to
|
|
81
|
+
`:low`. Previously `voter_id` and `upi_id` were reported as confidently as a
|
|
82
|
+
real constraint, which made `--min-confidence` useless for this analyzer.
|
|
83
|
+
Nothing is dropped. (#3, mitkush)
|
|
84
|
+
- **The report says what it contains.** The header now carries the finding
|
|
85
|
+
count, how many tables are affected, and the type breakdown — previously only
|
|
86
|
+
a tally in the footer, 245 lines below on a large report. (#4, mitkush)
|
|
87
|
+
- **Findings that differ only in the names they mention are rolled up.** 37
|
|
88
|
+
missing foreign-key indexes spent 111 lines repeating one sentence. They now
|
|
89
|
+
collapse into a single entry that keeps the confidence bar, states the fix as
|
|
90
|
+
a template, and lists every target. Nothing is summarised away, and findings
|
|
91
|
+
whose evidence genuinely differs — every `duplicate_index` names a different
|
|
92
|
+
index pair — stay itemised. 246 lines to 124 on one app. (#5, mitkush)
|
|
93
|
+
|
|
3
94
|
## [1.0.10] - 2026-09-09
|
|
4
95
|
|
|
5
96
|
Supersedes 1.0.9, which was published to RubyGems from an incomplete cut and
|
data/README.md
CHANGED
|
@@ -47,10 +47,13 @@ bundle exec schema_reaper trend # snapshot + progress delta
|
|
|
47
47
|
bundle exec schema_reaper generate-migration users legacy_api_token
|
|
48
48
|
```
|
|
49
49
|
|
|
50
|
-
The `scan` report
|
|
50
|
+
The `scan` report rolls up findings that say the same thing about different
|
|
51
|
+
tables, then groups the rest by table and sorts by confidence:
|
|
51
52
|
|
|
52
53
|
```
|
|
53
|
-
schema_reaper 5 findings
|
|
54
|
+
schema_reaper 5 findings across 2 tables
|
|
55
|
+
missing_fk_index 3 · always_null_column 1 · dead_column 1
|
|
56
|
+
~93.8 KB reclaimable
|
|
54
57
|
|
|
55
58
|
users
|
|
56
59
|
█████ 90% medium missing_fk_index team_id
|
|
@@ -58,12 +58,15 @@ module SchemaReaper
|
|
|
58
58
|
end
|
|
59
59
|
|
|
60
60
|
# Builds a Finding, filling in reclaimable_bytes from the row count.
|
|
61
|
+
# An unknown row count leaves reclaimable_bytes nil: "we cannot say" is
|
|
62
|
+
# not the same claim as "zero bytes", and reporters need to tell them
|
|
63
|
+
# apart. Finding#reclaimable_bytes still reads 0 for arithmetic.
|
|
61
64
|
def finding(table:, bytes_per_row: 0, row_count: nil, **rest)
|
|
62
|
-
rows = row_count || schema.table(table)&.row_count
|
|
65
|
+
rows = row_count || schema.table(table)&.row_count
|
|
63
66
|
Finding.new(
|
|
64
67
|
table: table,
|
|
65
68
|
bytes_per_row: bytes_per_row,
|
|
66
|
-
reclaimable_bytes: bytes_per_row * rows,
|
|
69
|
+
reclaimable_bytes: rows && (bytes_per_row * rows),
|
|
67
70
|
**rest
|
|
68
71
|
)
|
|
69
72
|
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
+
require "set"
|
|
4
|
+
|
|
3
5
|
module SchemaReaper
|
|
4
6
|
module Analyzers
|
|
5
7
|
# A foreign-key column with no index: every parent delete/update scans the
|
|
@@ -12,6 +14,15 @@ module SchemaReaper
|
|
|
12
14
|
class MissingFkIndex < Base
|
|
13
15
|
Registry.register(self)
|
|
14
16
|
|
|
17
|
+
# How sure we are that the column really is a foreign key:
|
|
18
|
+
# a declared constraint is proof;
|
|
19
|
+
# a *_id name plus a table it plausibly points at is good evidence;
|
|
20
|
+
# a *_id name alone is a guess -- voter_id and upi_id are not foreign
|
|
21
|
+
# keys, they just end in _id.
|
|
22
|
+
CONSTRAINT_CONFIDENCE = 0.9
|
|
23
|
+
REFERENT_CONFIDENCE = 0.7
|
|
24
|
+
NAME_ONLY_CONFIDENCE = 0.5
|
|
25
|
+
|
|
15
26
|
def call
|
|
16
27
|
schema.tables.reject { |t| config.ignore_tables.include?(t.name) }
|
|
17
28
|
.flat_map { |t| missing_in(t) }
|
|
@@ -27,7 +38,7 @@ module SchemaReaper
|
|
|
27
38
|
type_col = polymorphic_type_for(table, col)
|
|
28
39
|
next if type_col && polymorphic_indexed?(table, type_col, col)
|
|
29
40
|
|
|
30
|
-
finding_for(table, col, type_col)
|
|
41
|
+
finding_for(table, col, type_col, declared: table.foreign_keys.include?(col))
|
|
31
42
|
end
|
|
32
43
|
end
|
|
33
44
|
|
|
@@ -40,23 +51,45 @@ module SchemaReaper
|
|
|
40
51
|
table.column(col)&.always_null? || false
|
|
41
52
|
end
|
|
42
53
|
|
|
43
|
-
def finding_for(table, col, type_col)
|
|
54
|
+
def finding_for(table, col, type_col, declared:)
|
|
55
|
+
referent = declared ? nil : referent_table(col)
|
|
56
|
+
confidence = confidence_for(declared, referent)
|
|
57
|
+
|
|
44
58
|
finding(
|
|
45
59
|
type: :missing_fk_index,
|
|
46
60
|
table: table.name,
|
|
47
61
|
column: col,
|
|
48
|
-
severity: :medium,
|
|
49
|
-
confidence:
|
|
62
|
+
severity: confidence >= REFERENT_CONFIDENCE ? :medium : :low,
|
|
63
|
+
confidence: confidence,
|
|
50
64
|
bytes_per_row: 0,
|
|
51
|
-
evidence: [evidence_for(col, type_col)],
|
|
65
|
+
evidence: [evidence_for(col, type_col, declared, referent)],
|
|
52
66
|
suggested_fix: fix_for(table, col, type_col)
|
|
53
67
|
)
|
|
54
68
|
end
|
|
55
69
|
|
|
56
|
-
def
|
|
57
|
-
return
|
|
70
|
+
def confidence_for(declared, referent)
|
|
71
|
+
return CONSTRAINT_CONFIDENCE if declared
|
|
72
|
+
return REFERENT_CONFIDENCE if referent
|
|
73
|
+
|
|
74
|
+
NAME_ONLY_CONFIDENCE
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def evidence_for(col, type_col, declared, referent)
|
|
78
|
+
subject =
|
|
79
|
+
if type_col
|
|
80
|
+
"#{col} is a polymorphic association with #{type_col} and has no (#{type_col}, #{col}) index"
|
|
81
|
+
else
|
|
82
|
+
"#{col} has no covering index"
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
"#{subject} (#{provenance(declared, referent)})"
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
def provenance(declared, referent)
|
|
89
|
+
return "declared foreign key constraint" if declared
|
|
90
|
+
return "no FK constraint; named like a reference to `#{referent}`" if referent
|
|
58
91
|
|
|
59
|
-
"
|
|
92
|
+
"no FK constraint and no table it plausibly references -- matched on the _id suffix alone"
|
|
60
93
|
end
|
|
61
94
|
|
|
62
95
|
def fix_for(table, col, type_col)
|
|
@@ -69,6 +102,24 @@ module SchemaReaper
|
|
|
69
102
|
(table.foreign_keys + table.column_names.grep(/_id\z/)).uniq
|
|
70
103
|
end
|
|
71
104
|
|
|
105
|
+
# A table the column plausibly points at: account_id -> accounts,
|
|
106
|
+
# company_id -> companies, address_id -> addresses. Absence is the signal
|
|
107
|
+
# that a *_id name is not really a foreign key.
|
|
108
|
+
def referent_table(col)
|
|
109
|
+
return nil unless col.end_with?("_id")
|
|
110
|
+
|
|
111
|
+
base = col[0..-4]
|
|
112
|
+
return nil if base.empty?
|
|
113
|
+
|
|
114
|
+
candidates = [base, "#{base}s", "#{base}es"]
|
|
115
|
+
candidates << "#{base[0..-2]}ies" if base.end_with?("y")
|
|
116
|
+
candidates.find { |name| table_names.include?(name) }
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
def table_names
|
|
120
|
+
@table_names ||= schema.tables.to_set(&:name)
|
|
121
|
+
end
|
|
122
|
+
|
|
72
123
|
def indexed?(table, col)
|
|
73
124
|
table.indexes.any? { |ix| ix.columns.first == col }
|
|
74
125
|
end
|
data/lib/schema_reaper/config.rb
CHANGED
|
@@ -28,10 +28,19 @@ module SchemaReaper
|
|
|
28
28
|
}.freeze
|
|
29
29
|
|
|
30
30
|
def self.load(path = ".schema_reaper.yml")
|
|
31
|
-
raw = File.exist?(path) ? (
|
|
31
|
+
raw = File.exist?(path) ? (safe_load_file(path) || {}) : {}
|
|
32
32
|
new(deep_merge(DEFAULTS, raw), root: File.dirname(File.expand_path(path)))
|
|
33
33
|
end
|
|
34
34
|
|
|
35
|
+
# Psych.safe_load_file arrived in Psych 3.3 (Ruby 3.0), but the gem supports
|
|
36
|
+
# Ruby 2.7. Reading the file ourselves is equivalent -- safe_load_file
|
|
37
|
+
# defaults to aliases: false, which is what safe_load does too.
|
|
38
|
+
def self.safe_load_file(path)
|
|
39
|
+
return YAML.safe_load_file(path) if YAML.respond_to?(:safe_load_file)
|
|
40
|
+
|
|
41
|
+
YAML.safe_load(File.read(path))
|
|
42
|
+
end
|
|
43
|
+
|
|
35
44
|
def self.deep_merge(base, override)
|
|
36
45
|
base.merge(override) do |_key, a, b|
|
|
37
46
|
a.is_a?(Hash) && b.is_a?(Hash) ? deep_merge(a, b) : b
|
|
@@ -40,7 +40,11 @@ module SchemaReaper
|
|
|
40
40
|
YAML.safe_load(rendered, [], [], true)
|
|
41
41
|
end
|
|
42
42
|
rescue StandardError
|
|
43
|
-
|
|
43
|
+
# ERB that reaches into Rails -- Rails.application.credentials is the
|
|
44
|
+
# common one -- cannot render outside a booted Rails process. Resolving
|
|
45
|
+
# nothing yields a clear "no database_url configured"; parsing the file
|
|
46
|
+
# un-rendered would build a connection URL out of template text.
|
|
47
|
+
nil
|
|
44
48
|
end
|
|
45
49
|
|
|
46
50
|
# Rails 6+ allows `development: { primary: {...}, replica: {...} }`. Pick the
|
|
@@ -67,12 +71,19 @@ module SchemaReaper
|
|
|
67
71
|
db = section["database"]
|
|
68
72
|
return nil if db.to_s.empty?
|
|
69
73
|
|
|
70
|
-
userinfo = [section["username"], section["password"]].compact.join(":")
|
|
74
|
+
userinfo = [section["username"], section["password"]].compact.map { |v| escape(v) }.join(":")
|
|
71
75
|
host = section["host"].to_s
|
|
72
76
|
hostport = host.empty? ? "" : "#{host}#{":#{section["port"]}" if section["port"]}"
|
|
73
77
|
auth = userinfo.empty? ? "" : "#{userinfo}@"
|
|
74
78
|
|
|
75
79
|
"postgresql://#{auth}#{hostport}/#{db}"
|
|
76
80
|
end
|
|
81
|
+
|
|
82
|
+
# Passwords routinely contain characters that are structural in a URL.
|
|
83
|
+
# An unescaped "@" makes libpq read the rest as the host, so it reports
|
|
84
|
+
# a bogus hostname rather than a credential problem.
|
|
85
|
+
def escape(value)
|
|
86
|
+
ERB::Util.url_encode(value.to_s)
|
|
87
|
+
end
|
|
77
88
|
end
|
|
78
89
|
end
|
|
@@ -36,6 +36,12 @@ module SchemaReaper
|
|
|
36
36
|
self[:reclaimable_bytes] || 0
|
|
37
37
|
end
|
|
38
38
|
|
|
39
|
+
# False when the row count was unavailable, so the byte estimate is
|
|
40
|
+
# unknown rather than zero.
|
|
41
|
+
def reclaim_known?
|
|
42
|
+
!self[:reclaimable_bytes].nil?
|
|
43
|
+
end
|
|
44
|
+
|
|
39
45
|
# Column and/or index this finding points at, without the table name
|
|
40
46
|
# (callers that group by table already show it). nil for a whole-table
|
|
41
47
|
# finding.
|
|
@@ -21,7 +21,9 @@ module SchemaReaper
|
|
|
21
21
|
|
|
22
22
|
# " schema_reaper trend · 3 snapshots"
|
|
23
23
|
def title(command, meta = nil)
|
|
24
|
-
|
|
24
|
+
# +"" : on Ruby <= 2.7 an interpolated string whose parts are all frozen
|
|
25
|
+
# literals is itself frozen, so the append below raises FrozenError.
|
|
26
|
+
line = +" #{@a.paint("schema_reaper", :bold)} #{@a.paint(command, :bold)}"
|
|
25
27
|
line << " #{@a.paint("· #{meta}", :dim)}" if meta
|
|
26
28
|
@out.puts
|
|
27
29
|
@out.puts line
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
|
-
require_relative "
|
|
3
|
+
require_relative "reclaim"
|
|
4
4
|
|
|
5
5
|
module SchemaReaper
|
|
6
6
|
module Reporters
|
|
@@ -18,8 +18,7 @@ module SchemaReaper
|
|
|
18
18
|
return
|
|
19
19
|
end
|
|
20
20
|
|
|
21
|
-
@io.puts "\n#{
|
|
22
|
-
"~**#{Bytes.human(total)}** reclaimable.\n\n"
|
|
21
|
+
@io.puts "\n#{summary_line}\n\n"
|
|
23
22
|
@io.puts "| Severity | Confidence | Type | Target | Reclaims | Fix |"
|
|
24
23
|
@io.puts "|---|---|---|---|---|---|"
|
|
25
24
|
rows.each { |r| @io.puts r }
|
|
@@ -27,17 +26,23 @@ module SchemaReaper
|
|
|
27
26
|
|
|
28
27
|
private
|
|
29
28
|
|
|
29
|
+
# "54 finding(s). **~93.8 KB reclaimable.**" -- or, when the row count is
|
|
30
|
+
# unknown or the estimate is genuinely zero, no reclaim clause at all,
|
|
31
|
+
# matching the terminal report rather than claiming a number it does not
|
|
32
|
+
# have.
|
|
33
|
+
def summary_line
|
|
34
|
+
text = Reclaim.summary(@findings)
|
|
35
|
+
line = "#{@findings.size} finding(s)."
|
|
36
|
+
line + (text ? " **#{text}.**" : "")
|
|
37
|
+
end
|
|
38
|
+
|
|
30
39
|
def rows
|
|
31
40
|
@findings.sort_by { |f| -f.confidence }.map do |f|
|
|
32
41
|
target = [f.table, f.column, f.index].compact.join("`.`")
|
|
33
42
|
"| #{f.severity} | #{(f.confidence * 100).round}% | `#{f.type}` | " \
|
|
34
|
-
"`#{target}` | #{
|
|
43
|
+
"`#{target}` | #{Reclaim.cell(f)} | #{f.suggested_fix} |"
|
|
35
44
|
end
|
|
36
45
|
end
|
|
37
|
-
|
|
38
|
-
def total
|
|
39
|
-
@findings.sum(&:reclaimable_bytes)
|
|
40
|
-
end
|
|
41
46
|
end
|
|
42
47
|
end
|
|
43
48
|
end
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require_relative "bytes"
|
|
4
|
+
|
|
5
|
+
module SchemaReaper
|
|
6
|
+
module Reporters
|
|
7
|
+
# Turns a set of findings' reclaimable-bytes into a human sentence, and a
|
|
8
|
+
# single finding's into a table cell. Shared by every reporter that states
|
|
9
|
+
# a reclaim total in prose, so the answer cannot drift between formats --
|
|
10
|
+
# which is how the terminal report learned to distinguish "no bytes" from
|
|
11
|
+
# "no idea" while the Markdown report kept treating them as the same 0.
|
|
12
|
+
module Reclaim
|
|
13
|
+
module_function
|
|
14
|
+
|
|
15
|
+
# A row count is required for a byte estimate, and pg reports
|
|
16
|
+
# reltuples = -1 for a table that has never been analysed. Printing 0.0 B
|
|
17
|
+
# for an unknown reads as "nothing to gain here", a different claim.
|
|
18
|
+
def summary(findings)
|
|
19
|
+
total = findings.sum(&:reclaimable_bytes)
|
|
20
|
+
return "~#{Bytes.human(total)} reclaimable" if total.positive?
|
|
21
|
+
return nil unless unmeasured?(findings)
|
|
22
|
+
|
|
23
|
+
"reclaim estimate unavailable — run ANALYZE to populate table statistics"
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
# True when some finding's byte estimate is unknown rather than zero.
|
|
27
|
+
# bytes_per_row.zero? findings (missing_fk_index, duplicate_index -- they
|
|
28
|
+
# add or drop an index, not data) always reclaim 0 regardless of row
|
|
29
|
+
# count, so an unknown row count does not make their answer unmeasured.
|
|
30
|
+
def unmeasured?(findings)
|
|
31
|
+
findings.any? { |f| f.bytes_per_row.to_i.positive? && !f.reclaim_known? }
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
# Per-finding cell for a table row: the human size, or a note that it is
|
|
35
|
+
# not known, using the same bytes_per_row-zero exception as summary.
|
|
36
|
+
def cell(finding)
|
|
37
|
+
return Bytes.human(finding.reclaimable_bytes) if measured?(finding)
|
|
38
|
+
|
|
39
|
+
"unknown"
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def measured?(finding)
|
|
43
|
+
finding.bytes_per_row.to_i.zero? || finding.reclaim_known?
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SchemaReaper
|
|
4
|
+
module Reporters
|
|
5
|
+
# Groups findings that say the same thing about different targets.
|
|
6
|
+
#
|
|
7
|
+
# A report of 37 missing foreign-key indexes repeats one sentence 37 times,
|
|
8
|
+
# varying only the table and column. Collapsing those into a single entry
|
|
9
|
+
# with its targets listed keeps every fact and drops the repetition.
|
|
10
|
+
module Rollup
|
|
11
|
+
# Findings that differ only in which table and column they name carry no
|
|
12
|
+
# new information per line. At or above this many, roll them up.
|
|
13
|
+
THRESHOLD = 4
|
|
14
|
+
|
|
15
|
+
module_function
|
|
16
|
+
|
|
17
|
+
# @return [Array(Array<[key, Array<Finding>]>, Array<Finding>)]
|
|
18
|
+
# the groups worth rolling up (largest first), and the findings to
|
|
19
|
+
# spell out individually
|
|
20
|
+
def partition(findings)
|
|
21
|
+
by_shape = findings.group_by { |f| key_for(f) }
|
|
22
|
+
big, small = by_shape.partition { |_key, group| group.size >= THRESHOLD }
|
|
23
|
+
[big.sort_by { |_key, group| -group.size }, small.flat_map { |_key, group| group }]
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
def key_for(finding)
|
|
27
|
+
[finding.type, shape(evidence_line(finding), finding), shape(finding.suggested_fix, finding)]
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def evidence_line(finding)
|
|
31
|
+
finding.evidence.join(" · ")
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
# Masks the names a finding is about, each with its own placeholder, so
|
|
35
|
+
# two findings saying the same thing about different targets compare
|
|
36
|
+
# equal -- and so the rolled-up fix still reads as a template.
|
|
37
|
+
#
|
|
38
|
+
# The lookarounds matter: without them, masking
|
|
39
|
+
# index_users_on_team_id would also eat it out of
|
|
40
|
+
# index_users_on_team_id_and_state, making two findings about different
|
|
41
|
+
# covering indexes look identical.
|
|
42
|
+
def shape(text, finding)
|
|
43
|
+
replacements(finding).reduce(text) do |acc, (name, token)|
|
|
44
|
+
acc.gsub(/(?<!\w)#{Regexp.escape(name)}(?!\w)/, token)
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def replacements(finding)
|
|
49
|
+
[[finding.table, "<table>"], [finding.column, "<column>"], [finding.index, "<index>"]]
|
|
50
|
+
.reject { |name, _| name.nil? || name.empty? }
|
|
51
|
+
.sort_by { |name, _| -name.length }
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
@@ -2,15 +2,21 @@
|
|
|
2
2
|
|
|
3
3
|
require_relative "bytes"
|
|
4
4
|
require_relative "ansi"
|
|
5
|
+
require_relative "rollup"
|
|
6
|
+
require_relative "reclaim"
|
|
5
7
|
|
|
6
8
|
module SchemaReaper
|
|
7
9
|
module Reporters
|
|
8
|
-
# Human-readable terminal report: a summary line, findings
|
|
9
|
-
#
|
|
10
|
-
#
|
|
10
|
+
# Human-readable terminal report: a summary line, then findings. Findings
|
|
11
|
+
# that say the same thing about different tables are rolled up into one
|
|
12
|
+
# entry; the rest are grouped by table and sorted by confidence. Colour is
|
|
13
|
+
# used only on an interactive terminal (see Ansi).
|
|
11
14
|
class Table
|
|
12
15
|
SEV_ORDER = { high: 0, medium: 1, low: 2 }.freeze
|
|
13
16
|
|
|
17
|
+
# Width to wrap the target list of a rolled-up entry.
|
|
18
|
+
TARGET_LINE_WIDTH = 92
|
|
19
|
+
|
|
14
20
|
def initialize(findings, io: $stdout, color: nil)
|
|
15
21
|
@findings = findings
|
|
16
22
|
@io = io
|
|
@@ -21,7 +27,8 @@ module SchemaReaper
|
|
|
21
27
|
return render_clean if @findings.empty?
|
|
22
28
|
|
|
23
29
|
header
|
|
24
|
-
|
|
30
|
+
rolled_up.each { |key, group| render_rollup(key, group) }
|
|
31
|
+
grouped(itemised).each { |table, group| render_table(table, group) }
|
|
25
32
|
footer
|
|
26
33
|
end
|
|
27
34
|
|
|
@@ -33,14 +40,71 @@ module SchemaReaper
|
|
|
33
40
|
|
|
34
41
|
def header
|
|
35
42
|
@io.puts
|
|
36
|
-
@io.puts " #{@a.paint("schema_reaper", :bold)} "
|
|
37
|
-
|
|
38
|
-
|
|
43
|
+
@io.puts " #{@a.paint("schema_reaper", :bold)} #{@a.paint(scope_text, :bold)}"
|
|
44
|
+
@io.puts " #{@a.paint(type_breakdown, :dim)}"
|
|
45
|
+
@io.puts " #{@a.paint(reclaim_text, :dim)}" if reclaim_text
|
|
39
46
|
@io.puts
|
|
40
47
|
end
|
|
41
48
|
|
|
42
|
-
def
|
|
43
|
-
@findings
|
|
49
|
+
def scope_text
|
|
50
|
+
tables = @findings.map(&:table).uniq.size
|
|
51
|
+
"#{@findings.size} finding#{"s" unless @findings.size == 1} across " \
|
|
52
|
+
"#{tables} table#{"s" unless tables == 1}"
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def type_breakdown
|
|
56
|
+
@findings.group_by(&:type).transform_values(&:size)
|
|
57
|
+
.sort_by { |_t, n| -n }
|
|
58
|
+
.map { |t, n| "#{t} #{n}" }.join(" · ")
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def reclaim_text
|
|
62
|
+
Reclaim.summary(@findings)
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
def rolled_up
|
|
66
|
+
partitioned.first
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
def itemised
|
|
70
|
+
partitioned.last
|
|
71
|
+
end
|
|
72
|
+
|
|
73
|
+
def partitioned
|
|
74
|
+
@partitioned ||= Rollup.partition(@findings)
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def render_rollup(key, group)
|
|
78
|
+
type, evidence, fix = key
|
|
79
|
+
first = group.first
|
|
80
|
+
@io.puts rollup_head(type, first, group.size)
|
|
81
|
+
@io.puts " #{@a.paint(evidence, :dim)}"
|
|
82
|
+
@io.puts " #{@a.paint("→ #{fix}", :green)}"
|
|
83
|
+
target_lines(group).each { |line| @io.puts " #{@a.paint(line, :dim)}" }
|
|
84
|
+
@io.puts
|
|
85
|
+
end
|
|
86
|
+
|
|
87
|
+
def rollup_head(type, first, count)
|
|
88
|
+
pct = (first.confidence * 100).round
|
|
89
|
+
bar = @a.confidence_bar(first.confidence, first.severity)
|
|
90
|
+
sev = @a.severity(first.severity, format("%-6s", first.severity))
|
|
91
|
+
format(" %s %3d%% %s %s %s", bar, pct, sev,
|
|
92
|
+
@a.paint(format("%-19s", type), :bold),
|
|
93
|
+
@a.paint("#{count} targets", :bold, :magenta))
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
# "users.team_id · orders.buyer_id · ..." wrapped to a readable width.
|
|
97
|
+
def target_lines(group)
|
|
98
|
+
labels = group.map { |f| [f.table, f.target_label].compact.join(".") }.sort
|
|
99
|
+
labels.each_with_object([+""]) do |label, lines|
|
|
100
|
+
lines << +"" if !lines.last.empty? && lines.last.length + label.length + 3 > TARGET_LINE_WIDTH
|
|
101
|
+
lines.last << " · " unless lines.last.empty?
|
|
102
|
+
lines.last << label
|
|
103
|
+
end
|
|
104
|
+
end
|
|
105
|
+
|
|
106
|
+
def grouped(findings)
|
|
107
|
+
findings
|
|
44
108
|
.sort_by { |f| [-f.confidence, SEV_ORDER.fetch(f.severity, 9)] }
|
|
45
109
|
.group_by(&:table)
|
|
46
110
|
end
|
|
@@ -89,12 +153,8 @@ module SchemaReaper
|
|
|
89
153
|
@a.severity(sev, "#{sev} #{n}") if n
|
|
90
154
|
end.join(" ")
|
|
91
155
|
|
|
92
|
-
by_type = @findings.group_by(&:type).transform_values(&:size)
|
|
93
|
-
.sort_by { |_t, n| -n }
|
|
94
|
-
.map { |t, n| "#{t} #{n}" }.join(" · ")
|
|
95
|
-
|
|
96
156
|
@io.puts " #{tally}"
|
|
97
|
-
@io.puts " #{@a.paint(
|
|
157
|
+
@io.puts " #{@a.paint(type_breakdown, :dim)}"
|
|
98
158
|
end
|
|
99
159
|
|
|
100
160
|
def total_reclaimable
|
metadata
CHANGED
|
@@ -1,15 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: schema_reaper
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.0.
|
|
4
|
+
version: 1.0.12
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- aksshatt
|
|
8
8
|
- mitkush
|
|
9
|
-
autorequire:
|
|
10
9
|
bindir: exe
|
|
11
10
|
cert_chain: []
|
|
12
|
-
date:
|
|
11
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
13
12
|
dependencies:
|
|
14
13
|
- !ruby/object:Gem::Dependency
|
|
15
14
|
name: prism
|
|
@@ -131,6 +130,8 @@ files:
|
|
|
131
130
|
- lib/schema_reaper/reporters/console.rb
|
|
132
131
|
- lib/schema_reaper/reporters/json.rb
|
|
133
132
|
- lib/schema_reaper/reporters/markdown.rb
|
|
133
|
+
- lib/schema_reaper/reporters/reclaim.rb
|
|
134
|
+
- lib/schema_reaper/reporters/rollup.rb
|
|
134
135
|
- lib/schema_reaper/reporters/sarif.rb
|
|
135
136
|
- lib/schema_reaper/reporters/table.rb
|
|
136
137
|
- lib/schema_reaper/reporters/trend.rb
|
|
@@ -155,7 +156,6 @@ metadata:
|
|
|
155
156
|
wiki_uri: https://github.com/aksshatt/schema_reaper/wiki
|
|
156
157
|
funding_uri: https://github.com/sponsors/aksshatt
|
|
157
158
|
rubygems_mfa_required: 'true'
|
|
158
|
-
post_install_message:
|
|
159
159
|
rdoc_options: []
|
|
160
160
|
require_paths:
|
|
161
161
|
- lib
|
|
@@ -170,8 +170,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
170
170
|
- !ruby/object:Gem::Version
|
|
171
171
|
version: '0'
|
|
172
172
|
requirements: []
|
|
173
|
-
rubygems_version: 3.
|
|
174
|
-
signing_key:
|
|
173
|
+
rubygems_version: 3.6.9
|
|
175
174
|
specification_version: 4
|
|
176
175
|
summary: Find and safely remove schema dead-weight in Rails + PostgreSQL apps.
|
|
177
176
|
test_files: []
|