safegres 1.14.2 → 1.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +90 -0
- package/checks/indexes.d.ts +21 -0
- package/checks/indexes.js +104 -0
- package/checks/stats.d.ts +52 -0
- package/checks/stats.js +192 -0
- package/cli/audit.js +34 -2
- package/commands/audit.d.ts +11 -0
- package/commands/audit.js +59 -2
- package/config/types.d.ts +42 -1
- package/esm/checks/indexes.d.ts +21 -0
- package/esm/checks/indexes.js +102 -0
- package/esm/checks/stats.d.ts +52 -0
- package/esm/checks/stats.js +184 -0
- package/esm/cli/audit.js +34 -2
- package/esm/commands/audit.d.ts +11 -0
- package/esm/commands/audit.js +60 -3
- package/esm/config/types.d.ts +42 -1
- package/esm/index.d.ts +12 -2
- package/esm/index.js +6 -1
- package/esm/perf/explain.d.ts +46 -0
- package/esm/perf/explain.js +201 -0
- package/esm/pg/indexes.d.ts +13 -0
- package/esm/pg/indexes.js +18 -2
- package/esm/pg/stats.d.ts +65 -0
- package/esm/pg/stats.js +122 -0
- package/esm/report/markdown.d.ts +20 -0
- package/esm/report/markdown.js +120 -0
- package/esm/report/pretty.js +15 -1
- package/esm/report/sarif.d.ts +40 -0
- package/esm/report/sarif.js +189 -0
- package/esm/rules/registry.d.ts +1 -1
- package/esm/rules/registry.js +54 -0
- package/esm/types.d.ts +35 -0
- package/index.d.ts +12 -2
- package/index.js +20 -2
- package/package.json +3 -3
- package/perf/explain.d.ts +46 -0
- package/perf/explain.js +204 -0
- package/pg/indexes.d.ts +13 -0
- package/pg/indexes.js +18 -2
- package/pg/stats.d.ts +65 -0
- package/pg/stats.js +126 -0
- package/report/markdown.d.ts +20 -0
- package/report/markdown.js +123 -0
- package/report/pretty.js +15 -1
- package/report/sarif.d.ts +40 -0
- package/report/sarif.js +193 -0
- package/rules/registry.d.ts +1 -1
- package/rules/registry.js +54 -0
- package/types.d.ts +35 -0
package/README.md
CHANGED
|
@@ -34,6 +34,39 @@ Pretty output prints the exposure line, score, and the exposed findings. Interna
|
|
|
34
34
|
- `--verbose` — expand the internal advisories instead of collapsing them to a count.
|
|
35
35
|
- `--exposed-only` — drop internal findings entirely.
|
|
36
36
|
- `--format json` / `--format json-pretty` — machine-readable output (always carries every finding).
|
|
37
|
+
- `--format markdown` — the same report as GitHub-flavoured markdown, for a job summary or a PR comment (see [CI](#ci)).
|
|
38
|
+
- `--format sarif` — SARIF 2.1.0 for GitHub code scanning (see [CI](#ci)).
|
|
39
|
+
|
|
40
|
+
### CI
|
|
41
|
+
|
|
42
|
+
A plain audit is one command and one gate; `--format markdown` writes the report where a reviewer will actually see it:
|
|
43
|
+
|
|
44
|
+
```yaml
|
|
45
|
+
- name: Audit RLS
|
|
46
|
+
run: |
|
|
47
|
+
npx safegres audit --format markdown >> "$GITHUB_STEP_SUMMARY"
|
|
48
|
+
npx safegres audit --fail-on-grade B --summary
|
|
49
|
+
env:
|
|
50
|
+
PGHOST: localhost
|
|
51
|
+
PGUSER: postgres
|
|
52
|
+
PGPASSWORD: postgres
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Scores lead, then the severity counts, then a table per dimension; internal (non-exposed) advisories and accepted baseline debt fold into `<details>` so the summary stays skimmable. To post it as a PR comment instead, pipe it to `gh pr comment --body-file -`. The same renderer is available to library callers as `renderMarkdown(report)`.
|
|
56
|
+
|
|
57
|
+
#### Code scanning (SARIF)
|
|
58
|
+
|
|
59
|
+
`--format sarif` emits SARIF 2.1.0, so findings become GitHub code-scanning alerts — Security tab, inline PR annotations, dismissals that stick:
|
|
60
|
+
|
|
61
|
+
```yaml
|
|
62
|
+
- run: npx safegres audit --perf --format sarif --sarif-sources ./deploy > safegres.sarif
|
|
63
|
+
- uses: github/codeql-action/upload-sarif@v3
|
|
64
|
+
with: { sarif_file: safegres.sarif }
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
An alert needs a file and a line, but safegres reads the catalog — a live database has no source location. `--sarif-sources <dir>` scans that directory's `.sql` for the `CREATE TABLE` / `CREATE POLICY` that defines each object, so a finding on `app_public.widgets` points at the migration that created it (policy findings resolve to the `CREATE POLICY` line). Findings that don't resolve are still emitted, without a location — GitHub drops those, other SARIF consumers keep them.
|
|
68
|
+
|
|
69
|
+
Results are fingerprinted by finding *identity* (code + relation + policy + subject, the same key the [perf baseline](#perf-baseline-the-ratchet) uses), never by message text, so rewording a rule in a later release doesn't close and reopen every alert. Perf rules are tagged `performance`, security rules `security`.
|
|
37
70
|
|
|
38
71
|
## What it checks
|
|
39
72
|
|
|
@@ -112,11 +145,15 @@ A slow database is a different problem from an unsafe one, so safegres scores th
|
|
|
112
145
|
| X4 | low | index | **Policy calls a non-LEAKPROOF function** — the qual can't be pushed below joins or subquery scans |
|
|
113
146
|
| X5 | low | index | **Redundant index** — an exact duplicate of, or a leading-column prefix of, another index |
|
|
114
147
|
| X6 | low | index | **No primary key** and no usable replica identity — rows cannot be addressed by updates, deletes, or logical replication |
|
|
148
|
+
| X7 | medium | index | **Search column with no index the search can use** — a `tsvector` without GIN/GiST, a `vector` without HNSW/IVFFlat |
|
|
149
|
+
| X8 | info | index | **Sort-shaped column leads no index** (`timestamptz`/`date`) — ordering or cursor-paginating a connection by it sorts the whole table |
|
|
115
150
|
| P1 | high | anti-pattern | Policy body calls a **VOLATILE function** — re-evaluated per row |
|
|
116
151
|
| P1b | medium | anti-pattern | Policy body calls a **STABLE function** in a per-row position |
|
|
117
152
|
|
|
118
153
|
Every check is pure catalog + AST analysis: deterministic, workload-free, and safe to run against an empty CI database. An index covers a foreign key only when its *leading* columns are the FK's columns and it covers every row — partial and expression indexes don't count, because the planner can't use them for the referential-integrity lookup. Constraint-backed, unique, partial, and expression indexes are never reported as redundant.
|
|
119
154
|
|
|
155
|
+
X7 exists because the column type *is* the API declaration: `graphile-search` exposes a full-text filter for every `tsvector` column and a similarity search for every `vector` column, purely from the codec — so an unindexed one is a first-class API field backed by a sequential scan plus a per-row match or distance computation. BM25 and pg_trgm are deliberately not checked: those adapters are discovered *from* their indexes, so a missing index means the feature was never exposed. X8 is the one heuristic in the set — any column is orderable over a connection, but timestamps are what feeds are actually sorted and keyset-paginated by — so it defaults to `info`, contributes 0 to the score, and is meant to be read, not gated on (`perf.rules: { "X8": "off" }` to silence it). Trailing-position and partial indexes don't count for either rule: neither can serve the sort or the search on its own.
|
|
156
|
+
|
|
120
157
|
X2–X4 are the checks a generic index linter can't make, because they read the policy predicate. RLS quals are evaluated *before* user quals, on every candidate row, for every caller — so an unindexed or cast-wrapped policy column is a whole-table tax rather than a slow query. X2 requires the policy column to be the *leading* column of some index (a trailing position can't serve the qual alone); X3 looks for an expression index matching the exact wrapped shape; X4 skips built-ins, whose leakproofness is a property of the server rather than a schema choice.
|
|
121
158
|
|
|
122
159
|
```bash
|
|
@@ -158,6 +195,59 @@ performance vs baseline:
|
|
|
158
195
|
|
|
159
196
|
Entries are identified by `code` + relation + policy + subject (the constraint, index, expression, column, or function the finding is about), so rewording a message or retuning a severity between safegres versions never invalidates a committed baseline, and two findings of the same code on the same table stay distinct. Findings that disappear are reported as fixed — re-run `--write-perf-baseline` to lock the win in and stop them regressing silently. The diff is also carried in JSON output as `perf.diff`, and available to library callers as `diffPerf(findings, baseline)`.
|
|
160
197
|
|
|
198
|
+
### Runtime statistics (`--stats`)
|
|
199
|
+
|
|
200
|
+
The `X*` rules read the schema; the `S*` rules read what the workload actually did to it. They come from `pg_stat_user_tables`, `pg_stat_user_indexes` and — when the extension is installed — `pg_stat_statements`, so they only mean something against a database that has served representative traffic. Opt in with `--stats` (which implies `--perf`):
|
|
201
|
+
|
|
202
|
+
| Code | Severity | Check |
|
|
203
|
+
| --- | --- | --- |
|
|
204
|
+
| S1 | medium | **Sequential-scan-dominant table** — seq scans outnumber index scans 10:1 on a table with indexes and ≥ 1000 live rows |
|
|
205
|
+
| S2 | low | **Index the planner has never chosen** (`idx_scan = 0`) and larger than 1 MiB — pure write cost, unless it's there for a rare report |
|
|
206
|
+
| S3 | low | **Dead-tuple bloat** — dead tuples ≥ 20% of live rows; autovacuum is not keeping up |
|
|
207
|
+
| S4 | info | **Statement hotspot** — a statement taking ≥ 5% of sampled execution time whose relations are in scope |
|
|
208
|
+
|
|
209
|
+
Every threshold is a floor, and every floor is configurable, because a counter is only evidence if there is enough of it:
|
|
210
|
+
|
|
211
|
+
```jsonc
|
|
212
|
+
{
|
|
213
|
+
"perf": {
|
|
214
|
+
"stats": {
|
|
215
|
+
"minRows": 1000, // S1/S3: below this a scan is the right plan
|
|
216
|
+
"seqScanRatio": 10, // S1: seq:idx scan ratio that counts as dominant
|
|
217
|
+
"minIndexBytes": 1048576, // S2: ignore indexes too small to be worth dropping
|
|
218
|
+
"deadTupleRatio": 0.2, // S3
|
|
219
|
+
"minTimeShare": 0.05, // S4: share of total sampled time
|
|
220
|
+
"topStatements": 5 // S4: cap
|
|
221
|
+
},
|
|
222
|
+
"scoring": { "includeStats": false } // demote S* to advisories
|
|
223
|
+
}
|
|
224
|
+
}
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
`S*` findings are **scored** on the perf axis — asking for `--stats` is the opt-in — but `perf.scoring.includeStats: false` demotes them to advisories if you want the grade to stay purely deterministic. The report carries its own provenance in `perf.stats`: how many tables were read, when the counters were last reset (the window the numbers describe), whether they were scored, and a note when `pg_stat_statements` isn't installed. Absent statistics are never an error; the audit just says so.
|
|
228
|
+
|
|
229
|
+
### Planner proof (`--explain`)
|
|
230
|
+
|
|
231
|
+
The `X*` rules *infer* from the catalog that nothing can serve a query shape. `--explain` asks the database instead: for each probeable finding it plans the query the finding is a claim about with `EXPLAIN (GENERIC_PLAN, FORMAT JSON)` — nothing is executed, and parameters stay parameters, so no value has to be invented for a column — and attaches the plan as `finding.evidence`.
|
|
232
|
+
|
|
233
|
+
The interesting outcome is disagreement. A finding whose probe plans as an index scan is **refuted**: some index the catalog rules didn't credit (a hash index on an FK column, say) does serve it, so the finding is acknowledged, reported as info, and dropped from the perf score. The reverse is deliberately not symmetrical — an empty or unanalyzed table always seq-scans, so a seq scan **confirms** a finding only above a planner row estimate of 1000 (`perf.explain.minRows`); below that the probe is `inconclusive` and the finding is left exactly as the static rule made it.
|
|
234
|
+
|
|
235
|
+
```bash
|
|
236
|
+
safegres audit --perf --explain --database mydb
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
```
|
|
240
|
+
[medium] X1 app_public.posts
|
|
241
|
+
foreign key posts_author_id_fkey has no covering index
|
|
242
|
+
plan (confirmed): Seq Scan
|
|
243
|
+
[info] X1 app_public.notes
|
|
244
|
+
foreign key notes_author_id_fkey has no covering index — refuted by EXPLAIN (the planner serves this shape with an index)
|
|
245
|
+
plan (refuted): Bitmap Heap Scan → Bitmap Index Scan
|
|
246
|
+
planner proof: 1 confirmed, 1 refuted, 1 inconclusive of 3 probed
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
Probes exist for X1, X2, X7 and X8 — the rules that name a query shape. X5, X6, `P*` and `S*` are claims about the schema or the workload rather than a plan, so nothing is planned speculatively on their behalf. `GENERIC_PLAN` requires PostgreSQL 16+; on older servers the audit reports `perf.explain.unavailable` and leaves the findings untouched.
|
|
250
|
+
|
|
161
251
|
## Call graph (`--call-graph`)
|
|
162
252
|
|
|
163
253
|
RLS findings tell you what the *tables* allow. The call graph tells you what the *functions* reach: starting from the exposed entry points (functions the API roles can `EXECUTE`), safegres statically walks each body and lists every **trust boundary** on the way — unscored, because a public `SECURITY DEFINER` calling private functions is the intended pattern (that's how `sign_in` works). The output is a deterministic checklist for human review:
|
package/checks/indexes.d.ts
CHANGED
|
@@ -36,3 +36,24 @@ export declare function checkRedundantIndexes(table: TableIndexSnapshot): Findin
|
|
|
36
36
|
* realtime/change-feed features break.
|
|
37
37
|
*/
|
|
38
38
|
export declare function checkMissingPrimaryKey(table: TableIndexSnapshot): Finding | null;
|
|
39
|
+
/**
|
|
40
|
+
* X7: a search column with no index the search can use.
|
|
41
|
+
*
|
|
42
|
+
* A `tsvector` or `vector` column is exposed as a search field by
|
|
43
|
+
* `graphile-search` purely from its type — the schema promises a fast path.
|
|
44
|
+
* Without a GIN/GiST (full-text) or HNSW/IVFFlat (vector) index the promise
|
|
45
|
+
* is served by a sequential scan plus a per-row distance or match computation,
|
|
46
|
+
* which is the single worst plan the API can produce.
|
|
47
|
+
*/
|
|
48
|
+
export declare function checkUnindexedSearchColumns(table: TableIndexSnapshot): Finding[];
|
|
49
|
+
/**
|
|
50
|
+
* X8: a sort-shaped column that leads no index.
|
|
51
|
+
*
|
|
52
|
+
* Every PostGraphile connection can be ordered by any column and paginated
|
|
53
|
+
* with a cursor over that order. Sorting on an unindexed column forces a full
|
|
54
|
+
* sort of the (RLS-filtered) result on every page, and keyset pagination
|
|
55
|
+
* degenerates into a scan-and-discard. Timestamp columns are singled out
|
|
56
|
+
* because they are what feeds are actually ordered by — a heuristic, hence
|
|
57
|
+
* `info` by default; turn it off with `perf.rules: { "X8": "off" }`.
|
|
58
|
+
*/
|
|
59
|
+
export declare function checkUnindexedSortColumns(table: TableIndexSnapshot): Finding[];
|
package/checks/indexes.js
CHANGED
|
@@ -9,6 +9,8 @@ Object.defineProperty(exports, "__esModule", { value: true });
|
|
|
9
9
|
exports.checkUnindexedForeignKeys = checkUnindexedForeignKeys;
|
|
10
10
|
exports.checkRedundantIndexes = checkRedundantIndexes;
|
|
11
11
|
exports.checkMissingPrimaryKey = checkMissingPrimaryKey;
|
|
12
|
+
exports.checkUnindexedSearchColumns = checkUnindexedSearchColumns;
|
|
13
|
+
exports.checkUnindexedSortColumns = checkUnindexedSortColumns;
|
|
12
14
|
/**
|
|
13
15
|
* X1: a foreign key with no index that can serve it.
|
|
14
16
|
*
|
|
@@ -114,6 +116,108 @@ function checkMissingPrimaryKey(table) {
|
|
|
114
116
|
context: { replicaIdentity: table.replicaIdentity }
|
|
115
117
|
};
|
|
116
118
|
}
|
|
119
|
+
/**
|
|
120
|
+
* Search column types that only perform with a specialised index, mapped to
|
|
121
|
+
* the access methods that can serve them.
|
|
122
|
+
*
|
|
123
|
+
* The column type *is* the declaration: `graphile-search` detects searchable
|
|
124
|
+
* columns by codec (tsvector → full-text filter, vector → similarity search),
|
|
125
|
+
* so a bare `tsvector` column is already an API promise. BM25 and pg_trgm are
|
|
126
|
+
* absent on purpose — they are discovered *from* their indexes, so a missing
|
|
127
|
+
* index means the feature was never exposed rather than exposed and slow.
|
|
128
|
+
*/
|
|
129
|
+
const SEARCH_TYPES = {
|
|
130
|
+
tsvector: {
|
|
131
|
+
methods: ['gin', 'gist'],
|
|
132
|
+
feature: 'full-text search',
|
|
133
|
+
example: 'USING gin (%s)'
|
|
134
|
+
},
|
|
135
|
+
vector: {
|
|
136
|
+
methods: ['hnsw', 'ivfflat'],
|
|
137
|
+
feature: 'vector similarity search',
|
|
138
|
+
example: 'USING hnsw (%s vector_cosine_ops)'
|
|
139
|
+
},
|
|
140
|
+
halfvec: {
|
|
141
|
+
methods: ['hnsw', 'ivfflat'],
|
|
142
|
+
feature: 'vector similarity search',
|
|
143
|
+
example: 'USING hnsw (%s halfvec_cosine_ops)'
|
|
144
|
+
},
|
|
145
|
+
sparsevec: {
|
|
146
|
+
methods: ['hnsw'],
|
|
147
|
+
feature: 'vector similarity search',
|
|
148
|
+
example: 'USING hnsw (%s sparsevec_cosine_ops)'
|
|
149
|
+
}
|
|
150
|
+
};
|
|
151
|
+
/**
|
|
152
|
+
* X7: a search column with no index the search can use.
|
|
153
|
+
*
|
|
154
|
+
* A `tsvector` or `vector` column is exposed as a search field by
|
|
155
|
+
* `graphile-search` purely from its type — the schema promises a fast path.
|
|
156
|
+
* Without a GIN/GiST (full-text) or HNSW/IVFFlat (vector) index the promise
|
|
157
|
+
* is served by a sequential scan plus a per-row distance or match computation,
|
|
158
|
+
* which is the single worst plan the API can produce.
|
|
159
|
+
*/
|
|
160
|
+
function checkUnindexedSearchColumns(table) {
|
|
161
|
+
if (table.isPartition)
|
|
162
|
+
return [];
|
|
163
|
+
const findings = [];
|
|
164
|
+
for (const column of table.columns) {
|
|
165
|
+
const spec = SEARCH_TYPES[column.baseType];
|
|
166
|
+
if (!spec)
|
|
167
|
+
continue;
|
|
168
|
+
const served = table.indexes.some((idx) => spec.methods.includes(idx.method) && idx.columns.includes(column.attnum));
|
|
169
|
+
if (served)
|
|
170
|
+
continue;
|
|
171
|
+
findings.push({
|
|
172
|
+
code: 'X7',
|
|
173
|
+
severity: 'medium',
|
|
174
|
+
category: 'index',
|
|
175
|
+
schema: table.schema,
|
|
176
|
+
table: table.name,
|
|
177
|
+
message: `${table.schema}.${table.name}.${column.name} is a ${column.type} column with no ${spec.methods.join('/')} index — ${spec.feature} scans the whole table`,
|
|
178
|
+
hint: `CREATE INDEX ON ${table.schema}.${table.name} ${spec.example.replace('%s', column.name)};`,
|
|
179
|
+
context: { column: column.name, type: column.type, methods: spec.methods }
|
|
180
|
+
});
|
|
181
|
+
}
|
|
182
|
+
return findings;
|
|
183
|
+
}
|
|
184
|
+
/** Types whose columns are, in practice, what a connection is sorted by. */
|
|
185
|
+
const SORT_TYPES = new Set(['timestamptz', 'timestamp', 'date']);
|
|
186
|
+
/**
|
|
187
|
+
* X8: a sort-shaped column that leads no index.
|
|
188
|
+
*
|
|
189
|
+
* Every PostGraphile connection can be ordered by any column and paginated
|
|
190
|
+
* with a cursor over that order. Sorting on an unindexed column forces a full
|
|
191
|
+
* sort of the (RLS-filtered) result on every page, and keyset pagination
|
|
192
|
+
* degenerates into a scan-and-discard. Timestamp columns are singled out
|
|
193
|
+
* because they are what feeds are actually ordered by — a heuristic, hence
|
|
194
|
+
* `info` by default; turn it off with `perf.rules: { "X8": "off" }`.
|
|
195
|
+
*/
|
|
196
|
+
function checkUnindexedSortColumns(table) {
|
|
197
|
+
if (table.isPartition)
|
|
198
|
+
return [];
|
|
199
|
+
// A leading column can serve both ASC and DESC ordering; a trailing one
|
|
200
|
+
// cannot serve the sort on its own.
|
|
201
|
+
const leading = new Set(table.indexes.filter((i) => !i.partial).map((i) => i.columns[0]).filter((c) => c !== undefined));
|
|
202
|
+
const findings = [];
|
|
203
|
+
for (const column of table.columns) {
|
|
204
|
+
if (!SORT_TYPES.has(column.baseType))
|
|
205
|
+
continue;
|
|
206
|
+
if (leading.has(column.attnum))
|
|
207
|
+
continue;
|
|
208
|
+
findings.push({
|
|
209
|
+
code: 'X8',
|
|
210
|
+
severity: 'info',
|
|
211
|
+
category: 'index',
|
|
212
|
+
schema: table.schema,
|
|
213
|
+
table: table.name,
|
|
214
|
+
message: `${table.schema}.${table.name}.${column.name} (${column.type}) leads no index — ordering or paginating a connection by it sorts the whole table`,
|
|
215
|
+
hint: `If the API orders by this column, CREATE INDEX ON ${table.schema}.${table.name} (${column.name} DESC); otherwise disable X8 or add the table to perf.ignore.`,
|
|
216
|
+
context: { column: column.name, type: column.type }
|
|
217
|
+
});
|
|
218
|
+
}
|
|
219
|
+
return findings;
|
|
220
|
+
}
|
|
117
221
|
/**
|
|
118
222
|
* True when `index` can serve an equality lookup on exactly `columns`:
|
|
119
223
|
* its leading columns are that same set, and it covers every row.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Runtime-statistics checks (`--stats`, `S*`).
|
|
3
|
+
*
|
|
4
|
+
* These read the cumulative statistics views rather than the catalog, so
|
|
5
|
+
* unlike the `X*` rules they describe a workload rather than a schema: the
|
|
6
|
+
* same database can pass or fail depending on what traffic it has served.
|
|
7
|
+
* Every rule therefore takes a floor, so a cold or freshly-reset database
|
|
8
|
+
* produces silence instead of noise.
|
|
9
|
+
*/
|
|
10
|
+
import type { StatementUsage, StatsSnapshot, TableUsage } from '../pg/stats';
|
|
11
|
+
import type { Finding } from '../types';
|
|
12
|
+
export interface StatsThresholds {
|
|
13
|
+
/** Ignore tables with fewer live rows than this. Default 1000. */
|
|
14
|
+
minRows: number;
|
|
15
|
+
/** S1 fires when seqScans exceed indexScans by this factor. Default 10. */
|
|
16
|
+
seqScanRatio: number;
|
|
17
|
+
/** S2 ignores indexes smaller than this many bytes. Default 1 MiB. */
|
|
18
|
+
minIndexBytes: number;
|
|
19
|
+
/** S3 fires above this dead/live tuple ratio. Default 0.2. */
|
|
20
|
+
deadTupleRatio: number;
|
|
21
|
+
/** S4 fires for statements at or above this share of total time. Default 0.05. */
|
|
22
|
+
minTimeShare: number;
|
|
23
|
+
/** S4 reports at most this many statements. Default 5. */
|
|
24
|
+
topStatements: number;
|
|
25
|
+
}
|
|
26
|
+
export declare const DEFAULT_STATS_THRESHOLDS: StatsThresholds;
|
|
27
|
+
/**
|
|
28
|
+
* S1: a table the planner reaches by sequential scan far more often than by
|
|
29
|
+
* index. On a table above the row floor that is a missing index, a policy
|
|
30
|
+
* predicate the planner can't use, or a query shape nothing covers.
|
|
31
|
+
*/
|
|
32
|
+
export declare function checkSeqScanDominant(table: TableUsage, thresholds: StatsThresholds): Finding | null;
|
|
33
|
+
/**
|
|
34
|
+
* S2: an index the planner has never chosen. It still costs write throughput,
|
|
35
|
+
* disk, and vacuum time. Constraint-backed and unique indexes are exempt:
|
|
36
|
+
* they exist to enforce a constraint, not to be scanned.
|
|
37
|
+
*/
|
|
38
|
+
export declare function checkUnusedIndexes(table: TableUsage, thresholds: StatsThresholds): Finding[];
|
|
39
|
+
/**
|
|
40
|
+
* S3: dead tuples the vacuum is not keeping up with. Bloat inflates every
|
|
41
|
+
* scan of the table and every index on it, and the planner's row estimates
|
|
42
|
+
* drift with it.
|
|
43
|
+
*/
|
|
44
|
+
export declare function checkDeadTuples(table: TableUsage, thresholds: StatsThresholds): Finding | null;
|
|
45
|
+
/**
|
|
46
|
+
* S4: the statements the database actually spends its time in, restricted to
|
|
47
|
+
* ones touching a table in scope. This is not a defect — it is the ranked
|
|
48
|
+
* list to read the other findings against, so it is `info` by default.
|
|
49
|
+
*/
|
|
50
|
+
export declare function checkTopStatements(statements: StatementUsage[], tables: TableUsage[], thresholds: StatsThresholds): Finding[];
|
|
51
|
+
/** Every stats finding for one snapshot. */
|
|
52
|
+
export declare function checkStats(snapshot: StatsSnapshot, thresholds?: StatsThresholds): Finding[];
|
package/checks/stats.js
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* Runtime-statistics checks (`--stats`, `S*`).
|
|
4
|
+
*
|
|
5
|
+
* These read the cumulative statistics views rather than the catalog, so
|
|
6
|
+
* unlike the `X*` rules they describe a workload rather than a schema: the
|
|
7
|
+
* same database can pass or fail depending on what traffic it has served.
|
|
8
|
+
* Every rule therefore takes a floor, so a cold or freshly-reset database
|
|
9
|
+
* produces silence instead of noise.
|
|
10
|
+
*/
|
|
11
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
12
|
+
exports.DEFAULT_STATS_THRESHOLDS = void 0;
|
|
13
|
+
exports.checkSeqScanDominant = checkSeqScanDominant;
|
|
14
|
+
exports.checkUnusedIndexes = checkUnusedIndexes;
|
|
15
|
+
exports.checkDeadTuples = checkDeadTuples;
|
|
16
|
+
exports.checkTopStatements = checkTopStatements;
|
|
17
|
+
exports.checkStats = checkStats;
|
|
18
|
+
exports.DEFAULT_STATS_THRESHOLDS = {
|
|
19
|
+
minRows: 1000,
|
|
20
|
+
seqScanRatio: 10,
|
|
21
|
+
minIndexBytes: 1024 * 1024,
|
|
22
|
+
deadTupleRatio: 0.2,
|
|
23
|
+
minTimeShare: 0.05,
|
|
24
|
+
topStatements: 5
|
|
25
|
+
};
|
|
26
|
+
/**
|
|
27
|
+
* S1: a table the planner reaches by sequential scan far more often than by
|
|
28
|
+
* index. On a table above the row floor that is a missing index, a policy
|
|
29
|
+
* predicate the planner can't use, or a query shape nothing covers.
|
|
30
|
+
*/
|
|
31
|
+
function checkSeqScanDominant(table, thresholds) {
|
|
32
|
+
if (table.liveTuples < thresholds.minRows)
|
|
33
|
+
return null;
|
|
34
|
+
if (table.seqScans === 0)
|
|
35
|
+
return null;
|
|
36
|
+
if (table.seqScans < table.indexScans * thresholds.seqScanRatio)
|
|
37
|
+
return null;
|
|
38
|
+
// A table with no index at all is X6/X1 territory; the stats add nothing.
|
|
39
|
+
if (table.indexes.length === 0)
|
|
40
|
+
return null;
|
|
41
|
+
const perScan = Math.round(table.seqTuplesRead / Math.max(table.seqScans, 1));
|
|
42
|
+
return {
|
|
43
|
+
code: 'S1',
|
|
44
|
+
severity: 'medium',
|
|
45
|
+
category: 'index',
|
|
46
|
+
schema: table.schema,
|
|
47
|
+
table: table.name,
|
|
48
|
+
message: `${table.schema}.${table.name} is scanned sequentially ${table.seqScans} times vs ${table.indexScans} index scans (${table.liveTuples} live rows, ~${perScan} rows read per scan)`,
|
|
49
|
+
hint: 'Find the query shape behind the scans (pg_stat_statements, auto_explain) and index for it — at this row count every seq scan reads the whole table.',
|
|
50
|
+
context: {
|
|
51
|
+
seqScans: table.seqScans,
|
|
52
|
+
indexScans: table.indexScans,
|
|
53
|
+
liveTuples: table.liveTuples
|
|
54
|
+
}
|
|
55
|
+
};
|
|
56
|
+
}
|
|
57
|
+
/**
|
|
58
|
+
* S2: an index the planner has never chosen. It still costs write throughput,
|
|
59
|
+
* disk, and vacuum time. Constraint-backed and unique indexes are exempt:
|
|
60
|
+
* they exist to enforce a constraint, not to be scanned.
|
|
61
|
+
*/
|
|
62
|
+
function checkUnusedIndexes(table, thresholds) {
|
|
63
|
+
const findings = [];
|
|
64
|
+
for (const index of table.indexes) {
|
|
65
|
+
if (index.scans > 0)
|
|
66
|
+
continue;
|
|
67
|
+
if (index.unique || index.constraint)
|
|
68
|
+
continue;
|
|
69
|
+
if (index.sizeBytes < thresholds.minIndexBytes)
|
|
70
|
+
continue;
|
|
71
|
+
findings.push({
|
|
72
|
+
code: 'S2',
|
|
73
|
+
severity: 'low',
|
|
74
|
+
category: 'index',
|
|
75
|
+
schema: table.schema,
|
|
76
|
+
table: table.name,
|
|
77
|
+
message: `Index ${index.name} on ${table.schema}.${table.name} has never been scanned (${formatBytes(index.sizeBytes)})`,
|
|
78
|
+
hint: `DROP INDEX ${table.schema}.${index.name}; — confirm the counters cover a representative window first (see perf.stats.since).`,
|
|
79
|
+
context: { index: index.name, sizeBytes: index.sizeBytes }
|
|
80
|
+
});
|
|
81
|
+
}
|
|
82
|
+
return findings;
|
|
83
|
+
}
|
|
84
|
+
/**
|
|
85
|
+
* S3: dead tuples the vacuum is not keeping up with. Bloat inflates every
|
|
86
|
+
* scan of the table and every index on it, and the planner's row estimates
|
|
87
|
+
* drift with it.
|
|
88
|
+
*/
|
|
89
|
+
function checkDeadTuples(table, thresholds) {
|
|
90
|
+
if (table.liveTuples < thresholds.minRows)
|
|
91
|
+
return null;
|
|
92
|
+
const ratio = table.deadTuples / Math.max(table.liveTuples, 1);
|
|
93
|
+
if (ratio < thresholds.deadTupleRatio)
|
|
94
|
+
return null;
|
|
95
|
+
return {
|
|
96
|
+
code: 'S3',
|
|
97
|
+
severity: 'low',
|
|
98
|
+
category: 'index',
|
|
99
|
+
schema: table.schema,
|
|
100
|
+
table: table.name,
|
|
101
|
+
message: `${table.schema}.${table.name} is ${Math.round(ratio * 100)}% dead tuples (${table.deadTuples} dead / ${table.liveTuples} live${table.lastVacuum ? `, last vacuumed ${table.lastVacuum}` : ', never vacuumed'})`,
|
|
102
|
+
hint: 'VACUUM (ANALYZE) the table, then tune autovacuum for it (autovacuum_vacuum_scale_factor) — bloat is read by every scan and every index on the table.',
|
|
103
|
+
context: {
|
|
104
|
+
deadTuples: table.deadTuples,
|
|
105
|
+
liveTuples: table.liveTuples,
|
|
106
|
+
lastVacuum: table.lastVacuum
|
|
107
|
+
}
|
|
108
|
+
};
|
|
109
|
+
}
|
|
110
|
+
/**
|
|
111
|
+
* S4: the statements the database actually spends its time in, restricted to
|
|
112
|
+
* ones touching a table in scope. This is not a defect — it is the ranked
|
|
113
|
+
* list to read the other findings against, so it is `info` by default.
|
|
114
|
+
*/
|
|
115
|
+
function checkTopStatements(statements, tables, thresholds) {
|
|
116
|
+
const totalTime = statements.reduce((sum, s) => sum + s.totalTimeMs, 0);
|
|
117
|
+
if (totalTime <= 0)
|
|
118
|
+
return [];
|
|
119
|
+
const findings = [];
|
|
120
|
+
for (const statement of statements) {
|
|
121
|
+
if (findings.length >= thresholds.topStatements)
|
|
122
|
+
break;
|
|
123
|
+
const share = statement.totalTimeMs / totalTime;
|
|
124
|
+
if (share < thresholds.minTimeShare)
|
|
125
|
+
continue;
|
|
126
|
+
const matched = tables.filter((t) => statementTouches(statement.query, t));
|
|
127
|
+
if (matched.length === 0)
|
|
128
|
+
continue;
|
|
129
|
+
const relations = matched.map((t) => `${t.schema}.${t.name}`);
|
|
130
|
+
findings.push({
|
|
131
|
+
code: 'S4',
|
|
132
|
+
severity: 'info',
|
|
133
|
+
category: 'index',
|
|
134
|
+
schema: matched[0].schema,
|
|
135
|
+
table: matched[0].name,
|
|
136
|
+
message: `${Math.round(share * 100)}% of database execution time is spent in one statement over ${relations.join(', ')} (${statement.calls} calls, ${Math.round(statement.meanTimeMs)}ms mean)`,
|
|
137
|
+
hint: `EXPLAIN (ANALYZE, BUFFERS) this statement before acting on any other finding for these tables:\n ${truncate(statement.query, 300)}`,
|
|
138
|
+
context: {
|
|
139
|
+
statement: truncate(statement.query, 300),
|
|
140
|
+
calls: statement.calls,
|
|
141
|
+
totalTimeMs: Math.round(statement.totalTimeMs),
|
|
142
|
+
meanTimeMs: Math.round(statement.meanTimeMs),
|
|
143
|
+
relations
|
|
144
|
+
}
|
|
145
|
+
});
|
|
146
|
+
}
|
|
147
|
+
return findings;
|
|
148
|
+
}
|
|
149
|
+
/** Every stats finding for one snapshot. */
|
|
150
|
+
function checkStats(snapshot, thresholds = exports.DEFAULT_STATS_THRESHOLDS) {
|
|
151
|
+
const findings = [];
|
|
152
|
+
for (const table of snapshot.tables) {
|
|
153
|
+
const s1 = checkSeqScanDominant(table, thresholds);
|
|
154
|
+
if (s1)
|
|
155
|
+
findings.push(s1);
|
|
156
|
+
findings.push(...checkUnusedIndexes(table, thresholds));
|
|
157
|
+
const s3 = checkDeadTuples(table, thresholds);
|
|
158
|
+
if (s3)
|
|
159
|
+
findings.push(s3);
|
|
160
|
+
}
|
|
161
|
+
if (snapshot.statements) {
|
|
162
|
+
findings.push(...checkTopStatements(snapshot.statements, snapshot.tables, thresholds));
|
|
163
|
+
}
|
|
164
|
+
return findings;
|
|
165
|
+
}
|
|
166
|
+
/**
|
|
167
|
+
* Whether a normalised statement references a table. `pg_stat_statements`
|
|
168
|
+
* stores text, not parsed relations, so this matches the qualified name or
|
|
169
|
+
* the bare name on a word boundary — good enough to rank hotspots, and
|
|
170
|
+
* deliberately not used for anything that changes a severity.
|
|
171
|
+
*/
|
|
172
|
+
function statementTouches(query, table) {
|
|
173
|
+
const qualified = new RegExp(`\\b${escapeRegExp(table.schema)}\\.${escapeRegExp(table.name)}\\b`, 'i');
|
|
174
|
+
if (qualified.test(query))
|
|
175
|
+
return true;
|
|
176
|
+
const bare = new RegExp(`\\b(from|join|into|update)\\s+"?${escapeRegExp(table.name)}"?\\b`, 'i');
|
|
177
|
+
return bare.test(query);
|
|
178
|
+
}
|
|
179
|
+
function escapeRegExp(value) {
|
|
180
|
+
return value.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
|
181
|
+
}
|
|
182
|
+
function truncate(value, max) {
|
|
183
|
+
const collapsed = value.replace(/\s+/g, ' ').trim();
|
|
184
|
+
return collapsed.length <= max ? collapsed : `${collapsed.slice(0, max - 1)}…`;
|
|
185
|
+
}
|
|
186
|
+
function formatBytes(bytes) {
|
|
187
|
+
if (bytes >= 1024 * 1024 * 1024)
|
|
188
|
+
return `${(bytes / 1024 / 1024 / 1024).toFixed(1)} GiB`;
|
|
189
|
+
if (bytes >= 1024 * 1024)
|
|
190
|
+
return `${(bytes / 1024 / 1024).toFixed(1)} MiB`;
|
|
191
|
+
return `${Math.round(bytes / 1024)} KiB`;
|
|
192
|
+
}
|
package/cli/audit.js
CHANGED
|
@@ -40,7 +40,9 @@ const audit_1 = require("../commands/audit");
|
|
|
40
40
|
const loader_1 = require("../config/loader");
|
|
41
41
|
const baseline_2 = require("../perf/baseline");
|
|
42
42
|
const json_1 = require("../report/json");
|
|
43
|
+
const markdown_1 = require("../report/markdown");
|
|
43
44
|
const pretty_1 = require("../report/pretty");
|
|
45
|
+
const sarif_1 = require("../report/sarif");
|
|
44
46
|
const score_1 = require("../score/score");
|
|
45
47
|
const types_1 = require("../types");
|
|
46
48
|
const shared_1 = require("./shared");
|
|
@@ -90,7 +92,7 @@ Exposure (what the score is computed against):
|
|
|
90
92
|
--exposed-only Hide internal (non-exposed) findings from output
|
|
91
93
|
|
|
92
94
|
Performance dimension (optional; scored separately from security):
|
|
93
|
-
--perf Also audit index hygiene (X1/X5/X6), policy-aware
|
|
95
|
+
--perf Also audit index hygiene (X1/X5/X6/X7/X8), policy-aware
|
|
94
96
|
index coverage (X2/X3/X4) and policy cost (P1/P1b),
|
|
95
97
|
scored on its own 0-100 axis (report.perf)
|
|
96
98
|
--fail-on-perf-score <n> Exit non-zero if the perf score is below n (0-100)
|
|
@@ -101,6 +103,15 @@ Performance dimension (optional; scored separately from security):
|
|
|
101
103
|
--perf-baseline <file> Diff perf findings against a committed baseline and
|
|
102
104
|
report only NEW debt (implies --perf)
|
|
103
105
|
--fail-on-new-perf Exit non-zero when --perf-baseline finds new debt
|
|
106
|
+
--stats Also run the runtime-statistics rules (S1-S4) from
|
|
107
|
+
pg_stat_user_tables and, when installed,
|
|
108
|
+
pg_stat_statements (implies --perf). Workload-
|
|
109
|
+
dependent: only meaningful against a database that
|
|
110
|
+
has served representative traffic
|
|
111
|
+
--explain Prove each probeable perf finding with
|
|
112
|
+
EXPLAIN (GENERIC_PLAN) and attach the plan as
|
|
113
|
+
evidence; findings the planner refutes are reported
|
|
114
|
+
but unscored (implies --perf, needs PostgreSQL 16+)
|
|
104
115
|
|
|
105
116
|
Call graph (unscored; human review):
|
|
106
117
|
--call-graph Analyze the functions reachable from the exposed entry
|
|
@@ -118,7 +129,12 @@ Audit options:
|
|
|
118
129
|
--exclude-schemas <csv> Skip these schemas
|
|
119
130
|
--roles <csv> Audit grants only for these roles (default: all)
|
|
120
131
|
--exclude-roles <csv> Skip grants for these roles
|
|
121
|
-
--format <fmt> "pretty" (default) | "json" | "json-pretty"
|
|
132
|
+
--format <fmt> "pretty" (default) | "json" | "json-pretty" | "markdown" | "sarif"
|
|
133
|
+
(markdown is for CI: a GitHub job summary or PR comment;
|
|
134
|
+
sarif uploads to GitHub code scanning)
|
|
135
|
+
--sarif-sources <dir> With --format sarif: scan <dir> for the CREATE TABLE /
|
|
136
|
+
CREATE POLICY that defines each object, so alerts point
|
|
137
|
+
at the SQL that produced the finding
|
|
122
138
|
--summary, -q Print only exposure, score, and severity counts (no findings)
|
|
123
139
|
--verbose Expand internal (non-exposed) advisories (listed as a count otherwise)
|
|
124
140
|
--fail-on <severity> Exit non-zero if any finding >= severity
|
|
@@ -150,6 +166,8 @@ exports.default = async (argv, _prompter, _options) => {
|
|
|
150
166
|
|| typeof argv['write-perf-baseline'] === 'string'
|
|
151
167
|
? true
|
|
152
168
|
: undefined,
|
|
169
|
+
stats: argv.stats === true ? true : undefined,
|
|
170
|
+
explain: argv.explain === true ? true : undefined,
|
|
153
171
|
callGraph: argv['call-graph'] === true
|
|
154
172
|
|| typeof argv.baseline === 'string'
|
|
155
173
|
|| typeof argv['write-baseline'] === 'string',
|
|
@@ -216,6 +234,20 @@ exports.default = async (argv, _prompter, _options) => {
|
|
|
216
234
|
case 'json-pretty':
|
|
217
235
|
output = (0, json_1.renderJson)(report, { pretty: true });
|
|
218
236
|
break;
|
|
237
|
+
case 'sarif':
|
|
238
|
+
output = (0, sarif_1.renderSarif)(report, {
|
|
239
|
+
sources: typeof argv['sarif-sources'] === 'string'
|
|
240
|
+
? (0, sarif_1.buildSourceIndex)(argv['sarif-sources'])
|
|
241
|
+
: undefined
|
|
242
|
+
});
|
|
243
|
+
break;
|
|
244
|
+
case 'markdown':
|
|
245
|
+
case 'md':
|
|
246
|
+
output = (0, markdown_1.renderMarkdown)(report, {
|
|
247
|
+
summary: argv.summary === true,
|
|
248
|
+
verbose: argv.verbose === true
|
|
249
|
+
});
|
|
250
|
+
break;
|
|
219
251
|
case 'pretty':
|
|
220
252
|
output = (0, pretty_1.renderPretty)(report, {
|
|
221
253
|
color: colorEnabled,
|
package/commands/audit.d.ts
CHANGED
|
@@ -33,6 +33,17 @@ export interface AuditOptions extends IntrospectOptions {
|
|
|
33
33
|
* `report.perf` with its own score. Overrides `config.perf.enabled`.
|
|
34
34
|
*/
|
|
35
35
|
perf?: boolean;
|
|
36
|
+
/**
|
|
37
|
+
* Collect runtime statistics (`pg_stat_user_tables`, `pg_stat_statements`)
|
|
38
|
+
* and run the `S*` rules. Implies `perf`. Overrides `config.perf.stats.enabled`.
|
|
39
|
+
*/
|
|
40
|
+
stats?: boolean;
|
|
41
|
+
/**
|
|
42
|
+
* Probe perf findings with `EXPLAIN (GENERIC_PLAN)` and attach the plan as
|
|
43
|
+
* evidence, acknowledging findings the planner refutes. Implies `perf`.
|
|
44
|
+
* Overrides `config.perf.explain.enabled`.
|
|
45
|
+
*/
|
|
46
|
+
explain?: boolean;
|
|
36
47
|
/**
|
|
37
48
|
* Merged safegres configuration (rules, overrides, scoring). Rule settings
|
|
38
49
|
* filter and retune findings; scoring settings drive the report score.
|