turbine-orm 0.55.0 → 0.56.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/cjs/cli/index.d.ts +3 -1
- package/dist/cjs/cli/index.js +117 -7
- package/dist/cjs/client.d.ts +16 -1
- package/dist/cjs/client.js +4 -0
- package/dist/cjs/dialect.d.ts +9 -0
- package/dist/cjs/index-stats.d.ts +46 -0
- package/dist/cjs/index-stats.js +42 -1
- package/dist/cjs/plan-divergence.d.ts +244 -0
- package/dist/cjs/plan-divergence.js +444 -0
- package/dist/cjs/powql.d.ts +11 -0
- package/dist/cjs/powql.js +22 -0
- package/dist/cjs/query/builder.d.ts +45 -0
- package/dist/cjs/query/builder.js +90 -17
- package/dist/cjs/query/deferred.d.ts +9 -0
- package/dist/cjs/query/types.d.ts +140 -0
- package/dist/cjs/query/utils.d.ts +14 -0
- package/dist/cjs/query/utils.js +17 -3
- package/dist/cli/index.d.ts +3 -1
- package/dist/cli/index.js +117 -7
- package/dist/client.d.ts +16 -1
- package/dist/client.js +4 -0
- package/dist/dialect.d.ts +9 -0
- package/dist/index-stats.d.ts +46 -0
- package/dist/index-stats.js +42 -1
- package/dist/plan-divergence.d.ts +244 -0
- package/dist/plan-divergence.js +438 -0
- package/dist/powql.d.ts +11 -0
- package/dist/powql.js +22 -0
- package/dist/query/builder.d.ts +45 -0
- package/dist/query/builder.js +90 -17
- package/dist/query/deferred.d.ts +9 -0
- package/dist/query/types.d.ts +140 -0
- package/dist/query/utils.d.ts +14 -0
- package/dist/query/utils.js +17 -3
- package/package.json +1 -1
|
@@ -0,0 +1,444 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* Plan-divergence advisor, the third question doctor asks about a probe column.
|
|
4
|
+
*
|
|
5
|
+
* `index-advisor.ts` answers "which relation probes have no index" from pure
|
|
6
|
+
* topology. `index-stats.ts` answers "is adding that index worth it" from live
|
|
7
|
+
* statistics. This module answers the third question on the SAME columns: "this
|
|
8
|
+
* column IS indexed, and its value distribution makes a NAMED prepared
|
|
9
|
+
* statement's generic plan unsafe".
|
|
10
|
+
*
|
|
11
|
+
* THE MECHANISM. Postgres may promote a named prepared statement to a GENERIC
|
|
12
|
+
* plan from its sixth execution onward, and ONLY when the generic plan's
|
|
13
|
+
* estimated cost is not worse than the average custom cost. A generic plan
|
|
14
|
+
* cannot see any parameter value, so it substitutes a default for each one:
|
|
15
|
+
*
|
|
16
|
+
* - an unknown equality `col = $1` is estimated as reltuples / n_distinct;
|
|
17
|
+
* - an unknown `LIMIT $n` is estimated as 10% of the CHILD node's row estimate.
|
|
18
|
+
*
|
|
19
|
+
* THE ONE SHAPE THIS MODELS, stated narrowly on purpose. A read shaped
|
|
20
|
+
* `WHERE col = $1 ORDER BY <other indexed column> LIMIT $n`, where the generic
|
|
21
|
+
* estimate for `col = $1` sits ABOVE the plan boundary (so the generic plan
|
|
22
|
+
* keeps the ordered index scan and filters) while some real values sit far
|
|
23
|
+
* BELOW it (so for those values the ordered scan has to walk a large fraction
|
|
24
|
+
* of the table before it accumulates one page of matches, where the custom
|
|
25
|
+
* plan takes a bitmap scan over the value's own rows and sorts them).
|
|
26
|
+
*
|
|
27
|
+
* WHAT THIS DELIBERATELY DOES NOT MODEL, because it was tried and it did not
|
|
28
|
+
* work. An earlier revision carried a second rule for the opposite direction (a
|
|
29
|
+
* physically clustered column whose densest value is far ABOVE the generic
|
|
30
|
+
* estimate). Measured against live fixtures it was wrong more often than right,
|
|
31
|
+
* and twice it was wrong with the SIGN INVERTED: it predicted "at least 16,032x"
|
|
32
|
+
* and "at least 2,675x" on columns where the generic plan was in fact 10x and
|
|
33
|
+
* 105x BETTER than the custom one, so acting on the finding would have made
|
|
34
|
+
* those reads dramatically slower. The reason is structural, not a bad constant:
|
|
35
|
+
* whether that flip helps or hurts turns on WHERE in the heap the value's rows
|
|
36
|
+
* physically sit, and no pg_stats input carries that. `correlation` is a
|
|
37
|
+
* whole-column property and is identical whether the dominant band is at the
|
|
38
|
+
* head of the heap or its tail. The rule was removed rather than retuned.
|
|
39
|
+
*
|
|
40
|
+
* The same blind spot bounds what remains: this check can see how MANY rows a
|
|
41
|
+
* value has, not WHERE they are, so it under-reports (a rare value packed at the
|
|
42
|
+
* end of the heap is worse than modelled) and it cannot see the third-party
|
|
43
|
+
* shapes where a generic plan is the better one. A clean report is not evidence
|
|
44
|
+
* of immunity.
|
|
45
|
+
*
|
|
46
|
+
* EXPOSURE IS NOT AN INCIDENT. A finding says the DISTRIBUTION admits a
|
|
47
|
+
* damaging flip. It does NOT say the backend is choosing the bad plan today:
|
|
48
|
+
* `auto` promotes only when the generic plan's ESTIMATED cost is not worse than
|
|
49
|
+
* the average custom cost, and on a shape whose generic plan is estimated
|
|
50
|
+
* expensive it never promotes at all. That is why every finding ships a
|
|
51
|
+
* diagnostic that checks `pg_prepared_statements.generic_plans` BEFORE it
|
|
52
|
+
* compares the two plans: the counter is the only thing that establishes real
|
|
53
|
+
* exposure.
|
|
54
|
+
*
|
|
55
|
+
* TWO UNITS, and mixing them up is what made the removed rule wrong. The
|
|
56
|
+
* PLANNER costs the boundary in PAGES (an ordered index scan reads about
|
|
57
|
+
* limit / matching of the table's pages before it fills the limit), which is why
|
|
58
|
+
* {@link crossoverRows} is derived from relpages and why measured flip points
|
|
59
|
+
* track it. The DAMAGE a user feels is also read in pages here, deliberately
|
|
60
|
+
* conservatively; when the ordering index is not correlated with heap order,
|
|
61
|
+
* each row examined is a separate block access and the true buffer count is
|
|
62
|
+
* larger, sometimes by an order of magnitude.
|
|
63
|
+
*
|
|
64
|
+
* FRESHNESS GATE. The cost-tier half of doctor gates on `stats_reset` age,
|
|
65
|
+
* because it normalizes WRITE COUNTERS by it. Nothing here reads a counter: every
|
|
66
|
+
* input comes from pg_stats, which is refreshed by ANALYZE. This check therefore
|
|
67
|
+
* gates on ANALYZE freshness instead (a column with no pg_stats row, or an
|
|
68
|
+
* n_distinct of 0, is suppressed with a notice, and every finding carries the
|
|
69
|
+
* table's last_analyze). Reusing the counter gate would have silenced the check
|
|
70
|
+
* on every cluster whose `pg_stat_database.stats_reset` is NULL, which is the
|
|
71
|
+
* default state and has nothing to do with whether ANALYZE has ever run.
|
|
72
|
+
*
|
|
73
|
+
* Postgres-only (pg_stats + plan_cache_mode). Pure: no pg import, no EXPLAIN.
|
|
74
|
+
*/
|
|
75
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
76
|
+
exports.PLAN_DIVERGENCE_THRESHOLDS = void 0;
|
|
77
|
+
exports.collectDivergenceCandidateColumns = collectDivergenceCandidateColumns;
|
|
78
|
+
exports.crossoverRows = crossoverRows;
|
|
79
|
+
exports.findPlanDivergence = findPlanDivergence;
|
|
80
|
+
const index_stats_js_1 = require("./index-stats.js");
|
|
81
|
+
const utils_js_1 = require("./query/utils.js");
|
|
82
|
+
// ---------------------------------------------------------------------------
|
|
83
|
+
// Thresholds
|
|
84
|
+
// ---------------------------------------------------------------------------
|
|
85
|
+
/**
|
|
86
|
+
* Gates behind a plan-divergence finding. Exported and printed with every
|
|
87
|
+
* finding so the reasoning is never a black box.
|
|
88
|
+
*
|
|
89
|
+
* Each one is either derived from the cost model and then checked against
|
|
90
|
+
* measurement, or fitted to a measured flip boundary; the comments record which.
|
|
91
|
+
*/
|
|
92
|
+
exports.PLAN_DIVERGENCE_THRESHOLDS = {
|
|
93
|
+
/**
|
|
94
|
+
* The LIMIT the advisor assumes, because it cannot see the application's.
|
|
95
|
+
*
|
|
96
|
+
* It is NOT a conservative bound in either direction, and an earlier comment
|
|
97
|
+
* here claimed it was. The crossover grows as sqrt(limit), so raising the
|
|
98
|
+
* limit moves BOTH gates: `rarestBucket < crossover` gets easier to satisfy
|
|
99
|
+
* and `genericEstimate >= crossover` gets harder, and the second one can turn
|
|
100
|
+
* a finding off. 20 is simply a common first page. Every finding also reports
|
|
101
|
+
* the crossover at {@link wideLimit} so a caller paginating in thousands can
|
|
102
|
+
* see where their own limit lands.
|
|
103
|
+
*/
|
|
104
|
+
assumedLimit: 20,
|
|
105
|
+
/** A second crossover reported alongside, for callers paginating in thousands. */
|
|
106
|
+
wideLimit: 1_000,
|
|
107
|
+
/**
|
|
108
|
+
* How many of the table's pages the WRONG plan must walk before this is worth
|
|
109
|
+
* a user's attention: ~400 KB of heap. Below it the flip cannot cost more than
|
|
110
|
+
* a millisecond or two however bad the ratio looks.
|
|
111
|
+
*
|
|
112
|
+
* This replaced a `relpages >= 1000` table-size floor, which was wrong and
|
|
113
|
+
* measurably so: a wrong plan on a SMALL table is not cheap, because an ordered
|
|
114
|
+
* index scan's cost is driven by how much of the table it walks, not by how big
|
|
115
|
+
* the table is. On a 285-page / 18,500-row fixture the generic plan read 336
|
|
116
|
+
* buffers against the custom plan's 7 (48x), and the old floor dropped that
|
|
117
|
+
* column without even counting it as considered.
|
|
118
|
+
*/
|
|
119
|
+
minWalkPages: 50,
|
|
120
|
+
/**
|
|
121
|
+
* ...and it must walk at least this FRACTION of the table. The two gates are
|
|
122
|
+
* different questions (absolute cost, and how badly the plan is mismatched to
|
|
123
|
+
* the value), and a finding needs both.
|
|
124
|
+
*/
|
|
125
|
+
minWalkFraction: 0.1,
|
|
126
|
+
};
|
|
127
|
+
// ---------------------------------------------------------------------------
|
|
128
|
+
// Candidate enumeration (schema-only)
|
|
129
|
+
// ---------------------------------------------------------------------------
|
|
130
|
+
/** True when `column` is unique on its own, so `col = $1` always matches one row. */
|
|
131
|
+
function isUniqueOnColumn(meta, column) {
|
|
132
|
+
if (meta.primaryKey.length === 1 && meta.primaryKey[0] === column)
|
|
133
|
+
return true;
|
|
134
|
+
return meta.uniqueColumns.some((cols) => cols.length === 1 && cols[0] === column);
|
|
135
|
+
}
|
|
136
|
+
/**
|
|
137
|
+
* Every column whose value distribution is worth reading: a column Turbine
|
|
138
|
+
* probes by equality (a relation FK) or that the schema already indexes as a
|
|
139
|
+
* leading key, minus the columns that are unique on their own (a unique
|
|
140
|
+
* equality matches one row, so both plans agree).
|
|
141
|
+
*
|
|
142
|
+
* Purely schema-derived, so the collector knows what to read BEFORE any stats
|
|
143
|
+
* exist. Whether the column is really served by a btree is decided later,
|
|
144
|
+
* against the live index list.
|
|
145
|
+
*/
|
|
146
|
+
function collectDivergenceCandidateColumns(schema) {
|
|
147
|
+
const seen = new Set();
|
|
148
|
+
const out = [];
|
|
149
|
+
const add = (table, column) => {
|
|
150
|
+
const meta = schema.tables[table];
|
|
151
|
+
if (!meta)
|
|
152
|
+
return;
|
|
153
|
+
if (!meta.allColumns.includes(column))
|
|
154
|
+
return;
|
|
155
|
+
if (isUniqueOnColumn(meta, column))
|
|
156
|
+
return;
|
|
157
|
+
const key = `${table}.${column}`;
|
|
158
|
+
if (seen.has(key))
|
|
159
|
+
return;
|
|
160
|
+
seen.add(key);
|
|
161
|
+
out.push({ table, column });
|
|
162
|
+
};
|
|
163
|
+
for (const meta of Object.values(schema.tables)) {
|
|
164
|
+
// Relation probe columns: the equality predicates Turbine itself emits.
|
|
165
|
+
for (const relDef of Object.values(meta.relations)) {
|
|
166
|
+
const target = schema.tables[relDef.to];
|
|
167
|
+
if (!target)
|
|
168
|
+
continue;
|
|
169
|
+
const keys = relDef.type === 'belongsTo' ? relDef.referenceKey : relDef.foreignKey;
|
|
170
|
+
const cols = Array.isArray(keys) ? keys : [keys];
|
|
171
|
+
// Only a SINGLE-column probe has a meaningful single-column distribution;
|
|
172
|
+
// a composite predicate multiplies selectivities and is not modelled here.
|
|
173
|
+
if (cols.length === 1 && cols[0] !== undefined)
|
|
174
|
+
add(relDef.to, cols[0]);
|
|
175
|
+
if (relDef.type === 'manyToMany' && relDef.through) {
|
|
176
|
+
const src = Array.isArray(relDef.through.sourceKey) ? relDef.through.sourceKey : [relDef.through.sourceKey];
|
|
177
|
+
if (src.length === 1 && src[0] !== undefined)
|
|
178
|
+
add(relDef.through.table, src[0]);
|
|
179
|
+
}
|
|
180
|
+
}
|
|
181
|
+
// Leading index columns: whatever the application already filters on.
|
|
182
|
+
for (const idx of meta.indexes) {
|
|
183
|
+
if (idx.docPath)
|
|
184
|
+
continue;
|
|
185
|
+
const lead = idx.columns[0];
|
|
186
|
+
if (lead !== undefined && lead !== index_stats_js_1.EXPRESSION_COLUMN)
|
|
187
|
+
add(meta.name, lead);
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
return out.sort((a, b) => a.table.localeCompare(b.table) || a.column.localeCompare(b.column));
|
|
191
|
+
}
|
|
192
|
+
// ---------------------------------------------------------------------------
|
|
193
|
+
// Live index shape
|
|
194
|
+
// ---------------------------------------------------------------------------
|
|
195
|
+
/** A plain, valid btree with no expression column and no partial predicate. */
|
|
196
|
+
function isPlainBtree(idx) {
|
|
197
|
+
if (!idx.isValid)
|
|
198
|
+
return false;
|
|
199
|
+
if (idx.accessMethod !== undefined && idx.accessMethod !== 'btree')
|
|
200
|
+
return false;
|
|
201
|
+
if (idx.hasExpressions === true)
|
|
202
|
+
return false;
|
|
203
|
+
if (idx.columns.includes(index_stats_js_1.EXPRESSION_COLUMN))
|
|
204
|
+
return false;
|
|
205
|
+
return idx.predicate == null;
|
|
206
|
+
}
|
|
207
|
+
/**
|
|
208
|
+
* Whether a plain btree LEADS with `column` (so the planner has an index path
|
|
209
|
+
* for `column = $1`), and whether the table also has a DIFFERENT ordering index
|
|
210
|
+
* the generic plan can run away with (in practice the primary key).
|
|
211
|
+
*
|
|
212
|
+
* Both halves matter. With NO index on the filter column the two plans are
|
|
213
|
+
* identical and equally bad, which is the EXISTING missing-index finding, not
|
|
214
|
+
* this one. With no alternative ordering index there is no other plan to flip to.
|
|
215
|
+
*/
|
|
216
|
+
function indexShapeFor(snapshot, meta, column) {
|
|
217
|
+
let hasIdx = false;
|
|
218
|
+
let orderColumn = null;
|
|
219
|
+
const pk = meta.primaryKey.length > 0 ? meta.primaryKey[0] : undefined;
|
|
220
|
+
for (const idx of snapshot.indexes) {
|
|
221
|
+
if (idx.table !== meta.name)
|
|
222
|
+
continue;
|
|
223
|
+
if (!isPlainBtree(idx))
|
|
224
|
+
continue;
|
|
225
|
+
const lead = idx.columns[0];
|
|
226
|
+
if (lead === undefined)
|
|
227
|
+
continue;
|
|
228
|
+
if (lead === column) {
|
|
229
|
+
hasIdx = true;
|
|
230
|
+
continue;
|
|
231
|
+
}
|
|
232
|
+
// Prefer the primary key as the stated ordering column: it is the one a
|
|
233
|
+
// paginated read almost always orders by, and it is the plan the generic
|
|
234
|
+
// estimate actually chose in every measured case.
|
|
235
|
+
if (orderColumn === null || lead === pk)
|
|
236
|
+
orderColumn = lead;
|
|
237
|
+
}
|
|
238
|
+
return { hasIdx, orderColumn };
|
|
239
|
+
}
|
|
240
|
+
// ---------------------------------------------------------------------------
|
|
241
|
+
// Detection
|
|
242
|
+
// ---------------------------------------------------------------------------
|
|
243
|
+
/**
|
|
244
|
+
* The plan-boundary crossover: below this many truly matching rows, running
|
|
245
|
+
* `ORDER BY <other column> LIMIT n` as bitmap-scan + sort is cheaper than
|
|
246
|
+
* walking the ordering index and filtering.
|
|
247
|
+
*
|
|
248
|
+
* Derivation: an ordered index scan expects to read about (limit / matching) x
|
|
249
|
+
* pages before it accumulates `limit` matches; a bitmap scan reads about
|
|
250
|
+
* min(matching, pages). Those are equal at matching = sqrt(limit x pages).
|
|
251
|
+
* Measured flip points tracked this within 13 to 26% across two orders of
|
|
252
|
+
* magnitude of limit, which is why the size input is relpages rather than a
|
|
253
|
+
* row count.
|
|
254
|
+
*
|
|
255
|
+
* PREMISE, and it is the same blind spot the whole check has: both halves
|
|
256
|
+
* assume the matching rows are SCATTERED through the heap. On a clustered
|
|
257
|
+
* column a bitmap scan reads far fewer than min(matching, pages), so there is
|
|
258
|
+
* no flip at any limit and this crossover does not describe the table at all.
|
|
259
|
+
* pg_stats.correlation hints at it but does not settle it (a column can be
|
|
260
|
+
* uncorrelated overall and still have one value packed in the tail, which is
|
|
261
|
+
* exactly the counterexample fixture). Treat a crossover as a reason to run
|
|
262
|
+
* the diagnostic block, never as a measurement.
|
|
263
|
+
*/
|
|
264
|
+
function crossoverRows(pages, limit) {
|
|
265
|
+
return Math.sqrt(Math.max(0, limit) * Math.max(0, pages));
|
|
266
|
+
}
|
|
267
|
+
/** Decode pg_stats.n_distinct: negative values are a fraction of the row count. */
|
|
268
|
+
function decodeDistinct(nDistinct, rows) {
|
|
269
|
+
return nDistinct > 0 ? nDistinct : -nDistinct * rows;
|
|
270
|
+
}
|
|
271
|
+
/**
|
|
272
|
+
* The check that settles a finding. Three steps, in the order that makes them
|
|
273
|
+
* mean something:
|
|
274
|
+
*
|
|
275
|
+
* 1. Does `auto` actually promote this shape? `pg_prepared_statements`
|
|
276
|
+
* reports `generic_plans` per cached statement, and until that counter
|
|
277
|
+
* leaves 0 the backend is planning with the real values and there is
|
|
278
|
+
* nothing to fix. This step comes FIRST because a finding describes
|
|
279
|
+
* exposure, not an incident, and on many shapes the generic plan is
|
|
280
|
+
* estimated expensive enough that `auto` never promotes at all.
|
|
281
|
+
* 2. What would the promoted plan cost? The two `plan_cache_mode` settings
|
|
282
|
+
* make both plans reachable on demand.
|
|
283
|
+
* 3. Put the session back, so the paste does not leave `plan_cache_mode`
|
|
284
|
+
* pinned for everything the user does next.
|
|
285
|
+
*/
|
|
286
|
+
function buildDiagnosticSql(table, column, orderColumn, columnType) {
|
|
287
|
+
const t = (0, utils_js_1.quoteIdent)(table);
|
|
288
|
+
const c = (0, utils_js_1.quoteIdent)(column);
|
|
289
|
+
const o = (0, utils_js_1.quoteIdent)(orderColumn);
|
|
290
|
+
return [
|
|
291
|
+
'SET synchronize_seqscans = off;',
|
|
292
|
+
'SET max_parallel_workers_per_gather = 0;',
|
|
293
|
+
`PREPARE turbine_divergence(${columnType}, int) AS`,
|
|
294
|
+
` SELECT * FROM ${t} WHERE ${c} = $1 ORDER BY ${o} LIMIT $2;`,
|
|
295
|
+
'-- 1. does this shape get promoted at all? run it six times, then look:',
|
|
296
|
+
'EXECUTE turbine_divergence(<your value>, 20); -- x6',
|
|
297
|
+
"SELECT generic_plans, custom_plans FROM pg_prepared_statements WHERE name = 'turbine_divergence';",
|
|
298
|
+
'-- generic_plans still 0 means the planner is refusing the generic plan: no exposure.',
|
|
299
|
+
'-- 2. what the promoted plan would cost:',
|
|
300
|
+
'SET plan_cache_mode = force_custom_plan;',
|
|
301
|
+
'EXPLAIN (ANALYZE, BUFFERS) EXECUTE turbine_divergence(<your value>, 20);',
|
|
302
|
+
'SET plan_cache_mode = force_generic_plan;',
|
|
303
|
+
'EXPLAIN (ANALYZE, BUFFERS) EXECUTE turbine_divergence(<your value>, 20);',
|
|
304
|
+
'-- 3. put the session back:',
|
|
305
|
+
'RESET plan_cache_mode; RESET synchronize_seqscans; RESET max_parallel_workers_per_gather;',
|
|
306
|
+
'DEALLOCATE turbine_divergence;',
|
|
307
|
+
].join('\n');
|
|
308
|
+
}
|
|
309
|
+
/**
|
|
310
|
+
* Score every candidate column against the snapshot and return the ones whose
|
|
311
|
+
* distribution admits a damaging generic-plan flip.
|
|
312
|
+
*
|
|
313
|
+
* ONE rule, for the direction that could be calibrated: the generic estimate
|
|
314
|
+
* sits ABOVE the plan boundary (so a promoted plan keeps the ordered index
|
|
315
|
+
* scan) while the table's rarest values sit below it, and the pages that
|
|
316
|
+
* ordered scan must walk for such a value are worth a user's attention both in
|
|
317
|
+
* absolute terms and as a fraction of the table.
|
|
318
|
+
*
|
|
319
|
+
* `correlation` is reported but no longer gates anything. It used to route
|
|
320
|
+
* clustered columns to a second rule; that rule is gone (see the module header),
|
|
321
|
+
* and routing on it also made the advisor STRUCTURALLY unable to report a real
|
|
322
|
+
* sparse-direction flip on any column that happened to be clustered, which was
|
|
323
|
+
* measured at 1,165x on one fixture.
|
|
324
|
+
*/
|
|
325
|
+
function findPlanDivergence(schema, snapshot) {
|
|
326
|
+
const t = exports.PLAN_DIVERGENCE_THRESHOLDS;
|
|
327
|
+
const findings = [];
|
|
328
|
+
const notices = [];
|
|
329
|
+
let candidatesConsidered = 0;
|
|
330
|
+
for (const candidate of collectDivergenceCandidateColumns(schema)) {
|
|
331
|
+
const meta = schema.tables[candidate.table];
|
|
332
|
+
if (!meta)
|
|
333
|
+
continue;
|
|
334
|
+
const stats = snapshot.tables[candidate.table];
|
|
335
|
+
if (!stats)
|
|
336
|
+
continue;
|
|
337
|
+
const rows = stats.reltuples;
|
|
338
|
+
const pages = stats.relpages;
|
|
339
|
+
// reltuples 0/-1 means never analyzed, and relpages is only read from
|
|
340
|
+
// pg_class: either being absent is an unknown, never a zero.
|
|
341
|
+
if (rows <= 0 || pages === undefined || pages <= 0)
|
|
342
|
+
continue;
|
|
343
|
+
const { hasIdx, orderColumn } = indexShapeFor(snapshot, meta, candidate.column);
|
|
344
|
+
// No btree leading with the column: both plans are equally bad and the
|
|
345
|
+
// existing missing-index finding already owns this case.
|
|
346
|
+
if (!hasIdx)
|
|
347
|
+
continue;
|
|
348
|
+
// No other ordering index: no alternative plan for the generic estimate to
|
|
349
|
+
// run away with.
|
|
350
|
+
if (orderColumn === null)
|
|
351
|
+
continue;
|
|
352
|
+
// Counted HERE, before any scored gate. Everything above is a SHAPE
|
|
353
|
+
// question (is there an index to flip between at all?); everything below is
|
|
354
|
+
// a verdict on live statistics. A user must be able to tell "considered and
|
|
355
|
+
// clean" from "never looked", and an earlier revision incremented this after
|
|
356
|
+
// a silent table-size floor, so the two were indistinguishable.
|
|
357
|
+
candidatesConsidered++;
|
|
358
|
+
const dist = snapshot.columnStats?.[`${candidate.table}.${candidate.column}`];
|
|
359
|
+
if (!dist) {
|
|
360
|
+
notices.push({
|
|
361
|
+
table: candidate.table,
|
|
362
|
+
column: candidate.column,
|
|
363
|
+
reason: 'no pg_stats row for this column (never analyzed, or the statistics read degraded): run ANALYZE',
|
|
364
|
+
});
|
|
365
|
+
continue;
|
|
366
|
+
}
|
|
367
|
+
if (dist.nDistinct === 0) {
|
|
368
|
+
notices.push({
|
|
369
|
+
table: candidate.table,
|
|
370
|
+
column: candidate.column,
|
|
371
|
+
reason: 'n_distinct is 0 (column never analyzed): run ANALYZE, then re-check',
|
|
372
|
+
});
|
|
373
|
+
continue;
|
|
374
|
+
}
|
|
375
|
+
if (dist.mostCommonFreqs === null || dist.mostCommonFreqs.length === 0) {
|
|
376
|
+
notices.push({
|
|
377
|
+
table: candidate.table,
|
|
378
|
+
column: candidate.column,
|
|
379
|
+
reason: 'pg_stats has no most_common_freqs for this column: the value distribution cannot be scored',
|
|
380
|
+
});
|
|
381
|
+
continue;
|
|
382
|
+
}
|
|
383
|
+
const distinctValues = decodeDistinct(dist.nDistinct, rows);
|
|
384
|
+
if (distinctValues < 2)
|
|
385
|
+
continue;
|
|
386
|
+
const freqs = dist.mostCommonFreqs;
|
|
387
|
+
const nmcv = dist.mcvCount > 0 ? dist.mcvCount : freqs.length;
|
|
388
|
+
const freqSum = freqs.reduce((a, b) => a + b, 0);
|
|
389
|
+
const densestBucket = Math.max(...freqs) * rows;
|
|
390
|
+
// The residual (non-MCV) bucket when the MCV list does not cover every value;
|
|
391
|
+
// when it does (the normal case for a tenant column, whose distinct count is
|
|
392
|
+
// below the default statistics target), the rarest MCV IS the rarest value.
|
|
393
|
+
const rarestBucket = nmcv < distinctValues ? (rows * Math.max(0, 1 - freqSum)) / (distinctValues - nmcv) : Math.min(...freqs) * rows;
|
|
394
|
+
const genericEstimate = rows / distinctValues;
|
|
395
|
+
const crossover = crossoverRows(pages, t.assumedLimit);
|
|
396
|
+
const correlation = dist.correlation ?? 0;
|
|
397
|
+
// Pages the ordered index scan the generic plan keeps must walk before it
|
|
398
|
+
// accumulates `assumedLimit` matches of the rarest value: limit / matching
|
|
399
|
+
// of the table, and the whole table once the value has fewer rows than the
|
|
400
|
+
// limit. The bitmap plan the custom planner takes instead reads about
|
|
401
|
+
// min(matching, pages).
|
|
402
|
+
const walkPages = rarestBucket > 0 ? Math.min(pages, (t.assumedLimit * pages) / rarestBucket) : pages;
|
|
403
|
+
const walkFraction = walkPages / pages;
|
|
404
|
+
const approxAmplification = walkPages / Math.max(1, Math.min(rarestBucket, pages));
|
|
405
|
+
const flips = genericEstimate >= crossover && rarestBucket < crossover;
|
|
406
|
+
if (!flips)
|
|
407
|
+
continue;
|
|
408
|
+
if (walkPages < t.minWalkPages || walkFraction < t.minWalkFraction)
|
|
409
|
+
continue;
|
|
410
|
+
let valuesBelowCrossover = freqs.filter((f) => f * rows < crossover).length;
|
|
411
|
+
if (nmcv < distinctValues && rarestBucket < crossover) {
|
|
412
|
+
valuesBelowCrossover += Math.round(distinctValues - nmcv);
|
|
413
|
+
}
|
|
414
|
+
const columnType = meta.pgTypes[candidate.column] ?? 'text';
|
|
415
|
+
findings.push({
|
|
416
|
+
table: candidate.table,
|
|
417
|
+
column: candidate.column,
|
|
418
|
+
rows,
|
|
419
|
+
pages,
|
|
420
|
+
distinctValues,
|
|
421
|
+
genericEstimate,
|
|
422
|
+
rarestBucket,
|
|
423
|
+
densestBucket,
|
|
424
|
+
correlation,
|
|
425
|
+
crossoverRows: crossover,
|
|
426
|
+
crossoverRowsWide: crossoverRows(pages, t.wideLimit),
|
|
427
|
+
assumedLimit: t.assumedLimit,
|
|
428
|
+
valuesBelowCrossover,
|
|
429
|
+
walkPages,
|
|
430
|
+
walkFraction,
|
|
431
|
+
approxAmplification,
|
|
432
|
+
orderColumn,
|
|
433
|
+
columnField: meta.reverseColumnMap[candidate.column] ?? candidate.column,
|
|
434
|
+
orderColumnField: meta.reverseColumnMap[orderColumn] ?? orderColumn,
|
|
435
|
+
lastAnalyze: stats.lastAnalyze ?? null,
|
|
436
|
+
diagnosticSql: buildDiagnosticSql(candidate.table, candidate.column, orderColumn, columnType),
|
|
437
|
+
thresholds: t,
|
|
438
|
+
});
|
|
439
|
+
}
|
|
440
|
+
findings.sort((a, b) => b.approxAmplification - a.approxAmplification ||
|
|
441
|
+
a.table.localeCompare(b.table) ||
|
|
442
|
+
a.column.localeCompare(b.column));
|
|
443
|
+
return { findings, notices, candidatesConsidered };
|
|
444
|
+
}
|
package/dist/cjs/powql.d.ts
CHANGED
|
@@ -125,6 +125,17 @@ export declare class PowqlInterface<T extends object = Record<string, unknown>>
|
|
|
125
125
|
* callers short-circuit it, because PowDB's projection fast path returned ONE
|
|
126
126
|
* row for `limit 0` below 0.20.
|
|
127
127
|
*/
|
|
128
|
+
/**
|
|
129
|
+
* Refuse the per-query `forceCustomPlan` read option.
|
|
130
|
+
*
|
|
131
|
+
* `FindManyArgs.forceCustomPlan` documents an `UnsupportedFeatureError` on any
|
|
132
|
+
* engine with no PostgreSQL plan cache, and PowDB is one: it is not a
|
|
133
|
+
* `Dialect` at all, so it never reaches the SQL builder's own refusal at
|
|
134
|
+
* `preparedNameFor`. Without this the flag would be silently ignored here,
|
|
135
|
+
* which is precisely the "reports a guarantee that was never made" failure the
|
|
136
|
+
* option's refusal exists to prevent.
|
|
137
|
+
*/
|
|
138
|
+
private assertNoForceCustomPlan;
|
|
128
139
|
private assertPagination;
|
|
129
140
|
/** A predicate that is always false, the empty-`in` / contradiction sentinel. */
|
|
130
141
|
private alwaysFalse;
|
package/dist/cjs/powql.js
CHANGED
|
@@ -328,6 +328,22 @@ class PowqlInterface {
|
|
|
328
328
|
* callers short-circuit it, because PowDB's projection fast path returned ONE
|
|
329
329
|
* row for `limit 0` below 0.20.
|
|
330
330
|
*/
|
|
331
|
+
/**
|
|
332
|
+
* Refuse the per-query `forceCustomPlan` read option.
|
|
333
|
+
*
|
|
334
|
+
* `FindManyArgs.forceCustomPlan` documents an `UnsupportedFeatureError` on any
|
|
335
|
+
* engine with no PostgreSQL plan cache, and PowDB is one: it is not a
|
|
336
|
+
* `Dialect` at all, so it never reaches the SQL builder's own refusal at
|
|
337
|
+
* `preparedNameFor`. Without this the flag would be silently ignored here,
|
|
338
|
+
* which is precisely the "reports a guarantee that was never made" failure the
|
|
339
|
+
* option's refusal exists to prevent.
|
|
340
|
+
*/
|
|
341
|
+
assertNoForceCustomPlan(args) {
|
|
342
|
+
if (args?.forceCustomPlan !== true)
|
|
343
|
+
return;
|
|
344
|
+
throw new errors_js_1.UnsupportedFeatureError('The forceCustomPlan query option', 'powdb', 'Forcing a per-query custom plan means keeping the statement out of the PostgreSQL plan cache, and PowDB ' +
|
|
345
|
+
'has no such cache to keep it out of. Remove the option, or set it only on PostgreSQL queries.');
|
|
346
|
+
}
|
|
331
347
|
assertPagination(limit, offset, context) {
|
|
332
348
|
for (const [name, value] of [
|
|
333
349
|
['limit', limit],
|
|
@@ -1157,6 +1173,7 @@ class PowqlInterface {
|
|
|
1157
1173
|
// Reads
|
|
1158
1174
|
// -------------------------------------------------------------------------
|
|
1159
1175
|
async findMany(args = {}) {
|
|
1176
|
+
this.assertNoForceCustomPlan(args);
|
|
1160
1177
|
return this.withMiddleware('findMany', args, async () => {
|
|
1161
1178
|
// `limit: 0` means "no rows" (SQL `LIMIT 0`), and answering it client-side
|
|
1162
1179
|
// is correct on every engine version: PowDB's projection fast path returned
|
|
@@ -1296,6 +1313,7 @@ class PowqlInterface {
|
|
|
1296
1313
|
.filter((line) => line.length > 0);
|
|
1297
1314
|
}
|
|
1298
1315
|
async findUnique(args) {
|
|
1316
|
+
this.assertNoForceCustomPlan(args);
|
|
1299
1317
|
// Prisma compound-unique selector → column conjunction (engine parity with
|
|
1300
1318
|
// the SQL findUnique family; pure metadata, so this is a one-line adoption).
|
|
1301
1319
|
if (args.where) {
|
|
@@ -1318,6 +1336,7 @@ class PowqlInterface {
|
|
|
1318
1336
|
});
|
|
1319
1337
|
}
|
|
1320
1338
|
async findFirst(args = {}) {
|
|
1339
|
+
this.assertNoForceCustomPlan(args);
|
|
1321
1340
|
return this.withMiddleware('findFirst', args, async () => {
|
|
1322
1341
|
const { rows, native, nestedPlans, linkPlans, residualWith } = await this.runFind({ ...args, limit: 1 }, 'findFirst');
|
|
1323
1342
|
if (!rows.length)
|
|
@@ -2524,6 +2543,7 @@ class PowqlInterface {
|
|
|
2524
2543
|
// Aggregates
|
|
2525
2544
|
// -------------------------------------------------------------------------
|
|
2526
2545
|
async count(args = {}) {
|
|
2546
|
+
this.assertNoForceCustomPlan(args);
|
|
2527
2547
|
return this.withMiddleware('count', (args ?? {}), async () => {
|
|
2528
2548
|
const params = [];
|
|
2529
2549
|
const resolvedWhere = await this.resolveRelationFilters(args.where, args.timeout);
|
|
@@ -2562,6 +2582,7 @@ class PowqlInterface {
|
|
|
2562
2582
|
'`_count` of a NOT NULL column are correct on every version and are never refused.');
|
|
2563
2583
|
}
|
|
2564
2584
|
async aggregate(args) {
|
|
2585
|
+
this.assertNoForceCustomPlan(args);
|
|
2565
2586
|
return this.withMiddleware('aggregate', args, async () => {
|
|
2566
2587
|
// One scalar query per aggregate, PowDB's bare-projection aggregate is broken.
|
|
2567
2588
|
const result = {};
|
|
@@ -2607,6 +2628,7 @@ class PowqlInterface {
|
|
|
2607
2628
|
});
|
|
2608
2629
|
}
|
|
2609
2630
|
async groupBy(args) {
|
|
2631
|
+
this.assertNoForceCustomPlan(args);
|
|
2610
2632
|
return this.withMiddleware('groupBy', args, async () => {
|
|
2611
2633
|
// DISTINCT ON has no PowQL equivalent (no DISTINCT ON row source).
|
|
2612
2634
|
if (args.distinctOn) {
|
|
@@ -712,6 +712,51 @@ export declare class QueryInterface<T extends object, R extends object = {}> {
|
|
|
712
712
|
*/
|
|
713
713
|
resetUnlimitedWarnings(): void;
|
|
714
714
|
private emitQueryEvent;
|
|
715
|
+
/**
|
|
716
|
+
* Resolve the prepared-statement name a read should execute under, honouring
|
|
717
|
+
* the per-query {@link FindManyArgs.forceCustomPlan} opt-in.
|
|
718
|
+
*
|
|
719
|
+
* `forceCustomPlan: true` returns `undefined`, which sends the statement
|
|
720
|
+
* UNNAMED. The mechanism is NOT "PostgreSQL treats an unnamed statement as a
|
|
721
|
+
* one-shot plan that never enters the plan cache": the backend builds and
|
|
722
|
+
* saves a `CachedPlanSource` for the unnamed statement too. It works because
|
|
723
|
+
* node-postgres only skips Parse for a statement it has already parsed BY
|
|
724
|
+
* NAME (`Query.hasBeenParsed` is `this.name && connection.parsedStatements[this.name]`),
|
|
725
|
+
* so an unnamed statement is re-Parsed on every execution, each Parse
|
|
726
|
+
* replaces the unnamed cached plan source with a fresh one whose custom-plan
|
|
727
|
+
* counter is zero, and the five-execution threshold that precedes promotion
|
|
728
|
+
* is never reached. Every execution is therefore planned with the real
|
|
729
|
+
* parameter values.
|
|
730
|
+
*
|
|
731
|
+
* No GUC is set, no `SET LOCAL` is emitted, no transaction is opened, and no
|
|
732
|
+
* extra round trip is added, which is exactly why the opt-in can be per query
|
|
733
|
+
* while the client-level `planCacheMode` (a connection parameter) cannot be.
|
|
734
|
+
*
|
|
735
|
+
* The refusal is deliberately here, at the one seam every read execution
|
|
736
|
+
* passes through, rather than in each build method: the flag changes NOTHING
|
|
737
|
+
* about the SQL text, so a build-time check would have had to be repeated in
|
|
738
|
+
* every builder and could still be bypassed by a hand-executed
|
|
739
|
+
* `DeferredQuery`.
|
|
740
|
+
*
|
|
741
|
+
* Engines whose dialect does not report {@link Dialect.supportsPlanCacheMode}
|
|
742
|
+
* throw {@link UnsupportedFeatureError} (E017): the flag names a PostgreSQL
|
|
743
|
+
* plan-cache guarantee, and an engine with no such cache cannot make it.
|
|
744
|
+
* The same flag left unset (or `false`) is accepted everywhere.
|
|
745
|
+
*
|
|
746
|
+
* THE ONE COMBINATION THAT IS REFUSED RATHER THAN HONOURED. A client-level
|
|
747
|
+
* `planCacheMode: 'force_generic_plan'` DEFEATS this option, and that was
|
|
748
|
+
* MEASURED rather than reasoned about: on PostgreSQL 16.14, five executions
|
|
749
|
+
* of one unnamed statement read 19,107 buffers with that setting in force and
|
|
750
|
+
* 55 buffers with the same connection set back to `auto`, against 19,107 for
|
|
751
|
+
* the named statement. So the setting governs the unnamed statement too, and
|
|
752
|
+
* withholding the name buys nothing against it. Accepting the flag there
|
|
753
|
+
* would report a guarantee the very next execution breaks, so the
|
|
754
|
+
* contradiction throws {@link ValidationError} (E003) naming both settings.
|
|
755
|
+
* Turbine can only see the setting IT applied: a `plan_cache_mode` installed
|
|
756
|
+
* by the caller's own `SET`, by `ALTER ROLE`, or by a pooler is invisible
|
|
757
|
+
* here and is not refused.
|
|
758
|
+
*/
|
|
759
|
+
private preparedNameFor;
|
|
715
760
|
/**
|
|
716
761
|
* Execute a pool.query with an optional timeout.
|
|
717
762
|
* If timeout is set, races the query against a timer and rejects on expiry.
|