turbine-orm 0.58.0 → 0.59.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -1
- package/dist/cjs/cli/index.js +2 -1
- package/dist/cjs/plan-flip-probe.d.ts +56 -18
- package/dist/cjs/plan-flip-probe.js +74 -30
- package/dist/cli/index.js +2 -1
- package/dist/plan-flip-probe.d.ts +56 -18
- package/dist/plan-flip-probe.js +74 -30
- package/package.json +3 -2
package/README.md
CHANGED
|
@@ -454,7 +454,7 @@ const db = turbine({
|
|
|
454
454
|
|
|
455
455
|
Where a pg-style alias exists (`max`, `idleTimeoutMillis`, `connectionTimeoutMillis`), the explicit Turbine field wins when both are set.
|
|
456
456
|
|
|
457
|
-
> **`planCacheMode` (Postgres only, opt-in).** PostgreSQL may promote a **named** prepared statement to a generic plan from its sixth execution onward, and a generic plan is costed blind to the bound values. On a predicate whose selectivity swings per value (a `tenant_id` equality on a shared table, where one value matches a handful of rows and another matches most of them), the statement can be locked onto a plan chosen for the average value, and it never reverts. `planCacheMode: 'auto' | 'force_custom_plan' | 'force_generic_plan'` pins the backend's choice; `'force_custom_plan'` re-plans every execution and removes the cliff. It is applied as a connection parameter (`options=-c plan_cache_mode=...`) when Turbine opens a connection, so it is in force for that connection's first statement and for every checkout, `$transaction`, stream and pipeline on it, and it cannot race your first query. Leave it unset (the default) and Turbine sends nothing at all. Reach for it when you have measured a statement getting slower after its fifth execution. **Correction to the 0.54 text, which said `findMany` / `findFirst` bind `LIMIT $n` and are "much less exposed":** that was false. PostgreSQL does not deny the planner a limit fraction for a bound limit, it substitutes a default of 10% of the child node's own row estimate (clamped at one row), and an unknown `OFFSET` triggers the same substitution even when the limit is a constant, which a paginated Turbine read always has. Two things also need saying about the sentence that opens this note. The sixth execution is a ceiling, not a trigger: `auto` promotes only when the generic plan's **estimated** cost is not worse than the average custom cost, so many statements are never promoted at all, and `pg_prepared_statements.generic_plans` is how you tell. And the shape that gets promoted unprompted is the one with **no limit**, not the limited one: measured on a skewed join predicate, an unlimited `count()`-shaped statement promoted under the default `auto` and ran a nested loop at 430x the buffers of the custom plan, while the same predicate under `LIMIT $n` was never promoted across eight executions (its substituted row count made the generic plan look more expensive). A limited `findMany` gives the planner two unknowns instead of one, which is not the same as more damage. `implicitPkOrdering` is **off by default in core**, so a default `findMany` emits no `ORDER BY`; switching it on adds an ordering a generic plan can walk the whole table in. Measure with `plan_cache_mode = force_generic_plan` against `force_custom_plan` rather than reasoning about which shapes ought to be safe; the fixtures and numbers are on the [relations page](https://turbineorm.dev/relations) and in the 0.55.0 changelog. **Two 0.56 corrections to the paragraph above.** First, "neither an ORDER BY nor a limit is required" is true, but it read as if ordering did not matter:
|
|
457
|
+
> **`planCacheMode` (Postgres only, opt-in).** PostgreSQL may promote a **named** prepared statement to a generic plan from its sixth execution onward, and a generic plan is costed blind to the bound values. On a predicate whose selectivity swings per value (a `tenant_id` equality on a shared table, where one value matches a handful of rows and another matches most of them), the statement can be locked onto a plan chosen for the average value, and it never reverts. `planCacheMode: 'auto' | 'force_custom_plan' | 'force_generic_plan'` pins the backend's choice; `'force_custom_plan'` re-plans every execution and removes the cliff. It is applied as a connection parameter (`options=-c plan_cache_mode=...`) when Turbine opens a connection, so it is in force for that connection's first statement and for every checkout, `$transaction`, stream and pipeline on it, and it cannot race your first query. Leave it unset (the default) and Turbine sends nothing at all. Reach for it when you have measured a statement getting slower after its fifth execution. **Correction to the 0.54 text, which said `findMany` / `findFirst` bind `LIMIT $n` and are "much less exposed":** that was false. PostgreSQL does not deny the planner a limit fraction for a bound limit, it substitutes a default of 10% of the child node's own row estimate (clamped at one row), and an unknown `OFFSET` triggers the same substitution even when the limit is a constant, which a paginated Turbine read always has. Two things also need saying about the sentence that opens this note. The sixth execution is a ceiling, not a trigger: `auto` promotes only when the generic plan's **estimated** cost is not worse than the average custom cost, so many statements are never promoted at all, and `pg_prepared_statements.generic_plans` is how you tell. And the shape that gets promoted unprompted is the one with **no limit**, not the limited one: measured on a skewed join predicate, an unlimited `count()`-shaped statement promoted under the default `auto` and ran a nested loop at 430x the buffers of the custom plan, while the same predicate under `LIMIT $n` was never promoted across eight executions (its substituted row count made the generic plan look more expensive). A limited `findMany` gives the planner two unknowns instead of one, which is not the same as more damage. `implicitPkOrdering` is **off by default in core**, so a default `findMany` emits no `ORDER BY`; switching it on adds an ordering a generic plan can walk the whole table in. Measure with `plan_cache_mode = force_generic_plan` against `force_custom_plan` rather than reasoning about which shapes ought to be safe; the fixtures and numbers are on the [relations page](https://turbineorm.dev/relations) and in the 0.55.0 changelog. **Two 0.56 corrections to the paragraph above.** First, "neither an ORDER BY nor a limit is required" is true, but it read as if ordering did not matter: in a table-by-table sweep of a multi-tenant schema, every divergent shape measured had an `ORDER BY` and every shape without one measured 1.00x, so it is not necessary in general and is still the strongest single predictor in practice. Second, a custom plan is not automatically the better one: on a reproducible fixture where one dense value's rows are packed at the end of the heap, `LIMIT 20` with no ordering reads 4,262 buffers custom against 71 generic (the default `auto` never promotes there, so it produces the 4,262-buffer plan too). Since 0.56 the per-query read arg **`forceCustomPlan: true`** covers the case a connection-wide setting cannot express, custom on one query and `auto` everywhere else, and `turbine doctor` detects the distribution that admits the flip. **0.57 correction:** that read arg reached the core client only. Through `turbine-orm/prisma-compat` it was accepted and silently dropped until 0.57.0, so a compat integration that followed this advice got a no-op; confirm at the wire with `pg_prepared_statements` rather than assuming. 0.57 also adds a third divergence mechanism to `doctor`: an **unindexed** filter column, where the good plan is a sequential scan the generic plan will not choose (measured 250 buffers against 20,074 on a 20,000-row / 247-page fixture). Three scope limits: it does nothing on an **external pool** (Turbine never opens those connections, so set the GUC in the driver's own setup; Turbine-owned string `replicas` on that same client DO get it); a Postgres wire-compatible engine without the setting (CockroachDB, YugabyteDB, pre-12 PostgreSQL) refuses the connection parameter itself; and a **connection pooler** may filter startup parameters (PgBouncer's `ignore_startup_parameters`), where `ALTER ROLE ... SET plan_cache_mode = ...` is the way in. Any value outside the three throws `ValidationError` at construction, and a non-Postgres engine throws `UnsupportedFeatureError` (`TURBINE_E017`).
|
|
458
458
|
|
|
459
459
|
> **`preparedStatements` and connection poolers.** With prepared statements on, Turbine submits queries as `{ name, text, values }` so Postgres caches the parse and plan **per backend connection**. That is a real win against a database you connect to directly, and a hazard behind a transaction-pooling proxy (PgBouncer in `transaction` mode, Supabase's pooler port, some serverless poolers): the named statement is prepared on one backend and your next query may land on another, which fails with `prepared statement "..." does not exist`. Turbine defaults it to `true` only for pools it creates itself and `false` for external pools passed via `pool` / `turbineHttp()`, because serverless drivers are the common case there. If you are pointing a Turbine-owned pool at a transaction pooler, set `preparedStatements: false`. The environment variable `TURBINE_DISABLE_PREPARED=1` turns it off globally without a code change.
|
|
460
460
|
|
package/dist/cjs/cli/index.js
CHANGED
|
@@ -2389,7 +2389,8 @@ async function cmdDoctor(args, config) {
|
|
|
2389
2389
|
: { findings: [], notices: [], candidatesConsidered: 0, consideredIndexed: 0, consideredUnindexed: 0 };
|
|
2390
2390
|
// Statistics can say how bad a flip WOULD be; only the planner can say whether
|
|
2391
2391
|
// it is reachable. The `unindexed-filter` branch shipped in 0.57 without that
|
|
2392
|
-
// question answered and
|
|
2392
|
+
// question answered, and a measured sample of 13 findings held up only 6 times,
|
|
2393
|
+
// so every one
|
|
2393
2394
|
// of its findings is now put to a plan-only EXPLAIN. Nothing is executed, and a
|
|
2394
2395
|
// probe that fails keeps its finding rather than dropping it.
|
|
2395
2396
|
const flipProbe = divergenceOn && scored.findings.some(plan_flip_probe_js_1.needsFlipProbe)
|
|
@@ -6,13 +6,14 @@
|
|
|
6
6
|
* `plan-divergence.ts` scores a column from statistics alone and answers "IF the
|
|
7
7
|
* cached plan flips, how bad is it". Its `unindexed-filter` branch (0.57) shipped
|
|
8
8
|
* without an answer to the prior question, "CAN it flip at all", and that turned
|
|
9
|
-
* out to be the majority case:
|
|
10
|
-
* findings
|
|
9
|
+
* out to be the majority case: in validation against a large schema the branch
|
|
10
|
+
* emitted 39 findings, and a measured sample of 13 of them held up only 6 times.
|
|
11
11
|
*
|
|
12
|
-
* Every false positive had one signature: **the generic plan
|
|
13
|
-
*
|
|
12
|
+
* Every false positive had one signature: **the generic plan was not the ordered
|
|
13
|
+
* index walk the finding claims.** There was no flip to be had, so the
|
|
14
14
|
* amplification the finding printed described a plan the planner would never
|
|
15
|
-
* pick.
|
|
15
|
+
* pick. See {@link verdictFromPlanJson} for the two ways a plan fails to be that
|
|
16
|
+
* walk; 0.58.0 shipped only one of them and 0.59.0 added the other.
|
|
16
17
|
*
|
|
17
18
|
* ## Why this is a probe and not another rule
|
|
18
19
|
*
|
|
@@ -54,11 +55,10 @@
|
|
|
54
55
|
* ```
|
|
55
56
|
*
|
|
56
57
|
* The custom plan is not needed. The finding's whole claim is that a promoted
|
|
57
|
-
* generic plan
|
|
58
|
-
*
|
|
59
|
-
*
|
|
60
|
-
*
|
|
61
|
-
* carry.
|
|
58
|
+
* generic plan performs an ordered index walk, so a generic plan that is NOT that
|
|
59
|
+
* walk refutes it no matter what the custom plan does. Asking one question
|
|
60
|
+
* instead of two halves the work and removes the need for a representative rare
|
|
61
|
+
* value, which statistics do not carry.
|
|
62
62
|
*
|
|
63
63
|
* `NULL` is a safe argument precisely because the plan is generic: a generic plan
|
|
64
64
|
* is built without looking at the value, which is the property the whole check is
|
|
@@ -80,11 +80,11 @@ import type { PlanDivergenceFinding, PlanDivergenceReport } from './plan-diverge
|
|
|
80
80
|
/**
|
|
81
81
|
* The planner's answer for one finding.
|
|
82
82
|
*
|
|
83
|
-
* - `'flip-reachable'`, the generic plan
|
|
83
|
+
* - `'flip-reachable'`, the generic plan IS an ordered index walk on the target
|
|
84
84
|
* table, so the divergence the finding describes is one the planner can
|
|
85
85
|
* actually choose.
|
|
86
|
-
* - `'no-flip'`, the generic plan
|
|
87
|
-
* access
|
|
86
|
+
* - `'no-flip'`, the generic plan is not that walk (a `Sort` bounds it, or the
|
|
87
|
+
* access is a plain seq scan). Nothing to diverge to.
|
|
88
88
|
* - `'unknown'`, the probe did not produce an answer. The finding is kept.
|
|
89
89
|
*/
|
|
90
90
|
export type FlipVerdict = 'flip-reachable' | 'no-flip' | 'unknown';
|
|
@@ -133,11 +133,49 @@ export declare function buildFlipProbeSql(finding: PlanDivergenceFinding, name:
|
|
|
133
133
|
* Exported for unit tests: the plan shapes this has to classify are exactly the
|
|
134
134
|
* ones that are tedious to produce live.
|
|
135
135
|
*
|
|
136
|
-
* The
|
|
137
|
-
*
|
|
138
|
-
*
|
|
139
|
-
*
|
|
140
|
-
*
|
|
136
|
+
* ## The question, stated exactly
|
|
137
|
+
*
|
|
138
|
+
* The finding claims the generic plan performs an ORDERED INDEX WALK along the
|
|
139
|
+
* `ORDER BY` column and fetches nearly every tuple before the `LIMIT` fills. So
|
|
140
|
+
* the refutation is not "the plan is a seq scan", it is **"the plan is not that
|
|
141
|
+
* ordered walk"**, and there are two ways for it not to be:
|
|
142
|
+
*
|
|
143
|
+
* 1. A `Sort` lies above the target table's scan. A sort materializes the whole
|
|
144
|
+
* matched set and orders it, so the cost is bounded by how many rows match,
|
|
145
|
+
* not by how far into the heap the ordered walk has to travel. Whatever feeds
|
|
146
|
+
* it (seq scan, bitmap heap scan) the catastrophic shape is absent.
|
|
147
|
+
* 2. The target's own scan node is a `Seq Scan`. Kept as an independent ground
|
|
148
|
+
* rather than folded into the first, so a hypothetical ordered seq scan with
|
|
149
|
+
* no sort still refutes.
|
|
150
|
+
*
|
|
151
|
+
* 0.58.0 shipped only the second ground and therefore missed every LOW-estimate
|
|
152
|
+
* column that has ANY usable index, because those plan as `Limit > Sort > Bitmap
|
|
153
|
+
* Heap Scan`. The case that surfaced it was a column carrying
|
|
154
|
+
* `btree (col) WHERE col IS NOT NULL`: an equality predicate implies not-null, so
|
|
155
|
+
* that partial index is fully usable and the plan never reaches a seq scan.
|
|
156
|
+
* Reproduced, and the fixture is the pair below at the same estimate:
|
|
157
|
+
*
|
|
158
|
+
* ```txt
|
|
159
|
+
* partial index, est 1.9 Limit > Sort > Bitmap Heap Scan <- 0.58 kept this
|
|
160
|
+
* no index, est 1.9 Limit > Sort > Seq Scan <- 0.58 refuted this
|
|
161
|
+
* either, est 500 Limit > Index Scan (no Sort) <- both keep, correctly
|
|
162
|
+
* ```
|
|
163
|
+
*
|
|
164
|
+
* ## Why this is not "exclude partial-index columns"
|
|
165
|
+
*
|
|
166
|
+
* Because a partial index whose predicate is NOT implied by the equality does not
|
|
167
|
+
* serve the query at all, and such a column produces a genuine finding: one was
|
|
168
|
+
* measured at 19,961x. The property that matters is whether the planner COULD
|
|
169
|
+
* use it, which is a proof obligation over predicates, and the plan already
|
|
170
|
+
* carries the answer. Reading the plan is cheaper and cannot drift from the
|
|
171
|
+
* planner's own implication rules.
|
|
172
|
+
*
|
|
173
|
+
* ## The safe direction is KEEP
|
|
174
|
+
*
|
|
175
|
+
* Over-refuting deletes real findings, which is invisible in the report;
|
|
176
|
+
* over-keeping only costs noise. So `Incremental Sort` does NOT refute: it means
|
|
177
|
+
* the index supplies a PREFIX of the ordering and the walk is still partly
|
|
178
|
+
* ordered, which is closer to the catastrophic shape than to the bounded one.
|
|
141
179
|
*/
|
|
142
180
|
export declare function verdictFromPlanJson(payload: unknown, table: string): FlipVerdict;
|
|
143
181
|
export interface ProbePlanFlipsOptions {
|
|
@@ -7,13 +7,14 @@
|
|
|
7
7
|
* `plan-divergence.ts` scores a column from statistics alone and answers "IF the
|
|
8
8
|
* cached plan flips, how bad is it". Its `unindexed-filter` branch (0.57) shipped
|
|
9
9
|
* without an answer to the prior question, "CAN it flip at all", and that turned
|
|
10
|
-
* out to be the majority case:
|
|
11
|
-
* findings
|
|
10
|
+
* out to be the majority case: in validation against a large schema the branch
|
|
11
|
+
* emitted 39 findings, and a measured sample of 13 of them held up only 6 times.
|
|
12
12
|
*
|
|
13
|
-
* Every false positive had one signature: **the generic plan
|
|
14
|
-
*
|
|
13
|
+
* Every false positive had one signature: **the generic plan was not the ordered
|
|
14
|
+
* index walk the finding claims.** There was no flip to be had, so the
|
|
15
15
|
* amplification the finding printed described a plan the planner would never
|
|
16
|
-
* pick.
|
|
16
|
+
* pick. See {@link verdictFromPlanJson} for the two ways a plan fails to be that
|
|
17
|
+
* walk; 0.58.0 shipped only one of them and 0.59.0 added the other.
|
|
17
18
|
*
|
|
18
19
|
* ## Why this is a probe and not another rule
|
|
19
20
|
*
|
|
@@ -55,11 +56,10 @@
|
|
|
55
56
|
* ```
|
|
56
57
|
*
|
|
57
58
|
* The custom plan is not needed. The finding's whole claim is that a promoted
|
|
58
|
-
* generic plan
|
|
59
|
-
*
|
|
60
|
-
*
|
|
61
|
-
*
|
|
62
|
-
* carry.
|
|
59
|
+
* generic plan performs an ordered index walk, so a generic plan that is NOT that
|
|
60
|
+
* walk refutes it no matter what the custom plan does. Asking one question
|
|
61
|
+
* instead of two halves the work and removes the need for a representative rare
|
|
62
|
+
* value, which statistics do not carry.
|
|
63
63
|
*
|
|
64
64
|
* `NULL` is a safe argument precisely because the plan is generic: a generic plan
|
|
65
65
|
* is built without looking at the value, which is the property the whole check is
|
|
@@ -166,39 +166,83 @@ function buildFlipProbeSql(finding, name, searchSchema) {
|
|
|
166
166
|
deallocate: `DEALLOCATE ${name}`,
|
|
167
167
|
};
|
|
168
168
|
}
|
|
169
|
-
function* walkPlan(node) {
|
|
170
|
-
yield node;
|
|
171
|
-
for (const child of node.Plans ?? [])
|
|
172
|
-
yield* walkPlan(child);
|
|
173
|
-
}
|
|
174
169
|
/**
|
|
175
170
|
* Read a verdict out of one `EXPLAIN (FORMAT JSON)` payload.
|
|
176
171
|
*
|
|
177
172
|
* Exported for unit tests: the plan shapes this has to classify are exactly the
|
|
178
173
|
* ones that are tedious to produce live.
|
|
179
174
|
*
|
|
180
|
-
* The
|
|
181
|
-
*
|
|
182
|
-
*
|
|
183
|
-
*
|
|
184
|
-
*
|
|
175
|
+
* ## The question, stated exactly
|
|
176
|
+
*
|
|
177
|
+
* The finding claims the generic plan performs an ORDERED INDEX WALK along the
|
|
178
|
+
* `ORDER BY` column and fetches nearly every tuple before the `LIMIT` fills. So
|
|
179
|
+
* the refutation is not "the plan is a seq scan", it is **"the plan is not that
|
|
180
|
+
* ordered walk"**, and there are two ways for it not to be:
|
|
181
|
+
*
|
|
182
|
+
* 1. A `Sort` lies above the target table's scan. A sort materializes the whole
|
|
183
|
+
* matched set and orders it, so the cost is bounded by how many rows match,
|
|
184
|
+
* not by how far into the heap the ordered walk has to travel. Whatever feeds
|
|
185
|
+
* it (seq scan, bitmap heap scan) the catastrophic shape is absent.
|
|
186
|
+
* 2. The target's own scan node is a `Seq Scan`. Kept as an independent ground
|
|
187
|
+
* rather than folded into the first, so a hypothetical ordered seq scan with
|
|
188
|
+
* no sort still refutes.
|
|
189
|
+
*
|
|
190
|
+
* 0.58.0 shipped only the second ground and therefore missed every LOW-estimate
|
|
191
|
+
* column that has ANY usable index, because those plan as `Limit > Sort > Bitmap
|
|
192
|
+
* Heap Scan`. The case that surfaced it was a column carrying
|
|
193
|
+
* `btree (col) WHERE col IS NOT NULL`: an equality predicate implies not-null, so
|
|
194
|
+
* that partial index is fully usable and the plan never reaches a seq scan.
|
|
195
|
+
* Reproduced, and the fixture is the pair below at the same estimate:
|
|
196
|
+
*
|
|
197
|
+
* ```txt
|
|
198
|
+
* partial index, est 1.9 Limit > Sort > Bitmap Heap Scan <- 0.58 kept this
|
|
199
|
+
* no index, est 1.9 Limit > Sort > Seq Scan <- 0.58 refuted this
|
|
200
|
+
* either, est 500 Limit > Index Scan (no Sort) <- both keep, correctly
|
|
201
|
+
* ```
|
|
202
|
+
*
|
|
203
|
+
* ## Why this is not "exclude partial-index columns"
|
|
204
|
+
*
|
|
205
|
+
* Because a partial index whose predicate is NOT implied by the equality does not
|
|
206
|
+
* serve the query at all, and such a column produces a genuine finding: one was
|
|
207
|
+
* measured at 19,961x. The property that matters is whether the planner COULD
|
|
208
|
+
* use it, which is a proof obligation over predicates, and the plan already
|
|
209
|
+
* carries the answer. Reading the plan is cheaper and cannot drift from the
|
|
210
|
+
* planner's own implication rules.
|
|
211
|
+
*
|
|
212
|
+
* ## The safe direction is KEEP
|
|
213
|
+
*
|
|
214
|
+
* Over-refuting deletes real findings, which is invisible in the report;
|
|
215
|
+
* over-keeping only costs noise. So `Incremental Sort` does NOT refute: it means
|
|
216
|
+
* the index supplies a PREFIX of the ordering and the walk is still partly
|
|
217
|
+
* ordered, which is closer to the catastrophic shape than to the bounded one.
|
|
185
218
|
*/
|
|
186
219
|
function verdictFromPlanJson(payload, table) {
|
|
187
220
|
const root = Array.isArray(payload) ? payload[0] : undefined;
|
|
188
221
|
const plan = root?.Plan;
|
|
189
222
|
if (!plan)
|
|
190
223
|
return 'unknown';
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
224
|
+
// Walk root-downward, tracking whether a full Sort sits ABOVE the target's
|
|
225
|
+
// scan. Depth matters: a Sort somewhere else in a larger plan says nothing
|
|
226
|
+
// about how this table is reached.
|
|
227
|
+
const search = (node, sortedAbove) => {
|
|
194
228
|
const type = node['Node Type'];
|
|
195
|
-
if (
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
229
|
+
if (node['Relation Name'] === table && type !== undefined) {
|
|
230
|
+
if (sortedAbove)
|
|
231
|
+
return 'no-flip';
|
|
232
|
+
return type === 'Seq Scan' ? 'no-flip' : 'flip-reachable';
|
|
233
|
+
}
|
|
234
|
+
// 'Incremental Sort' is deliberately excluded, see the header.
|
|
235
|
+
const nowSorted = sortedAbove || type === 'Sort';
|
|
236
|
+
for (const child of node.Plans ?? []) {
|
|
237
|
+
const found = search(child, nowSorted);
|
|
238
|
+
if (found !== null)
|
|
239
|
+
return found;
|
|
240
|
+
}
|
|
241
|
+
return null;
|
|
242
|
+
};
|
|
243
|
+
// The target table not appearing at all should not happen for a statement that
|
|
244
|
+
// selects from it. Treated as unknown rather than as a refutation.
|
|
245
|
+
return search(plan, false) ?? 'unknown';
|
|
202
246
|
}
|
|
203
247
|
/**
|
|
204
248
|
* Ask the planner, once per candidate finding, whether the flip is reachable.
|
package/dist/cli/index.js
CHANGED
|
@@ -2339,7 +2339,8 @@ async function cmdDoctor(args, config) {
|
|
|
2339
2339
|
: { findings: [], notices: [], candidatesConsidered: 0, consideredIndexed: 0, consideredUnindexed: 0 };
|
|
2340
2340
|
// Statistics can say how bad a flip WOULD be; only the planner can say whether
|
|
2341
2341
|
// it is reachable. The `unindexed-filter` branch shipped in 0.57 without that
|
|
2342
|
-
// question answered and
|
|
2342
|
+
// question answered, and a measured sample of 13 findings held up only 6 times,
|
|
2343
|
+
// so every one
|
|
2343
2344
|
// of its findings is now put to a plan-only EXPLAIN. Nothing is executed, and a
|
|
2344
2345
|
// probe that fails keeps its finding rather than dropping it.
|
|
2345
2346
|
const flipProbe = divergenceOn && scored.findings.some(needsFlipProbe)
|
|
@@ -6,13 +6,14 @@
|
|
|
6
6
|
* `plan-divergence.ts` scores a column from statistics alone and answers "IF the
|
|
7
7
|
* cached plan flips, how bad is it". Its `unindexed-filter` branch (0.57) shipped
|
|
8
8
|
* without an answer to the prior question, "CAN it flip at all", and that turned
|
|
9
|
-
* out to be the majority case:
|
|
10
|
-
* findings
|
|
9
|
+
* out to be the majority case: in validation against a large schema the branch
|
|
10
|
+
* emitted 39 findings, and a measured sample of 13 of them held up only 6 times.
|
|
11
11
|
*
|
|
12
|
-
* Every false positive had one signature: **the generic plan
|
|
13
|
-
*
|
|
12
|
+
* Every false positive had one signature: **the generic plan was not the ordered
|
|
13
|
+
* index walk the finding claims.** There was no flip to be had, so the
|
|
14
14
|
* amplification the finding printed described a plan the planner would never
|
|
15
|
-
* pick.
|
|
15
|
+
* pick. See {@link verdictFromPlanJson} for the two ways a plan fails to be that
|
|
16
|
+
* walk; 0.58.0 shipped only one of them and 0.59.0 added the other.
|
|
16
17
|
*
|
|
17
18
|
* ## Why this is a probe and not another rule
|
|
18
19
|
*
|
|
@@ -54,11 +55,10 @@
|
|
|
54
55
|
* ```
|
|
55
56
|
*
|
|
56
57
|
* The custom plan is not needed. The finding's whole claim is that a promoted
|
|
57
|
-
* generic plan
|
|
58
|
-
*
|
|
59
|
-
*
|
|
60
|
-
*
|
|
61
|
-
* carry.
|
|
58
|
+
* generic plan performs an ordered index walk, so a generic plan that is NOT that
|
|
59
|
+
* walk refutes it no matter what the custom plan does. Asking one question
|
|
60
|
+
* instead of two halves the work and removes the need for a representative rare
|
|
61
|
+
* value, which statistics do not carry.
|
|
62
62
|
*
|
|
63
63
|
* `NULL` is a safe argument precisely because the plan is generic: a generic plan
|
|
64
64
|
* is built without looking at the value, which is the property the whole check is
|
|
@@ -80,11 +80,11 @@ import type { PlanDivergenceFinding, PlanDivergenceReport } from './plan-diverge
|
|
|
80
80
|
/**
|
|
81
81
|
* The planner's answer for one finding.
|
|
82
82
|
*
|
|
83
|
-
* - `'flip-reachable'`, the generic plan
|
|
83
|
+
* - `'flip-reachable'`, the generic plan IS an ordered index walk on the target
|
|
84
84
|
* table, so the divergence the finding describes is one the planner can
|
|
85
85
|
* actually choose.
|
|
86
|
-
* - `'no-flip'`, the generic plan
|
|
87
|
-
* access
|
|
86
|
+
* - `'no-flip'`, the generic plan is not that walk (a `Sort` bounds it, or the
|
|
87
|
+
* access is a plain seq scan). Nothing to diverge to.
|
|
88
88
|
* - `'unknown'`, the probe did not produce an answer. The finding is kept.
|
|
89
89
|
*/
|
|
90
90
|
export type FlipVerdict = 'flip-reachable' | 'no-flip' | 'unknown';
|
|
@@ -133,11 +133,49 @@ export declare function buildFlipProbeSql(finding: PlanDivergenceFinding, name:
|
|
|
133
133
|
* Exported for unit tests: the plan shapes this has to classify are exactly the
|
|
134
134
|
* ones that are tedious to produce live.
|
|
135
135
|
*
|
|
136
|
-
* The
|
|
137
|
-
*
|
|
138
|
-
*
|
|
139
|
-
*
|
|
140
|
-
*
|
|
136
|
+
* ## The question, stated exactly
|
|
137
|
+
*
|
|
138
|
+
* The finding claims the generic plan performs an ORDERED INDEX WALK along the
|
|
139
|
+
* `ORDER BY` column and fetches nearly every tuple before the `LIMIT` fills. So
|
|
140
|
+
* the refutation is not "the plan is a seq scan", it is **"the plan is not that
|
|
141
|
+
* ordered walk"**, and there are two ways for it not to be:
|
|
142
|
+
*
|
|
143
|
+
* 1. A `Sort` lies above the target table's scan. A sort materializes the whole
|
|
144
|
+
* matched set and orders it, so the cost is bounded by how many rows match,
|
|
145
|
+
* not by how far into the heap the ordered walk has to travel. Whatever feeds
|
|
146
|
+
* it (seq scan, bitmap heap scan) the catastrophic shape is absent.
|
|
147
|
+
* 2. The target's own scan node is a `Seq Scan`. Kept as an independent ground
|
|
148
|
+
* rather than folded into the first, so a hypothetical ordered seq scan with
|
|
149
|
+
* no sort still refutes.
|
|
150
|
+
*
|
|
151
|
+
* 0.58.0 shipped only the second ground and therefore missed every LOW-estimate
|
|
152
|
+
* column that has ANY usable index, because those plan as `Limit > Sort > Bitmap
|
|
153
|
+
* Heap Scan`. The case that surfaced it was a column carrying
|
|
154
|
+
* `btree (col) WHERE col IS NOT NULL`: an equality predicate implies not-null, so
|
|
155
|
+
* that partial index is fully usable and the plan never reaches a seq scan.
|
|
156
|
+
* Reproduced, and the fixture is the pair below at the same estimate:
|
|
157
|
+
*
|
|
158
|
+
* ```txt
|
|
159
|
+
* partial index, est 1.9 Limit > Sort > Bitmap Heap Scan <- 0.58 kept this
|
|
160
|
+
* no index, est 1.9 Limit > Sort > Seq Scan <- 0.58 refuted this
|
|
161
|
+
* either, est 500 Limit > Index Scan (no Sort) <- both keep, correctly
|
|
162
|
+
* ```
|
|
163
|
+
*
|
|
164
|
+
* ## Why this is not "exclude partial-index columns"
|
|
165
|
+
*
|
|
166
|
+
* Because a partial index whose predicate is NOT implied by the equality does not
|
|
167
|
+
* serve the query at all, and such a column produces a genuine finding: one was
|
|
168
|
+
* measured at 19,961x. The property that matters is whether the planner COULD
|
|
169
|
+
* use it, which is a proof obligation over predicates, and the plan already
|
|
170
|
+
* carries the answer. Reading the plan is cheaper and cannot drift from the
|
|
171
|
+
* planner's own implication rules.
|
|
172
|
+
*
|
|
173
|
+
* ## The safe direction is KEEP
|
|
174
|
+
*
|
|
175
|
+
* Over-refuting deletes real findings, which is invisible in the report;
|
|
176
|
+
* over-keeping only costs noise. So `Incremental Sort` does NOT refute: it means
|
|
177
|
+
* the index supplies a PREFIX of the ordering and the walk is still partly
|
|
178
|
+
* ordered, which is closer to the catastrophic shape than to the bounded one.
|
|
141
179
|
*/
|
|
142
180
|
export declare function verdictFromPlanJson(payload: unknown, table: string): FlipVerdict;
|
|
143
181
|
export interface ProbePlanFlipsOptions {
|
package/dist/plan-flip-probe.js
CHANGED
|
@@ -6,13 +6,14 @@
|
|
|
6
6
|
* `plan-divergence.ts` scores a column from statistics alone and answers "IF the
|
|
7
7
|
* cached plan flips, how bad is it". Its `unindexed-filter` branch (0.57) shipped
|
|
8
8
|
* without an answer to the prior question, "CAN it flip at all", and that turned
|
|
9
|
-
* out to be the majority case:
|
|
10
|
-
* findings
|
|
9
|
+
* out to be the majority case: in validation against a large schema the branch
|
|
10
|
+
* emitted 39 findings, and a measured sample of 13 of them held up only 6 times.
|
|
11
11
|
*
|
|
12
|
-
* Every false positive had one signature: **the generic plan
|
|
13
|
-
*
|
|
12
|
+
* Every false positive had one signature: **the generic plan was not the ordered
|
|
13
|
+
* index walk the finding claims.** There was no flip to be had, so the
|
|
14
14
|
* amplification the finding printed described a plan the planner would never
|
|
15
|
-
* pick.
|
|
15
|
+
* pick. See {@link verdictFromPlanJson} for the two ways a plan fails to be that
|
|
16
|
+
* walk; 0.58.0 shipped only one of them and 0.59.0 added the other.
|
|
16
17
|
*
|
|
17
18
|
* ## Why this is a probe and not another rule
|
|
18
19
|
*
|
|
@@ -54,11 +55,10 @@
|
|
|
54
55
|
* ```
|
|
55
56
|
*
|
|
56
57
|
* The custom plan is not needed. The finding's whole claim is that a promoted
|
|
57
|
-
* generic plan
|
|
58
|
-
*
|
|
59
|
-
*
|
|
60
|
-
*
|
|
61
|
-
* carry.
|
|
58
|
+
* generic plan performs an ordered index walk, so a generic plan that is NOT that
|
|
59
|
+
* walk refutes it no matter what the custom plan does. Asking one question
|
|
60
|
+
* instead of two halves the work and removes the need for a representative rare
|
|
61
|
+
* value, which statistics do not carry.
|
|
62
62
|
*
|
|
63
63
|
* `NULL` is a safe argument precisely because the plan is generic: a generic plan
|
|
64
64
|
* is built without looking at the value, which is the property the whole check is
|
|
@@ -124,39 +124,83 @@ export function buildFlipProbeSql(finding, name, searchSchema) {
|
|
|
124
124
|
deallocate: `DEALLOCATE ${name}`,
|
|
125
125
|
};
|
|
126
126
|
}
|
|
127
|
-
function* walkPlan(node) {
|
|
128
|
-
yield node;
|
|
129
|
-
for (const child of node.Plans ?? [])
|
|
130
|
-
yield* walkPlan(child);
|
|
131
|
-
}
|
|
132
127
|
/**
|
|
133
128
|
* Read a verdict out of one `EXPLAIN (FORMAT JSON)` payload.
|
|
134
129
|
*
|
|
135
130
|
* Exported for unit tests: the plan shapes this has to classify are exactly the
|
|
136
131
|
* ones that are tedious to produce live.
|
|
137
132
|
*
|
|
138
|
-
* The
|
|
139
|
-
*
|
|
140
|
-
*
|
|
141
|
-
*
|
|
142
|
-
*
|
|
133
|
+
* ## The question, stated exactly
|
|
134
|
+
*
|
|
135
|
+
* The finding claims the generic plan performs an ORDERED INDEX WALK along the
|
|
136
|
+
* `ORDER BY` column and fetches nearly every tuple before the `LIMIT` fills. So
|
|
137
|
+
* the refutation is not "the plan is a seq scan", it is **"the plan is not that
|
|
138
|
+
* ordered walk"**, and there are two ways for it not to be:
|
|
139
|
+
*
|
|
140
|
+
* 1. A `Sort` lies above the target table's scan. A sort materializes the whole
|
|
141
|
+
* matched set and orders it, so the cost is bounded by how many rows match,
|
|
142
|
+
* not by how far into the heap the ordered walk has to travel. Whatever feeds
|
|
143
|
+
* it (seq scan, bitmap heap scan) the catastrophic shape is absent.
|
|
144
|
+
* 2. The target's own scan node is a `Seq Scan`. Kept as an independent ground
|
|
145
|
+
* rather than folded into the first, so a hypothetical ordered seq scan with
|
|
146
|
+
* no sort still refutes.
|
|
147
|
+
*
|
|
148
|
+
* 0.58.0 shipped only the second ground and therefore missed every LOW-estimate
|
|
149
|
+
* column that has ANY usable index, because those plan as `Limit > Sort > Bitmap
|
|
150
|
+
* Heap Scan`. The case that surfaced it was a column carrying
|
|
151
|
+
* `btree (col) WHERE col IS NOT NULL`: an equality predicate implies not-null, so
|
|
152
|
+
* that partial index is fully usable and the plan never reaches a seq scan.
|
|
153
|
+
* Reproduced, and the fixture is the pair below at the same estimate:
|
|
154
|
+
*
|
|
155
|
+
* ```txt
|
|
156
|
+
* partial index, est 1.9 Limit > Sort > Bitmap Heap Scan <- 0.58 kept this
|
|
157
|
+
* no index, est 1.9 Limit > Sort > Seq Scan <- 0.58 refuted this
|
|
158
|
+
* either, est 500 Limit > Index Scan (no Sort) <- both keep, correctly
|
|
159
|
+
* ```
|
|
160
|
+
*
|
|
161
|
+
* ## Why this is not "exclude partial-index columns"
|
|
162
|
+
*
|
|
163
|
+
* Because a partial index whose predicate is NOT implied by the equality does not
|
|
164
|
+
* serve the query at all, and such a column produces a genuine finding: one was
|
|
165
|
+
* measured at 19,961x. The property that matters is whether the planner COULD
|
|
166
|
+
* use it, which is a proof obligation over predicates, and the plan already
|
|
167
|
+
* carries the answer. Reading the plan is cheaper and cannot drift from the
|
|
168
|
+
* planner's own implication rules.
|
|
169
|
+
*
|
|
170
|
+
* ## The safe direction is KEEP
|
|
171
|
+
*
|
|
172
|
+
* Over-refuting deletes real findings, which is invisible in the report;
|
|
173
|
+
* over-keeping only costs noise. So `Incremental Sort` does NOT refute: it means
|
|
174
|
+
* the index supplies a PREFIX of the ordering and the walk is still partly
|
|
175
|
+
* ordered, which is closer to the catastrophic shape than to the bounded one.
|
|
143
176
|
*/
|
|
144
177
|
export function verdictFromPlanJson(payload, table) {
|
|
145
178
|
const root = Array.isArray(payload) ? payload[0] : undefined;
|
|
146
179
|
const plan = root?.Plan;
|
|
147
180
|
if (!plan)
|
|
148
181
|
return 'unknown';
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
182
|
+
// Walk root-downward, tracking whether a full Sort sits ABOVE the target's
|
|
183
|
+
// scan. Depth matters: a Sort somewhere else in a larger plan says nothing
|
|
184
|
+
// about how this table is reached.
|
|
185
|
+
const search = (node, sortedAbove) => {
|
|
152
186
|
const type = node['Node Type'];
|
|
153
|
-
if (
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
187
|
+
if (node['Relation Name'] === table && type !== undefined) {
|
|
188
|
+
if (sortedAbove)
|
|
189
|
+
return 'no-flip';
|
|
190
|
+
return type === 'Seq Scan' ? 'no-flip' : 'flip-reachable';
|
|
191
|
+
}
|
|
192
|
+
// 'Incremental Sort' is deliberately excluded, see the header.
|
|
193
|
+
const nowSorted = sortedAbove || type === 'Sort';
|
|
194
|
+
for (const child of node.Plans ?? []) {
|
|
195
|
+
const found = search(child, nowSorted);
|
|
196
|
+
if (found !== null)
|
|
197
|
+
return found;
|
|
198
|
+
}
|
|
199
|
+
return null;
|
|
200
|
+
};
|
|
201
|
+
// The target table not appearing at all should not happen for a statement that
|
|
202
|
+
// selects from it. Treated as unknown rather than as a refutation.
|
|
203
|
+
return search(plan, false) ?? 'unknown';
|
|
160
204
|
}
|
|
161
205
|
/**
|
|
162
206
|
* Ask the planner, once per candidate finding, whether the flip is reachable.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "turbine-orm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.59.1",
|
|
4
4
|
"description": "Postgres-native TypeScript ORM, runs on Neon, Vercel Postgres, Cloudflare, Supabase. Streaming cursors, typed errors, single-query nested relations. One dependency, no WASM engine",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"//exports": "Each subpath declares its types PER CONDITION. A single shared top-level \"types\" resolves to the ESM declarations for `require` too, which is TS1479 (\"is an ES module ... cannot be require()d\") for any CJS consumer on moduleResolution node16/nodenext. The require condition points at dist/cjs, which ships its own {\"type\":\"commonjs\"} package.json, so those declarations are CJS declarations. Gated in CI by publint + @arethetypeswrong/cli + a real .cts consumer typecheck (see the package-types job in ci.yml).",
|
|
@@ -126,8 +126,9 @@
|
|
|
126
126
|
"lint:fix": "biome check --write src/",
|
|
127
127
|
"format": "biome format --write src/",
|
|
128
128
|
"check:error-codes": "tsx scripts/check-error-codes.ts",
|
|
129
|
+
"check:changelog": "node scripts/check-changelog-headings.mjs",
|
|
129
130
|
"check:package": "publint --strict && attw --pack . --profile node16",
|
|
130
|
-
"prepublishOnly": "npm run build && npm run typecheck && npm run lint && npm run test:unit && npm run check:error-codes && npm run size",
|
|
131
|
+
"prepublishOnly": "npm run build && npm run typecheck && npm run lint && npm run test:unit && npm run check:error-codes && npm run check:changelog && npm run size",
|
|
131
132
|
"prepack": "node scripts/strip-prepare.mjs",
|
|
132
133
|
"postpack": "node scripts/restore-prepare.mjs",
|
|
133
134
|
"size": "size-limit",
|