turbine-orm 0.56.0 → 0.58.0

package/dist/cjs/plan-divergence.d.ts

@@ -4,8 +4,31 @@
  * `index-advisor.ts` answers "which relation probes have no index" from pure
  * topology. `index-stats.ts` answers "is adding that index worth it" from live
  * statistics. This module answers the third question on the SAME columns: "this
- * column IS indexed, and its value distribution makes a NAMED prepared
- * statement's generic plan unsafe".
+ * column's value distribution makes a NAMED prepared statement's generic plan
+ * unsafe".
+ *
+ * TWO BRANCHES, one skeleton. They differ in what the CUSTOM plan does, which is
+ * what decides both the boundary and the units of the damage:
+ *
+ *   - `sparse-value`: the filter column IS served by a btree, so the custom plan
+ *     takes a bitmap scan over the rare value's own rows and the boundary is a
+ *     page comparison. This is the original rule and is unchanged.
+ *   - `unindexed-filter`: NO index can serve `col = $1`, so the custom plan's
+ *     only alternative is a seq scan + top-N sort. Added after an earlier
+ *     revision dropped every unindexed column BEFORE counting it as considered,
+ *     which put a whole class of real divergence outside the scored population
+ *     and outside the notices too. See the branch's own header below.
+ *
+ * "Served by an index" here means a VALID, non-partial, non-expression index
+ * whose leading key column is the filter column and whose access method is btree
+ * or HASH. Hash belongs in the first branch, not the second: a hash index gives
+ * the planner the same bitmap-over-this-value's-rows path a btree does, which is
+ * the plan the crossover is derived against. Measured on the fixture below with
+ * a hash index on the filter column, the custom plan is a Bitmap Heap Scan
+ * reading 7 buffers, not the 247-page seq scan the second branch would have
+ * described. Any OTHER access method leading with the column (brin, gin, gist,
+ * spgist) is not scored at all and is reported as a notice: those paths are
+ * lossy or shaped differently, and neither branch's model describes them.
  *
  * THE MECHANISM. Postgres may promote a named prepared statement to a GENERIC
  * plan from its sixth execution onward, and ONLY when the generic plan's
@@ -15,13 +38,37 @@
  *   - an unknown equality `col = $1` is estimated as reltuples / n_distinct;
  *   - an unknown `LIMIT $n` is estimated as 10% of the CHILD node's row estimate.
  *
- * THE ONE SHAPE THIS MODELS, stated narrowly on purpose. A read shaped
- * `WHERE col = $1 ORDER BY <other indexed column> LIMIT $n`, where the generic
- * estimate for `col = $1` sits ABOVE the plan boundary (so the generic plan
- * keeps the ordered index scan and filters) while some real values sit far
- * BELOW it (so for those values the ordered scan has to walk a large fraction
- * of the table before it accumulates one page of matches, where the custom
- * plan takes a bitmap scan over the value's own rows and sorts them).
+ * THE SHAPE BOTH BRANCHES MODEL, stated narrowly on purpose. A read shaped
+ * `WHERE col = $1 ORDER BY <other indexed column> LIMIT $n`, where the promoted
+ * generic plan keeps the ordered index scan and filters, while for the table's
+ * rarest values the custom planner picks something else entirely.
+ *
+ *   - `sparse-value` (the filter column is indexed): the generic estimate for
+ *     `col = $1` sits ABOVE the plan boundary while some real values sit far
+ *     BELOW it, so for those values the ordered scan walks a large fraction of
+ *     the table before it accumulates one page of matches, where the custom plan
+ *     takes a bitmap scan over the value's own rows and sorts them.
+ *   - `unindexed-filter` (no equality path on the filter column): the custom plan
+ *     takes a seq scan + top-N sort, which is bounded by the table's PAGES. The
+ *     generic plan still takes the ordered index walk and, because it cannot see
+ *     the value is rare, it walks nearly every TUPLE before it fills the LIMIT.
+ *     Measured on the fixture below: 250 buffers against 20,074, and Postgres
+ *     promoted it after five executions of the rare value.
+ *
+ * THE FIXTURE every `unindexed-filter` number in this file was measured on,
+ * printed once here so each number is checkable rather than asserted. PostgreSQL
+ * 16, warm cache, `synchronize_seqscans`, `max_parallel_workers_per_gather` and
+ * `jit` all off:
+ *
+ *   CREATE TABLE t (id int PRIMARY KEY, organization_id int NOT NULL, payload text NOT NULL);
+ *   INSERT INTO t SELECT g, <bucket(g)>, repeat('p', 60)
+ *     FROM generate_series(1, N) g ORDER BY (g * 2654435761::bigint) % 1000003;
+ *
+ * which is 81 rows per page (20,000 rows in 247 pages) and, because of the
+ * hash-ordered INSERT, a primary key UNCORRELATED with physical position. The
+ * read is `WHERE organization_id = $1 ORDER BY id LIMIT $2` at limit 20, on the
+ * rarest bucket. The same DDL is what `plan-divergence-unindexed.integration.test.ts`
+ * builds, so the ladders below can be re-run from the repo.
  *
  * WHAT THIS DELIBERATELY DOES NOT MODEL, because it was tried and it did not
  * work. An earlier revision carried a second rule for the opposite direction (a
@@ -42,6 +89,26 @@
  * shapes where a generic plan is the better one. A clean report is not evidence
  * of immunity.
  *
+ * A SECOND, STRUCTURAL blind spot bounds both branches: the candidate set is
+ * relation probes plus leading index columns. A plain filter column that is
+ * neither an FK nor indexed (a `status`, a `deleted_at`, a tenant column with no
+ * FK constraint) is invisible to this check in EITHER branch. Feeding the index
+ * advisor's recommendations in does not fix it, because the advisor derives from
+ * the same relation topology; closing it would need a workload source
+ * (pg_stat_statements or _turbine_metrics). The report says so where it prints
+ * how many columns were scored, rather than letting that count imply coverage.
+ *
+ * A THIRD blind spot, specific to `unindexed-filter` and disclosed on the
+ * finding rather than hidden: the branch cannot tell a heap that is in near-exact
+ * {@link PlanDivergenceFinding.orderColumn} order from one that is not, and the
+ * two read 1.2x and 80x on otherwise identical fixtures. The ordering column's
+ * pg_stats.correlation separates them, but only at the fifth decimal place of a
+ * sampled statistic, so it is reported and used to QUALIFY the finding
+ * ({@link PlanDivergenceFinding.heapNearlyOrdered}), never to suppress one. Its
+ * recall is bounded on the other side by a constant: gate 1 admits only a rarest
+ * bucket below the assumed LIMIT, which declines a measured 81x flip at bucket
+ * 60 (see the gate).
+ *
  * EXPOSURE IS NOT AN INCIDENT. A finding says the DISTRIBUTION admits a
  * damaging flip. It does NOT say the backend is choosing the bad plan today:
  * `auto` promotes only when the generic plan's ESTIMATED cost is not worse than
@@ -51,14 +118,22 @@
  * compares the two plans: the counter is the only thing that establishes real
  * exposure.
  *
- * TWO UNITS, and mixing them up is what made the removed rule wrong. The
- * PLANNER costs the boundary in PAGES (an ordered index scan reads about
- * limit / matching of the table's pages before it fills the limit), which is why
- * {@link crossoverRows} is derived from relpages and why measured flip points
- * track it. The DAMAGE a user feels is also read in pages here, deliberately
- * conservatively; when the ordering index is not correlated with heap order,
- * each row examined is a separate block access and the true buffer count is
- * larger, sometimes by an order of magnitude.
+ * TWO UNITS, and mixing them up is what made the removed rule wrong. In the
+ * `sparse-value` branch the PLANNER costs the boundary in PAGES (an ordered
+ * index scan reads about limit / matching of the table's pages before it fills
+ * the limit), which is why {@link crossoverRows} is derived from relpages and
+ * why measured flip points track it. The DAMAGE is also read in pages there,
+ * deliberately conservatively; when the ordering index is not correlated with
+ * heap order, each row examined is a separate block access and the true buffer
+ * count is larger, sometimes by an order of magnitude.
+ *
+ * The `unindexed-filter` branch reads damage in TUPLES on the generic side and
+ * PAGES on the custom side, because that is what the two plans actually are: an
+ * index-order heap fetch per row against one sequential pass. Carrying
+ * {@link PlanDivergenceFinding.walkPages} into it would under-report the generic
+ * side by exactly the rows-per-page factor, which is the entire finding. That is
+ * why the page-shaped fields are left UNSET on that branch instead of being
+ * filled with numbers from a model that does not apply.
  *
  * FRESHNESS GATE. The cost-tier half of doctor gates on `stats_reset` age,
  * because it normalizes WRITE COUNTERS by it. Nothing here reads a counter: every
@@ -103,20 +178,82 @@ export declare const PLAN_DIVERGENCE_THRESHOLDS: {
      * This replaced a `relpages >= 1000` table-size floor, which was wrong and
      * measurably so: a wrong plan on a SMALL table is not cheap, because an ordered
      * index scan's cost is driven by how much of the table it walks, not by how big
-     * the table is. On a 285-page / 18,500-row fixture the generic plan read 336
-     * buffers against the custom plan's 7 (48x), and the old floor dropped that
-     * column without even counting it as considered.
+     * the table is. Fixture, stated in full so the number is checkable:
+     *
+     *   CREATE TABLE t (id int PRIMARY KEY, organization_id int NOT NULL, payload text NOT NULL);
+     *   -- 18,500 rows, repeat('p', 85) payload => 285 relpages, buckets
+     *   -- 9,000 / 5,000 / 4,498 / 2, CREATE INDEX ON t (organization_id).
+     *   -- read: WHERE organization_id = $1 ORDER BY id LIMIT $2, rarest value, limit 20.
+     *
+     * Inserted in `id` order the generic plan read 337 buffers against the custom
+     * plan's 6 (56x); inserted in hash order, the same shape read 18,574 against 7.
+     * The old floor dropped that column without even counting it as considered.
      */
     readonly minWalkPages: 50;
     /**
      * ...and it must walk at least this FRACTION of the table. The two gates are
      * different questions (absolute cost, and how badly the plan is mismatched to
      * the value), and a finding needs both.
+     *
+     * `sparse-value` only. In the `unindexed-filter` branch the walk is the whole
+     * table by construction (gate 1 there puts the rarest bucket below the LIMIT,
+     * so the ordered scan can never fill it), which makes this gate vacuous.
+     * Reusing it there would read as a gate that did work.
      */
     readonly minWalkFraction: 0.1;
+    /**
+     * `unindexed-filter` only: how many TUPLES the promoted plan must walk before
+     * this is worth a user's attention.
+     *
+     * Amplification in that branch is rows-per-page and nothing else, so it is
+     * size independent and only an absolute floor separates a real finding from
+     * noise. Measured on the fixture in the module header, warm cache, rarest
+     * bucket 2, limit 20 (rows / relpages / custom buffers / custom ms / generic
+     * buffers / generic ms):
+     *
+     *   1,500 /    19 /    22 / 0.14 /   1,506 /  0.28
+     *   5,000 /    62 /    65 / 0.17 /   5,020 /  0.64
+     *  10,000 /   124 /   127 / 0.29 /  10,038 /  2.20
+     *  20,000 /   247 /   250 / 0.83 /  20,074 /  5.84
+     * 200,000 / 2,470 / 2,473 / 6.06 / 200,587 / 63.07
+     *
+     * (An earlier revision of this table carried a page column about 8% high, from
+     * a wider row than any fixture in this repo. The numbers above come from the
+     * DDL printed in the module header and nothing else.)
+     *
+     * 10,000 tuples is where the warm-cache delta reaches the "a millisecond or
+     * two" line {@link minWalkPages} is calibrated to, and where the cold-cache
+     * exposure (10,000 candidate random page accesses) stops being trivial. Not
+     * gated on pages instead: at 10,000 rows any realistic row width already
+     * clears 50 pages, and a narrow table with 10,000 rows in 40 pages is still a
+     * 250x flip.
+     */
+    readonly minGenericTupleWalk: 10000;
+    /**
+     * `unindexed-filter` only, and it QUALIFIES a finding rather than gating it:
+     * at or above this correlation between the heap and {@link
+     * PlanDivergenceFinding.orderColumn}, the flip is real but reads ~1x, so the
+     * finding is printed with that stated instead of with a rows-per-page ratio
+     * it would not reach.
+     *
+     * Fitted to the measured ladder on {@link
+     * PlanDivergenceFinding.orderColumnCorrelation}: 0.99998 (one page of local
+     * disorder) reads 3.1x and 0.99993 (two pages) reads 41x, so the boundary is
+     * between them. That is the fifth decimal place of a SAMPLED statistic, which
+     * is exactly why nothing is suppressed on it.
+     */
+    readonly nearExactOrderCorrelation: 0.99995;
 };
 export type PlanDivergenceThresholds = typeof PLAN_DIVERGENCE_THRESHOLDS;
+/**
+ * Which scoring rule produced a finding. They are NOT interchangeable: the
+ * boundary, the damage units, the remedy and the rendered text all differ, and
+ * the branch-shaped fields below are populated per branch.
+ */
+export type PlanDivergenceBranch = 'sparse-value' | 'unindexed-filter';
 export interface PlanDivergenceFinding {
+    /** Which rule scored this column. See {@link PlanDivergenceBranch}. */
+    branch: PlanDivergenceBranch;
     table: string;
     column: string;
     /** pg_class.reltuples. */
@@ -131,19 +268,111 @@ export interface PlanDivergenceFinding {
     rarestBucket: number;
     /** Estimated rowcount of the densest value (MCV maximum). Reported, not gated on. */
     densestBucket: number;
-    /** pg_stats.correlation, signed as reported. Reported, not gated on. */
+    /**
+     * pg_stats.correlation OF THE FILTER COLUMN, signed as reported. Reported,
+     * never gated on.
+     *
+     * It is NOT the statistic that decides how much the `unindexed-filter` flip
+     * costs. That one is {@link orderColumnCorrelation}, and an earlier revision
+     * printed THIS number next to a sentence about the ORDER column's physical
+     * order, which is a different column and, on the fixtures below, a different
+     * number by three orders of magnitude.
+     */
     correlation: number;
-    /** sqrt(assumedLimit x pages): below this many true rows, bitmap+sort wins. */
-    crossoverRows: number;
-    /** The same crossover at {@link PLAN_DIVERGENCE_THRESHOLDS.wideLimit}. */
-    crossoverRowsWide: number;
+    /**
+     * pg_stats.correlation OF {@link orderColumn}: how closely the heap's physical
+     * order tracks the column the generic plan walks. Null when the column has no
+     * pg_stats row (it is read opportunistically, and a missing one is an unknown,
+     * never a zero).
+     *
+     * This is what decides the SIZE of an `unindexed-filter` flip, because the
+     * generic plan's cost is one heap fetch per index entry: when consecutive
+     * entries land on the same already-pinned page the fetch is free, and when
+     * they do not it is a buffer access. Measured on six otherwise identical
+     * 20,000-row / 247-page fixtures (the module-header DDL), varying ONLY the
+     * INSERT ordering, custom plan 250 buffers in every one:
+     *
+     *   1.00000  exact id order                    generic     303 buffers,  1.2x
+     *   0.99998  shuffled within ~1 page           generic     783 buffers,  3.1x
+     *   0.99993  shuffled within ~2 pages          generic  10,303 buffers,   41x
+     *   0.99974  shuffled within ~4 pages          generic  15,148 buffers,   61x
+     *   0.99372  shuffled within ~20 pages         generic  19,046 buffers,   76x
+     *  -0.00065  hash order                        generic  20,074 buffers,   80x
+     *
+     * (An earlier revision of this table claimed 25x for the one-page row. It does
+     * not reproduce: the transition is between a one-page and a two-page window,
+     * because an index scan holds its heap pin, so disorder WITHIN a page costs
+     * nothing.)
+     *
+     * The plan flips in all six; only the magnitude differs. The boundary sits
+     * between two adjacent sampled values, so this is used to QUALIFY a finding
+     * ({@link heapNearlyOrdered}), never to suppress one: gating here would be
+     * fitted to the fifth decimal place of a sampled statistic and would drop
+     * genuine 41x to 80x findings.
+     */
+    orderColumnCorrelation?: number | null;
+    /**
+     * True when {@link orderColumnCorrelation} is at or above
+     * {@link PLAN_DIVERGENCE_THRESHOLDS.nearExactOrderCorrelation}, i.e. the heap
+     * is in near-exact {@link orderColumn} order and the measured amplification is
+     * ~1x rather than the rows-per-page figure in
+     * {@link worstCaseAmplification}.
+     *
+     * The known false positive of the `unindexed-filter` branch, disclosed as a
+     * field so a `--json` consumer sees it without re-deriving the rule. It is a
+     * hint to measure, not a verdict: it is one sampled statistic away from the
+     * 41x row of the ladder above, in both directions.
+     */
+    heapNearlyOrdered?: boolean;
+    /**
+     * sqrt(assumedLimit x pages): below this many true rows, bitmap+sort wins.
+     *
+     * `sparse-value` ONLY, and absent on the other branch rather than zero. The
+     * derivation compares an ordered index scan against a BITMAP scan; with no
+     * index on the filter column there is no bitmap path, and the real boundary is
+     * linear in the limit and independent of pages (see the branch-B header).
+     * Emitting a sqrt number there would be a measurement-shaped lie.
+     */
+    crossoverRows?: number;
+    /** The same crossover at {@link PLAN_DIVERGENCE_THRESHOLDS.wideLimit}. `sparse-value` only. */
+    crossoverRowsWide?: number;
     assumedLimit: number;
-    /** How many distinct values sit on the wrong side of the crossover. */
-    valuesBelowCrossover: number;
-    /** Pages the generic plan's ordered scan walks for the rarest value. */
-    walkPages: number;
-    /** {@link walkPages} as a fraction of the table. */
-    walkFraction: number;
+    /** How many distinct values sit on the wrong side of the crossover. `sparse-value` only. */
+    valuesBelowCrossover?: number;
+    /** Pages the generic plan's ordered scan walks for the rarest value. `sparse-value` only. */
+    walkPages?: number;
+    /** {@link walkPages} as a fraction of the table. `sparse-value` only. */
+    walkFraction?: number;
+    /**
+     * `unindexed-filter` ONLY: how many tuples the promoted ordered scan fetches
+     * before it fills the LIMIT, `min(rows, rows x limit / rarestBucket)`.
+     *
+     * On this branch it is ALWAYS exactly `rows`, and that is arithmetic rather
+     * than a coincidence: gate 1 puts `rarestBucket` below the limit, so
+     * `rows x limit / rarestBucket > rows` and the `min` always takes `rows`. The
+     * general formula is kept because gate 1 is the thing most likely to become
+     * size-aware (see its comment), and it was checked against measurement on one
+     * table at three bucket sizes: bucket 2 predicted 20,000 measured 19,992;
+     * bucket 20 predicted 20,000 measured 19,854; bucket 60 predicted 6,667
+     * measured 6,774. Two of those three bucket sizes are outside what gate 1
+     * currently admits, and the formula also assumes the rare value is spread
+     * uniformly in ORDER-column space: on a fixture with the rare rows contiguous
+     * at the tail of `id` order it predicted 66,667 and measured 200,547.
+     */
+    tuplesWalked?: number;
+    /**
+     * `unindexed-filter` ONLY: {@link tuplesWalked} / pages, i.e. the generic
+     * plan's buffer accesses against the seq scan's. Given the collapse above it
+     * is exactly the table's rows-per-page, which is a table property: it says how
+     * bad the flip is IF it happens, and nothing about how likely this column is
+     * to be the one that flips.
+     *
+     * An ESTIMATE, not a bound, in both directions. It assumes each index-order
+     * heap fetch is a separate buffer access, which is true when the heap is not
+     * ordered by {@link orderColumn} and false when it is: see
+     * {@link heapNearlyOrdered}, where the measured reading is ~1x instead.
+     */
+    worstCaseAmplification?: number;
     /**
      * walkPages / the pages the custom plan's bitmap scan reads, as a ROUGH scale
      * only. It is an estimate from statistics, not a bound in either direction:
@@ -153,8 +382,10 @@ export interface PlanDivergenceFinding {
      * while calibrating this, the true amplification came out LARGER than the
      * estimate, never smaller, but that is three fixtures and not a guarantee.
      * The EXPLAIN pair shipped with the finding is what settles it.
+     *
+     * `sparse-value` only.
      */
-    approxAmplification: number;
+    approxAmplification?: number;
     /** The other indexed column the generic plan can order by (usually the PK). */
     orderColumn: string;
     /**
@@ -186,6 +417,28 @@ export interface PlanDivergenceReport {
     notices: PlanDivergenceNotice[];
     /** How many (table, column) candidates were examined against live statistics. */
     candidatesConsidered: number;
+    /**
+     * The same count split by whether the filter column has a plain btree. Reported
+     * separately because "N columns were scored" used to be true only of indexed
+     * ones, and the unindexed population was silently outside it.
+     */
+    consideredIndexed: number;
+    consideredUnindexed: number;
+    /**
+     * True when every `unindexed-filter` finding was put to the planner via
+     * `plan-flip-probe.ts` rather than reported on statistics alone.
+     *
+     * Absent or false means the findings below are UNVERIFIED: the probe was
+     * skipped, the engine is not Postgres, or the connection refused it.
+     */
+    flipProbed?: boolean;
+    /**
+     * How many candidate findings the planner refuted, i.e. columns where the
+     * generic plan keeps the same sequential scan the good plan uses so there is
+     * no divergence to reach. Reported because a check that silently discards two
+     * thirds of what it found should say so.
+     */
+    flipRefuted?: number;
 }
 /** A (table, column) pair whose distribution statistics the collector should read. */
 export interface DivergenceCandidate {
@@ -201,8 +454,29 @@ export interface DivergenceCandidate {
  * Purely schema-derived, so the collector knows what to read BEFORE any stats
  * exist. Whether the column is really served by a btree is decided later,
  * against the live index list.
+ *
+ * NOTE, because it is the obvious next idea and it is a no-op: feeding
+ * `findMissingRelationIndexes`' recommended columns in here adds nothing. Both
+ * derive from the same relation topology, this walker already enumerates every
+ * single-column relation probe whether or not it is indexed, and the advisor's
+ * only extra members are COMPOSITE probes, which this check deliberately does
+ * not model. The unindexed population was never missing from the candidate set;
+ * it was dropped later, by a shape gate. That gate is where it was fixed.
  */
 export declare function collectDivergenceCandidateColumns(schema: SchemaMetadata): DivergenceCandidate[];
+/**
+ * The columns that could end up as a finding's {@link
+ * PlanDivergenceFinding.orderColumn}, so the collector reads their correlation
+ * too.
+ *
+ * A superset, deliberately: which one the generic plan would order by is decided
+ * later against the live index list, and a pg_stats read is cheap next to
+ * getting the wrong column's statistic. It is a SEPARATE list from the candidate
+ * columns because the two are filtered differently: a primary key is excluded
+ * from the candidates (a unique equality matches one row) and is the single most
+ * likely ordering column.
+ */
+export declare function collectDivergenceOrderColumns(schema: SchemaMetadata): DivergenceCandidate[];
 /**
  * The plan-boundary crossover: below this many truly matching rows, running
  * `ORDER BY <other column> LIMIT n` as bitmap-scan + sort is cheaper than
@@ -229,13 +503,21 @@ export declare function crossoverRows(pages: number, limit: number): number;
  * Score every candidate column against the snapshot and return the ones whose
  * distribution admits a damaging generic-plan flip.
  *
- * ONE rule, for the direction that could be calibrated: the generic estimate
- * sits ABOVE the plan boundary (so a promoted plan keeps the ordered index
- * scan) while the table's rarest values sit below it, and the pages that
- * ordered scan must walk for such a value are worth a user's attention both in
- * absolute terms and as a fraction of the table.
+ * TWO rules, split on whether a plain btree serves the filter column, because
+ * that is what decides what the CUSTOM plan does and therefore where the
+ * boundary is and what units the damage is in.
+ *
+ * `sparse-value` (indexed): the generic estimate sits ABOVE the plan boundary
+ * (so a promoted plan keeps the ordered index scan) while the table's rarest
+ * values sit below it, and the pages that ordered scan must walk for such a
+ * value are worth a user's attention both in absolute terms and as a fraction of
+ * the table.
+ *
+ * `unindexed-filter` (no btree): see the block above its gates. Its boundary is
+ * linear in the LIMIT rather than sqrt in the pages, and it carries NO
+ * generic-side gate at all, for a measured reason recorded there.
  *
- * `correlation` is reported but no longer gates anything. It used to route
+ * `correlation` is reported but gates nothing in EITHER branch. It used to route
  * clustered columns to a second rule; that rule is gone (see the module header),
  * and routing on it also made the advisor STRUCTURALLY unable to report a real
  * sparse-direction flip on any column that happened to be clustered, which was