vanara 0.1.0 → 0.2.0

package/catalog/skills/sql-index-tuning/SKILL.md

@@ -0,0 +1,171 @@
+---
+name: sql-index-tuning
+description: Diagnose slow SQL queries and add the right indexes without over-indexing — B-tree mechanics, composite ordering (equality-before-range), the leftmost-prefix rule, covering/index-only scans, reading EXPLAIN ANALYZE, selectivity, and write-amplification costs. Worked SQL examples and a runnable index-suggester.
+type: skill
+version: 2.0.0
+updated: 2026-06-29
+---
+# SQL Index Tuning
+
+Indexing is the highest-leverage performance work in most data-backed systems: the right index turns a
+multi-second sequential scan into a sub-millisecond lookup, and the wrong one quietly taxes every write
+forever. This skill is the deep reference for doing it deliberately — how B-tree indexes actually work,
+how to order composite columns, how to read a query plan, and when an index is the wrong answer. Heavy
+detail lives in `references/`; copy-paste DDL and plans in `examples/`; a runnable index-suggester in
+`scripts/`.
+
+## Mental model
+
+An index is a **sorted, redundant copy** of one or more columns plus a pointer back to the row. The
+database maintains that sort order on every write so reads can binary-search instead of scanning. Two
+consequences follow directly and explain almost every tuning decision:
+
+| Property | Why it matters |
+|---|---|
+| Indexes are **sorted** | They serve equality, range, prefix, `ORDER BY`, and `MIN/MAX` from order alone |
+| Indexes are **redundant** | Every `INSERT`/`UPDATE`/`DELETE` must also update them — pure write cost |
+| Indexes are **left-anchored** | A composite `(a, b, c)` can seek on `a`, `a,b`, `a,b,c` — never `b` or `c` alone |
+
+Keep both halves in mind at once: you are buying read speed with write speed and storage. The job is to
+buy only the indexes that pay for themselves.
+
+## 1. B-tree basics
+
+The default index type everywhere (`CREATE INDEX` builds a B-tree unless you ask otherwise) is a
+balanced tree whose leaf nodes hold sorted key values. Lookups are `O(log n)` — a few page reads even
+for billions of rows. Because the leaves are sorted and linked, a B-tree efficiently serves:
+
+- **Equality**: `WHERE status = 'open'`
+- **Range**: `WHERE created_at > '2026-01-01'`, `BETWEEN`, `<`, `>`
+- **Sorted output**: `ORDER BY created_at` with no separate sort step
+- **Prefix matches**: `WHERE email LIKE 'jay%'` (but not `'%jay'` — a leading wildcard cannot seek)
+
+It cannot help with operations that don't respect that ordering: `WHERE lower(email) = ...` (unless you
+index the expression), `!=`, or `%suffix` searches. Full mechanics, page splits, and why a low-cardinality
+boolean is a poor leading column are in [`references/btree-internals.md`](references/btree-internals.md).
+
+## 2. Composite indexes & the leftmost-prefix rule
+
+A composite index `(a, b, c)` is sorted by `a`, then `b` within equal `a`, then `c`. This is the single
+most misunderstood part of indexing. The index can satisfy a predicate only as a **left-anchored prefix**:
+
+```sql
+-- Index: (tenant_id, status, created_at)
+WHERE tenant_id = 9 AND status = 'open'                  -- uses (tenant_id, status)      ✓
+WHERE tenant_id = 9 AND status = 'open' AND created_at>… -- uses all three                ✓
+WHERE tenant_id = 9                                      -- uses (tenant_id)              ✓
+WHERE status = 'open'                                    -- CANNOT seek (skips tenant_id) ✗
+WHERE tenant_id = 9 AND created_at > …                   -- seeks tenant_id, filters rest ◑
+```
+
+**Column ordering rule: equality columns first, then one range column, then columns needed only for
+sort.** A range predicate (`>`, `<`, `BETWEEN`) "uses up" the index — columns after the range column
+can no longer be used for seeking, only as a filter. So `(tenant_id, status, created_at)` is right for
+`tenant_id = ? AND status = ? AND created_at > ?`, but putting `created_at` before `status` would waste
+the `status` equality. Full reasoning, plus covering indexes and index-only scans, in
+[`references/composite-and-covering.md`](references/composite-and-covering.md).
+
+## 3. Covering indexes & index-only scans
+
+If an index contains **every column a query touches** (in `SELECT`, `WHERE`, and `ORDER BY`), the engine
+answers from the index alone and never visits the table heap — an **index-only scan**. This eliminates the
+random I/O of fetching rows and is often a 5–50x win on hot read paths.
+
+```sql
+-- Query reads only id, tenant_id, status:
+SELECT id, status FROM orders WHERE tenant_id = 9 AND status = 'open';
+
+-- Covering index (Postgres INCLUDE keeps non-key columns in the leaf):
+CREATE INDEX ix_orders_cover ON orders (tenant_id, status) INCLUDE (id);
+```
+
+The trade-off: covering indexes are wider, so they cost more to write and store. Add them only for
+high-frequency queries you have measured. See `examples/index-ddl.sql` for `INCLUDE` vs composite forms.
+
+## 4. Reading EXPLAIN / EXPLAIN ANALYZE
+
+`EXPLAIN` shows the planner's chosen plan and **estimated** cost; `EXPLAIN ANALYZE` actually runs the
+query and reports **real** timings and row counts. Always tune against `ANALYZE` output. What to look for:
+
+- **`Seq Scan` on a large table** under a selective `WHERE` — the prime candidate for an index.
+- **`rows` estimate vs `actual`** wildly diverging — stale statistics; run `ANALYZE <table>`.
+- **`Sort` / `Hash` nodes with high cost** — an ordered index may remove the sort.
+- **`Index Scan` vs `Index Only Scan`** — the latter means your index is covering.
+- **`Rows Removed by Filter`** — rows read then thrown away; a better index seeks past them.
+
+```sql
+EXPLAIN (ANALYZE, BUFFERS)
+SELECT id FROM orders WHERE tenant_id = 9 AND created_at > now() - interval '7 days';
+```
+
+A full annotated walkthrough — before/after plans for the same query — is in `examples/explain-walkthrough.sql`
+and the node-by-node guide in [`references/reading-explain.md`](references/reading-explain.md).
+
+## 5. Selectivity & cardinality
+
+**Selectivity** is the fraction of rows a predicate keeps; **cardinality** is the number of distinct
+values in a column. An index helps in proportion to its selectivity — seeking to 50 rows out of 10M is a
+huge win; seeking to 4M out of 10M is slower than a sequential scan, and the planner will (correctly)
+ignore the index. Rules of thumb:
+
+- Lead a composite index with the **most selective equality column** the query actually filters on.
+- A boolean or low-cardinality status column is a poor standalone index but fine deeper in a composite.
+- Watch for **skew**: `status = 'archived'` (95% of rows) is unselective; `status = 'pending'` (0.1%) is
+  highly selective. Partial indexes (`WHERE status = 'pending'`) target exactly the selective slice.
+
+Details and how the planner uses histograms: [`references/selectivity-and-cardinality.md`](references/selectivity-and-cardinality.md).
+
+## 6. The write-amplification cost — when NOT to add an index
+
+Every index is extra work on every write to the table. Five indexes mean an `INSERT` does roughly six
+B-tree maintenance operations. This is **write amplification**, and it's the reason "just add an index"
+is not free advice.
+
+**Do NOT add an index when:**
+
+- The table is **write-heavy and rarely read** on that column (a log/append table).
+- The column has **very low cardinality** and the query isn't selective (a 2-value flag on its own).
+- A **redundant** index already covers the need — `(a, b)` makes a separate `(a)` index unnecessary,
+  since the composite serves `a`-only seeks via the leftmost prefix. Drop the narrower one.
+- The table is **small** (a few thousand rows); a sequential scan is already fast and the planner skips
+  the index anyway.
+- You haven't measured. Speculative indexes accumulate into pure write tax and bloat.
+
+## Common pitfalls (failure modes)
+
+- **Wrapping the indexed column in a function**: `WHERE date(created_at) = '2026-06-29'` cannot use an
+  index on `created_at`. Rewrite as a range, or build an expression index.
+- **Leading wildcard `LIKE '%foo'`** — no prefix to seek; needs a trigram or full-text index instead.
+- **Wrong composite order** — `(created_at, tenant_id)` for `tenant_id = ? AND created_at > ?` forces a
+  scan; the equality column must lead. (See the leftmost-prefix rule above.)
+- **Type/collation mismatch** — comparing a `bigint` column to a string literal, or a join across
+  mismatched collations, silently disables the index.
+- **Redundant overlapping indexes** — `(a)`, `(a, b)`, `(a, b, c)` where the widest alone suffices: pure
+  write cost, no read benefit.
+- **Trusting `EXPLAIN` over `EXPLAIN ANALYZE`** — estimates lie when statistics are stale; always verify
+  with real timings, and `ANALYZE` the table after big data changes.
+- **OR across columns** — `WHERE a = 1 OR b = 2` often can't use a single composite; consider a `UNION`
+  of two indexed seeks or separate indexes.
+
+## When NOT to use / trade-offs
+
+Indexing is the wrong lever when the bottleneck is elsewhere: an `N+1` query pattern needs batching, not
+an index; a query returning most of a table should be paginated and is already best served by a scan; and
+analytical scans over whole tables belong in a column store, not a B-tree. Reach for a different index
+type when B-tree can't help — GIN/trigram for substring and full-text search, GiST for geospatial,
+BRIN for naturally-ordered append-only data (huge time-series tables) where a tiny block-range index
+beats a full B-tree. And remember the meta-cost: every index slows writes and consumes storage and cache.
+On a write-dominated workload, fewer, well-chosen indexes beat many.
+
+Pairs with `database-scaling`, `data-modeling`, and `database-migrations` (add indexes concurrently /
+online to avoid locking production tables during deploys).
+
+## Files in this package
+
+- `references/btree-internals.md` — how B-trees store keys, page splits, what they can and can't serve
+- `references/composite-and-covering.md` — leftmost-prefix rule, equality-before-range, covering/index-only scans
+- `references/reading-explain.md` — node-by-node guide to EXPLAIN / EXPLAIN ANALYZE output
+- `references/selectivity-and-cardinality.md` — selectivity math, histograms, partial indexes, skew
+- `examples/explain-walkthrough.sql` — before/after query plans for the same query, annotated
+- `examples/index-ddl.sql` — composite, covering (`INCLUDE`), partial, and expression index DDL
+- `scripts/suggest-index.mjs` — proposes a composite index from a query's columns (equality-before-range + leftmost-prefix); run `--selftest` to verify

package/catalog/skills/sql-index-tuning/examples/explain-walkthrough.sql

@@ -0,0 +1,91 @@
+-- explain-walkthrough.sql
+-- A before/after EXPLAIN ANALYZE walkthrough for the same query, showing how the
+-- right composite index changes the plan. Annotations are inline (-- ...).
+-- Target: PostgreSQL. The same shape applies to MySQL EXPLAIN ANALYZE.
+
+-- ---------------------------------------------------------------------------
+-- Setup: a multi-tenant orders table, ~10M rows, no useful index yet.
+-- ---------------------------------------------------------------------------
+CREATE TABLE orders (
+    id          bigserial PRIMARY KEY,
+    tenant_id   bigint      NOT NULL,
+    status      text        NOT NULL,         -- 'open' | 'paid' | 'archived'
+    total_cents bigint      NOT NULL,
+    created_at  timestamptz NOT NULL DEFAULT now()
+);
+-- (assume 10M rows loaded; one tenant owns ~5k rows; ~2% are 'open')
+
+-- The query we want to make fast: a tenant's recent open orders.
+-- WHERE has two equality columns and one range column, plus an ORDER BY.
+
+-- ===========================================================================
+-- BEFORE: only the primary-key index exists.
+-- ===========================================================================
+EXPLAIN (ANALYZE, BUFFERS)
+SELECT id, total_cents
+FROM orders
+WHERE tenant_id = 9
+  AND status = 'open'
+  AND created_at > now() - interval '30 days'
+ORDER BY created_at DESC
+LIMIT 20;
+
+-- Typical plan (abridged):
+--
+--  Limit  (actual time=812.4..812.4 rows=20)
+--    ->  Sort  (actual rows=20 loops=1)               -- explicit sort step
+--          Sort Key: created_at DESC
+--          ->  Seq Scan on orders                     -- reads ALL 10M rows
+--                Filter: (tenant_id = 9 AND status = 'open' AND created_at > ...)
+--                Rows Removed by Filter: 9994800       -- threw away ~10M rows
+--                Buffers: shared read=98112            -- heavy disk I/O
+--  Execution Time: 813.1 ms
+--
+-- Diagnosis: Seq Scan + huge "Rows Removed by Filter" + a Sort node. The query
+-- is highly selective (keeps ~100 rows of 10M) but the engine has no index to
+-- seek with, so it scans everything and sorts the survivors.
+
+-- ===========================================================================
+-- THE FIX: composite index following equality-before-range + leftmost-prefix.
+-- Two equality columns lead (tenant_id, status); the range/sort column last.
+-- ===========================================================================
+CREATE INDEX CONCURRENTLY ix_orders_tenant_status_created
+    ON orders (tenant_id, status, created_at DESC);
+--                ^equality  ^equality ^range + ORDER BY (matching DESC direction)
+
+-- Refresh stats so the planner trusts the new index.
+ANALYZE orders;
+
+-- ===========================================================================
+-- AFTER: re-run the identical query.
+-- ===========================================================================
+EXPLAIN (ANALYZE, BUFFERS)
+SELECT id, total_cents
+FROM orders
+WHERE tenant_id = 9
+  AND status = 'open'
+  AND created_at > now() - interval '30 days'
+ORDER BY created_at DESC
+LIMIT 20;
+
+-- Typical plan (abridged):
+--
+--  Limit  (actual time=0.05..0.12 rows=20)
+--    ->  Index Scan using ix_orders_tenant_status_created on orders
+--          Index Cond: (tenant_id = 9 AND status = 'open' AND created_at > ...)
+--          Buffers: shared hit=6                       -- a handful of cached pages
+--  Execution Time: 0.18 ms                             -- ~4500x faster
+--
+-- Note: NO Sort node — the index is already ordered by created_at DESC, so the
+-- ORDER BY is satisfied for free. "Rows Removed by Filter" is gone because the
+-- Index Cond seeked straight to the matching slice.
+
+-- ===========================================================================
+-- BONUS: make it an index-only scan by covering total_cents (and id) so the
+-- heap is never touched.  total_cents is payload-only -> use INCLUDE.
+-- ===========================================================================
+DROP INDEX CONCURRENTLY ix_orders_tenant_status_created;
+CREATE INDEX CONCURRENTLY ix_orders_cover
+    ON orders (tenant_id, status, created_at DESC) INCLUDE (id, total_cents);
+ANALYZE orders;
+-- Re-running now yields "Index Only Scan ... Heap Fetches: 0" -- no heap I/O.

package/catalog/skills/sql-index-tuning/examples/index-ddl.sql

@@ -0,0 +1,67 @@
+-- index-ddl.sql
+-- Copy-paste index DDL covering the common shapes. Target: PostgreSQL (notes
+-- mark MySQL/SQL Server differences). Always create indexes CONCURRENTLY in
+-- production so you don't hold an exclusive lock on the table during the build.
+
+-- ---------------------------------------------------------------------------
+-- 1. Single-column index.
+--    Use for a selective equality or range on one column.
+-- ---------------------------------------------------------------------------
+CREATE INDEX CONCURRENTLY ix_users_email
+    ON users (email);
+
+-- ---------------------------------------------------------------------------
+-- 2. Composite index: equality columns first, then ONE range column, then any
+--    column needed only for ORDER BY. Serves the leftmost prefixes too:
+--    (tenant_id), (tenant_id, status), and the full triple.
+--    Query: WHERE tenant_id = ? AND status = ? AND created_at > ? ORDER BY created_at
+-- ---------------------------------------------------------------------------
+CREATE INDEX CONCURRENTLY ix_orders_tenant_status_created
+    ON orders (tenant_id, status, created_at);
+--             ^equality   ^equality ^range / sort
+
+-- A separate (tenant_id) or (tenant_id, status) index would now be REDUNDANT
+-- because the composite already serves those prefixes. Drop such overlaps:
+-- DROP INDEX CONCURRENTLY ix_orders_tenant;   -- redundant with the composite
+
+-- ---------------------------------------------------------------------------
+-- 3. Covering index (index-only scan). INCLUDE keeps payload columns in the
+--    leaf only -- they're returnable without a heap visit but don't widen the
+--    tree or impose sort semantics. Use for hot read paths you've measured.
+--    Query: SELECT id, total_cents FROM orders WHERE tenant_id = ? AND status = ?
+-- ---------------------------------------------------------------------------
+CREATE INDEX CONCURRENTLY ix_orders_cover
+    ON orders (tenant_id, status) INCLUDE (id, total_cents);
+-- MySQL/InnoDB has no INCLUDE: add the columns as trailing key columns instead:
+--   CREATE INDEX ix_orders_cover ON orders (tenant_id, status, id, total_cents);
+
+-- ---------------------------------------------------------------------------
+-- 4. Partial index: target only the selective slice of a skewed column. Tiny
+--    and cheap to maintain. Perfect for queue/worker tables scanning a rare
+--    state.  Query: WHERE status = 'pending' ORDER BY created_at
+-- ---------------------------------------------------------------------------
+CREATE INDEX CONCURRENTLY ix_orders_pending
+    ON orders (tenant_id, created_at)
+    WHERE status = 'pending';
+
+-- ---------------------------------------------------------------------------
+-- 5. Expression (functional) index: needed when the query wraps the column in a
+--    function, which otherwise disables a plain index.
+--    Query: WHERE lower(email) = ?
+-- ---------------------------------------------------------------------------
+CREATE INDEX CONCURRENTLY ix_users_lower_email
+    ON users (lower(email));
+
+-- ---------------------------------------------------------------------------
+-- 6. Unique composite constraint (also builds a usable index).
+-- ---------------------------------------------------------------------------
+CREATE UNIQUE INDEX CONCURRENTLY ux_memberships_tenant_user
+    ON memberships (tenant_id, user_id);
+
+-- ---------------------------------------------------------------------------
+-- Housekeeping: find unused indexes (pure write tax) and drop them.
+-- ---------------------------------------------------------------------------
+-- SELECT relname AS table, indexrelname AS index, idx_scan AS scans
+-- FROM pg_stat_user_indexes
+-- WHERE idx_scan = 0          -- never used since stats reset
+-- ORDER BY pg_relation_size(indexrelid) DESC;

package/catalog/skills/sql-index-tuning/references/btree-internals.md

@@ -0,0 +1,58 @@
+# B-tree Internals
+
+The B-tree (technically a B+tree in most databases) is the default index structure. Understanding its
+shape explains exactly which queries it can accelerate.
+
+## Structure
+
+- **Root and internal nodes** hold separator keys and pointers to child pages. They route a search.
+- **Leaf nodes** hold the actual sorted key values plus a pointer (tuple id / row locator) back to the
+  table row. In a B+tree, leaves are doubly-linked, so a range scan walks neighbours without re-descending
+  the tree.
+- The tree is **balanced**: every leaf is the same depth from the root. A lookup is `O(log n)` — for a
+  billion rows that's roughly 4–5 page reads.
+
+```
+                 [ . m . ]                 root (separators)
+                /         \
+        [ . f . ]         [ . t . ]        internal
+        /    |    \        /    |   \
+     a→ ... f→ ... m→ ... t→ ...           leaves: sorted keys → row pointers
+     └──── linked leaf chain for range scans ────┘
+```
+
+## Page splits and write cost
+
+Keys live in fixed-size pages. When an insert lands in a full leaf page, the page **splits** into two and
+the separator propagates up — occasionally cascading to the root. This is why inserts into the *middle* of
+the key space are costlier than appending to the end. A monotonically increasing key (an autoincrement id,
+or `created_at` on an append table) always inserts at the right edge, minimizing splits but concentrating
+contention on the rightmost page.
+
+## What a B-tree can serve
+
+| Operation | Served by B-tree? | Note |
+|---|---|---|
+| `col = v` | Yes | descend to leaf, done |
+| `col > v` / `BETWEEN` | Yes | seek then walk the leaf chain |
+| `ORDER BY col` | Yes | leaves are already sorted; no sort node |
+| `MIN(col)` / `MAX(col)` | Yes | first / last leaf entry |
+| `col LIKE 'pre%'` | Yes | prefix is a range |
+| `col LIKE '%suf'` | No | no prefix to anchor the descent |
+| `f(col) = v` | No | unless an expression index on `f(col)` exists |
+| `col != v` | No | matches almost everything; scan is cheaper |
+
+## Cardinality and the leading column
+
+Because the tree branches on the leading column first, a **low-cardinality** leading column (e.g. a
+boolean) creates only two huge subtrees — a seek lands you in 50% of the rows, which the planner will
+reject in favour of a sequential scan. Lead with selective, high-cardinality columns. Low-cardinality
+columns are useful only deeper in a composite, or as a **partial index** predicate.
+
+## Heap vs index-organized
+
+In Postgres and MySQL/InnoDB the details differ: InnoDB's primary key **is** the table (clustered /
+index-organized), so secondary indexes store the primary key as the row pointer and a lookup is two
+B-tree descents. Postgres keeps a separate heap, so secondary and primary indexes are symmetric but the
+heap can require a visibility check (mitigated by the visibility map for index-only scans). This is why a
+covering index that avoids the heap visit is such a large win in Postgres.

package/catalog/skills/sql-index-tuning/references/composite-and-covering.md

@@ -0,0 +1,60 @@
+# Composite Indexes, the Leftmost-Prefix Rule, and Covering Scans
+
+## The leftmost-prefix rule
+
+A composite index `(a, b, c)` physically sorts rows by `a`, then `b` within each `a`, then `c` within each
+`(a, b)`. A seek can therefore use the index only for a **left-anchored prefix** of the columns:
+
+| Predicate | Index columns used for seek |
+|---|---|
+| `a = ?` | `(a)` |
+| `a = ? AND b = ?` | `(a, b)` |
+| `a = ? AND b = ? AND c = ?` | `(a, b, c)` |
+| `b = ?` | none (gap at `a`) |
+| `a = ? AND c = ?` | `(a)` only; `c` becomes a filter (gap at `b`) |
+
+The moment there's a gap, the columns after the gap can't be used for seeking — only for filtering rows
+already located by the prefix.
+
+## Equality before range
+
+A range predicate (`>`, `<`, `>=`, `<=`, `BETWEEN`, `LIKE 'x%'`) returns a contiguous slice of the index.
+Everything **after** a range column in the index can no longer narrow the seek — the engine has to scan the
+whole slice and filter. Therefore:
+
+> **Order composite columns: all equality columns first, then a single range column, then any columns
+> needed only to satisfy `ORDER BY`.**
+
+Example — query `WHERE tenant_id = ? AND status = ? AND created_at > ? ORDER BY created_at`:
+
+- Good: `(tenant_id, status, created_at)` — two equalities seek precisely, the range walks the slice, and
+  the slice is already ordered by `created_at` so the sort disappears.
+- Bad: `(tenant_id, created_at, status)` — the range on `created_at` comes before the `status` equality,
+  so `status` degrades to a filter and the seek is wider than necessary.
+
+## ORDER BY and indexes
+
+An index can supply sort order for free if the `ORDER BY` columns are a prefix of the remaining index
+columns **with matching direction**. `(tenant_id, created_at)` serves
+`WHERE tenant_id = ? ORDER BY created_at ASC` with no sort node. For `ORDER BY created_at DESC`, most
+engines can scan the index backwards. Mixed directions (`a ASC, b DESC`) need a matching mixed-direction
+index.
+
+## Covering indexes and index-only scans
+
+If the index contains every column the query references, the engine never touches the table heap — an
+**index-only scan**. Two ways to make an index cover:
+
+1. **Add the columns as key columns**: `(tenant_id, status, created_at)` covers a query selecting those
+   three. Downside: they participate in sorting and widen every internal node.
+2. **`INCLUDE` non-key columns** (Postgres, SQL Server): `(tenant_id, status) INCLUDE (total, created_at)`
+   stores the extra columns only in the leaf, so they're available for the scan without bloating the tree
+   or imposing sort semantics. Prefer `INCLUDE` for payload columns you only need to return, not filter.
+
+Covering indexes are wider, so they cost more to write and store. Reserve them for measured hot paths.
+
+## Redundancy
+
+`(a, b, c)` already serves seeks on `(a)` and `(a, b)`. A separate `(a)` or `(a, b)` index is redundant —
+drop it to reclaim the write cost, unless the narrower one is also covering for a specific query or has a
+different sort direction you depend on.

package/catalog/skills/sql-index-tuning/references/reading-explain.md

@@ -0,0 +1,54 @@
+# Reading EXPLAIN / EXPLAIN ANALYZE
+
+`EXPLAIN` prints the planner's chosen execution plan and its **estimated** costs. `EXPLAIN ANALYZE`
+executes the query and adds **actual** times and row counts. Tune against `ANALYZE` — estimates are only
+as good as the table statistics.
+
+```sql
+EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
+SELECT id FROM orders
+WHERE tenant_id = 9 AND created_at > now() - interval '7 days';
+```
+
+## Plan nodes you'll meet
+
+| Node | Meaning | Signal |
+|---|---|---|
+| `Seq Scan` | read every row | bad under a selective filter on a big table — index candidate |
+| `Index Scan` | seek the index, then fetch rows from the heap | good; random heap I/O remains |
+| `Index Only Scan` | answer entirely from the index | best; the index is covering |
+| `Bitmap Heap Scan` | gather many matches, then fetch heap pages in order | good for medium-selectivity ranges |
+| `Sort` | explicit sort step | removable if an ordered index matches the `ORDER BY` |
+| `Hash Join` / `Nested Loop` | join strategies | nested loop without an inner index can explode |
+
+## The numbers that matter
+
+- **`cost=`** — planner's unitless estimate (`startup..total`). Compare alternatives, don't read as ms.
+- **`rows=` (estimate) vs `actual rows=`** — a big divergence means **stale statistics**. Run
+  `ANALYZE <table>` (or `VACUUM ANALYZE`) so the planner has fresh histograms.
+- **`actual time=`** — real `startup..total` milliseconds per loop. Multiply by `loops` for total.
+- **`Rows Removed by Filter: N`** — rows the node read and then discarded. A large number means the index
+  located far more rows than the query keeps — the leading column isn't selective enough, or a predicate
+  isn't part of the index.
+- **`Buffers: shared hit/read`** — `hit` is cache, `read` is disk. High `read` on a Seq Scan is the I/O
+  you're trying to eliminate.
+
+## A tuning loop
+
+1. Run `EXPLAIN ANALYZE` on the slow query; identify the dominant node (usually a `Seq Scan` or a `Sort`).
+2. Check `actual rows` returned vs scanned — confirm the query is actually selective (an index won't help
+   a query that keeps most of the table).
+3. Propose an index following equality-before-range and leftmost-prefix (see
+   `composite-and-covering.md`, or run `scripts/suggest-index.mjs`).
+4. Create it (use `CREATE INDEX CONCURRENTLY` in production to avoid locking).
+5. Re-run `EXPLAIN ANALYZE`. Confirm the plan switched to `Index Scan`/`Index Only Scan`, `Rows Removed by
+   Filter` dropped, and `actual time` fell. If the planner still chose a `Seq Scan`, your predicate
+   probably isn't selective enough — the index won't pay off.
+
+## Gotchas
+
+- **`EXPLAIN` without `ANALYZE` never runs the query** — safe on writes, but shows estimates only.
+- **`EXPLAIN ANALYZE` on an `INSERT`/`UPDATE`/`DELETE` executes it.** Wrap in a transaction and `ROLLBACK`.
+- The first run pays cold-cache I/O; run twice to see warm timings.
+- A loop count in a `Nested Loop` multiplies per-row cost — a cheap-looking inner node run 10,000 times is
+  your bottleneck.

package/catalog/skills/sql-index-tuning/references/selectivity-and-cardinality.md

@@ -0,0 +1,62 @@
+# Selectivity & Cardinality
+
+These two concepts decide whether an index is worth building and which column should lead a composite.
+
+## Definitions
+
+- **Cardinality** — the number of distinct values in a column. `user_id` has high cardinality; a boolean
+  `is_active` has cardinality 2.
+- **Selectivity** — the fraction of rows a predicate keeps. `WHERE id = 42` on a unique column is maximally
+  selective (1 row). `WHERE is_active = true` on a table that's 90% active is poorly selective (0.9 of
+  rows).
+
+An index is worthwhile in proportion to selectivity. Seeking to retrieve 0.001 of a table is a massive
+win; seeking to retrieve 0.5 of it is slower than a sequential scan because of random heap fetches — and
+the planner will ignore the index, correctly.
+
+## How the planner decides
+
+Databases keep **statistics** per column: number of distinct values, most-common-values (MCV) list, and a
+**histogram** of the value distribution. From these it estimates how many rows a predicate returns and
+compares the cost of an index scan (seek + random heap I/O) against a sequential scan (streaming reads).
+If the estimate is wrong because stats are stale, it makes the wrong choice — run `ANALYZE <table>` after
+bulk loads or large deletes to refresh them.
+
+```sql
+ANALYZE orders;                              -- refresh statistics
+SELECT n_distinct, most_common_vals
+FROM pg_stats WHERE tablename = 'orders' AND attname = 'status';
+```
+
+## Choosing the leading composite column
+
+Lead with the **most selective equality column the query filters on**. For
+`WHERE tenant_id = ? AND status = ?`:
+
+- If a tenant has thousands of orders and `status` has 5 values, `tenant_id` is more selective → lead with
+  it: `(tenant_id, status)`.
+- The leading column must be one the query constrains with equality (leftmost-prefix rule), so selectivity
+  is the tiebreaker among eligible equality columns, not a free choice.
+
+## Skew and partial indexes
+
+Selectivity can differ wildly **per value**. If `status = 'archived'` is 95% of rows but `status =
+'pending'` is 0.1%, an index on `status` is useless for the former and gold for the latter. A **partial
+index** targets exactly the selective slice and stays tiny:
+
+```sql
+CREATE INDEX ix_orders_pending
+ON orders (tenant_id, created_at)
+WHERE status = 'pending';
+```
+
+This index only contains pending rows, so it's small, cheap to maintain, and the planner uses it whenever
+the query includes `status = 'pending'`. Partial indexes are the standard fix for queue/worker tables
+where you repeatedly scan for a rare state.
+
+## Rules of thumb
+
+- Unique or near-unique column in the predicate → index almost always wins.
+- Predicate keeps more than ~10–20% of the table → index probably loses to a scan.
+- Low-cardinality column → only useful inside a composite, or as a partial-index predicate.
+- Re-`ANALYZE` after big data shifts; most "the index isn't being used" reports are stale statistics.