@harness-engineering/cli 1.24.0 → 1.24.1

package/dist/agents/skills/gemini-cli/db-query-statistics/SKILL.md

@@ -0,0 +1,210 @@
+# Query Statistics and Selectivity
+
+> How the planner uses table statistics (pg_stats, histograms, most-common-values) to estimate row counts and choose execution plans.
+
+## When to Use
+
+- Diagnosing planner misestimations (estimated rows vs actual rows differ by 10x+)
+- Understanding why the planner chose a bad plan despite correct indexes
+- Tuning statistics targets for columns with skewed distributions
+- After bulk data loads, migrations, or large deletes that change data distribution
+- Investigating queries where EXPLAIN shows correct indexes but wrong join strategies
+
+## Instructions
+
+### Key Concepts
+
+The planner does not look at actual table data at query time. Instead, it uses pre-computed statistics stored in `pg_statistic` (accessible via the `pg_stats` view). These statistics are sampled approximations, not exact counts.
+
+**Key statistics columns in `pg_stats`:**
+
+- `n_distinct` -- estimated number of distinct values. Positive values are absolute counts; negative values are fractions of total rows (e.g., -1.0 means every row is unique)
+- `most_common_vals` -- the N most frequent values in the column
+- `most_common_freqs` -- the frequency of each most-common value (fractions of total rows)
+- `histogram_bounds` -- equal-frequency histogram bucket boundaries for values not in the MCV list
+- `null_frac` -- fraction of rows that are NULL
+- `correlation` -- how well the physical row order matches the logical (sorted) order. Values near 1.0 or -1.0 mean high correlation
+
+**The ANALYZE command** samples the table and updates these statistics:
+
+```sql
+ANALYZE orders;                    -- analyze one table
+ANALYZE orders (status);            -- analyze one column
+ANALYZE;                            -- analyze all tables in the database
+```
+
+Autovacuum runs ANALYZE automatically, but it may lag behind large data changes.
+
+### Worked Example
+
+Examining statistics for the `status` column on an orders table:
+
+```sql
+SELECT
+  attname,
+  n_distinct,
+  most_common_vals,
+  most_common_freqs,
+  null_frac,
+  correlation
+FROM pg_stats
+WHERE tablename = 'orders' AND attname = 'status';
+```
+
+```
+ attname | n_distinct | most_common_vals            | most_common_freqs        | null_frac | correlation
+---------+------------+-----------------------------+--------------------------+-----------+------------
+ status  |          3 | {completed,active,cancelled} | {0.85,0.12,0.03}         |         0 |       0.15
+```
+
+**How the planner uses this:** For `WHERE status = 'active'`, the estimated rows = total_rows \* 0.12. On a 10M-row table, the estimate is 1.2M rows. For `WHERE status = 'cancelled'`, the estimate is 300K rows.
+
+**Misestimation scenario:** After a data migration that marked 90% of orders as cancelled (previously 3%), the statistics are stale:
+
+```sql
+EXPLAIN ANALYZE
+SELECT * FROM orders WHERE status = 'cancelled';
+```
+
+```
+Index Scan using idx_orders_status on orders
+  (cost=0.43..12345.67 rows=300000 width=52)
+  (actual time=0.031..8923.450 rows=9000000 loops=1)
+```
+
+Estimated 300K rows, actual 9M -- a 30x misestimation. The planner chose Index Scan (good for 300K) instead of Seq Scan (better for 9M). Fix:
+
+```sql
+ANALYZE orders;
+```
+
+After ANALYZE, the planner sees the updated distribution and switches to Seq Scan:
+
+```
+Seq Scan on orders (cost=0.00..223456.00 rows=9010000 width=52)
+  (actual time=0.012..1234.567 rows=9000000 loops=1)
+  Filter: (status = 'cancelled')
+Execution Time: 1345.678 ms
+```
+
+**Increasing statistics granularity** for skewed distributions:
+
+```sql
+ALTER TABLE orders ALTER COLUMN status SET STATISTICS 1000;
+ANALYZE orders;
+```
+
+The default `default_statistics_target` is 100 (100 histogram buckets and 100 MCV entries). For highly skewed columns, increasing to 500-1000 gives the planner a more accurate picture.
+
+### Anti-Patterns
+
+1. **Never running ANALYZE after bulk operations.** After a large INSERT, DELETE, or UPDATE that changes data distribution, statistics are stale. The planner uses the old distribution, producing bad plans. Always `ANALYZE tablename;` after bulk changes.
+
+2. **Setting default_statistics_target too low.** The default of 100 works for uniform distributions but fails for skewed data. Columns with thousands of distinct values at varying frequencies need higher targets.
+
+3. **Ignoring n_distinct misestimates.** For high-cardinality columns (e.g., user_id with 50M distinct values), the sampled n_distinct can be significantly off. Override with: `ALTER TABLE orders ALTER COLUMN user_id SET (n_distinct = -1);` (tells the planner every value is unique).
+
+4. **Disabling autovacuum.** Autovacuum also runs auto-ANALYZE. Disabling it means statistics are never refreshed automatically, guaranteeing plan degradation over time.
+
+### PostgreSQL Specifics
+
+**default_statistics_target** controls the number of histogram buckets and MCV entries. Default: 100. Maximum: 10000. Higher values increase ANALYZE time but improve estimation accuracy:
+
+```sql
+-- Global setting:
+SET default_statistics_target = 200;
+
+-- Per-column override:
+ALTER TABLE events ALTER COLUMN event_type SET STATISTICS 500;
+```
+
+**Extended statistics** for correlated columns (PostgreSQL 10+):
+
+```sql
+CREATE STATISTICS stat_orders_status_region (dependencies)
+ON status, region FROM orders;
+ANALYZE orders;
+```
+
+Without extended statistics, the planner assumes columns are independent. If `status = 'active'` AND `region = 'us-east'` are correlated (e.g., 80% of us-east orders are active), the default estimation multiplies their individual frequencies, producing a significant underestimate.
+
+**Monitoring ANALYZE freshness:**
+
+```sql
+SELECT relname, last_analyze, last_autoanalyze, n_live_tup, n_dead_tup
+FROM pg_stat_user_tables
+WHERE relname = 'orders';
+```
+
+If `last_autoanalyze` is days old and `n_dead_tup` is high, autovacuum is falling behind.
+
+## Details
+
+### Advanced Topics
+
+**Multivariate statistics evolution:**
+
+- PostgreSQL 10: functional dependencies (`dependencies`)
+- PostgreSQL 12: MCV lists for column combinations (`mcv`)
+- PostgreSQL 14: expression statistics (`expressions`)
+
+```sql
+CREATE STATISTICS stat_orders_multi (dependencies, mcv)
+ON status, region, tenant_id FROM orders;
+```
+
+**Selectivity estimation for complex predicates.** AND clauses multiply individual selectivities (assuming independence). OR clauses use inclusion-exclusion. NOT inverts selectivity. These assumptions break down for correlated columns -- extended statistics fix this.
+
+**The 1/n_distinct fallback.** For values not in the MCV list and not in the histogram range, PostgreSQL estimates selectivity as 1/n_distinct. This is a rough guess and can be significantly wrong for new or rare values.
+
+**pg_statistic_ext** stores extended statistics data. Query it to verify that your extended statistics are being computed:
+
+```sql
+SELECT stxname, stxkeys, stxkind
+FROM pg_statistic_ext
+WHERE stxrelid = 'orders'::regclass;
+```
+
+### Engine Differences
+
+**MySQL** uses `ANALYZE TABLE` (not just `ANALYZE`) to update statistics:
+
+```sql
+ANALYZE TABLE orders;
+```
+
+MySQL stores statistics in `mysql.innodb_index_stats` and `mysql.innodb_table_stats`. Key differences:
+
+- **Persistent statistics** (`innodb_stats_persistent = ON` by default since 5.6): statistics survive restarts
+- **Sampling pages** (`innodb_stats_persistent_sample_pages`, default 20): controls sample size for ANALYZE. Much smaller than PostgreSQL's default of 30,000 \* `default_statistics_target` rows
+- **Histogram support** added in MySQL 8.0: `ANALYZE TABLE t UPDATE HISTOGRAM ON col WITH 100 BUCKETS;`
+- **No extended/multivariate statistics** -- MySQL assumes column independence for all multi-column predicates
+
+MySQL's optimizer uses a simpler statistics model overall. For complex queries with correlated columns, PostgreSQL's extended statistics provide significantly better estimates.
+
+### Real-World Case Studies
+
+**Reporting system with daily batch inserts of 5M rows.** After each nightly batch, morning report queries degraded from 2 seconds to 30+ seconds. Investigation showed that autovacuum's ANALYZE had not run since the batch completed -- the statistics still reflected the previous day's distribution. Adding `ANALYZE reporting_events;` to the batch job's post-load step ensured fresh statistics. Query performance remained consistent at 2 seconds. Selectivity estimation error dropped from 100x (stale stats) to under 2x (fresh stats).
+
+## Source
+
+- [PostgreSQL Planner Statistics](https://www.postgresql.org/docs/current/planner-stats.html)
+- [PostgreSQL Row Estimation Examples](https://www.postgresql.org/docs/current/row-estimation-examples.html)
+
+## Process
+
+1. Read the key concepts to understand how the planner uses pg_stats for selectivity estimation.
+2. Apply ANALYZE after bulk data changes, and increase statistics targets for columns with skewed distributions.
+3. Verify by checking `pg_stats` for your key columns and comparing EXPLAIN estimated rows to actual rows.
+
+## Harness Integration
+
+- **Type:** knowledge -- this skill is a reference document, not a procedural workflow.
+- **No tools or state** -- consumed as context by other skills and agents.
+- **related_skills:** db-explain-reading, db-scan-types, db-query-rewriting, db-btree-index
+
+## Success Criteria
+
+- Statistics freshness is verified after bulk operations (`ANALYZE` runs in post-load steps).
+- Misestimations are diagnosed by comparing pg_stats values to actual data distribution.
+- Extended statistics are created for correlated columns that cause multi-column estimation errors.

package/dist/agents/skills/gemini-cli/db-query-statistics/skill.yaml

@@ -0,0 +1,41 @@
+name: db-query-statistics
+version: '1.0.0'
+description: pg_stats, histogram bounds, selectivity estimation, and the ANALYZE command
+stability: static
+cognitive_mode: advisory-guide
+type: knowledge
+tier: 3
+triggers:
+  - manual
+platforms:
+  - claude-code
+  - gemini-cli
+  - cursor
+  - codex
+tools: []
+paths: []
+related_skills:
+  - db-explain-reading
+  - db-scan-types
+  - db-query-rewriting
+  - db-btree-index
+stack_signals:
+  - postgresql
+  - mysql
+  - database
+keywords:
+  - pg_stats
+  - histogram
+  - selectivity
+  - ANALYZE
+  - statistics
+  - n_distinct
+  - most_common_vals
+  - cardinality-estimation
+metadata:
+  author: community
+  upstream: postgresql.org/docs/current/planner-stats.html
+state:
+  persistent: false
+  files: []
+depends_on: []

package/dist/agents/skills/gemini-cli/db-read-phenomena/SKILL.md

@@ -0,0 +1,205 @@
+# Read Phenomena
+
+> The SQL standard defines three read anomalies (dirty, non-repeatable, phantom) that isolation levels progressively prevent, plus PostgreSQL adds write skew as a fourth anomaly relevant to Serializable.
+
+## When to Use
+
+- Debugging data inconsistencies in concurrent systems
+- Understanding the trade-offs table in `SET TRANSACTION ISOLATION LEVEL` documentation
+- Choosing the correct isolation level to prevent a specific anomaly
+- Explaining why a query "sees stale data" or returns unexpected results
+- Diagnosing race conditions in multi-user applications
+
+## Instructions
+
+### Key Concepts
+
+**1. Dirty Read**
+
+Reading uncommitted data from another transaction. If that transaction rolls back, you acted on data that never existed.
+
+```sql
+-- Session A                           -- Session B
+BEGIN;
+UPDATE accounts SET balance = 0
+  WHERE id = 1;
+                                       -- Dirty read: sees balance = 0
+                                       -- even though A has not committed
+ROLLBACK;
+                                       -- The balance was never actually 0
+```
+
+Not possible in PostgreSQL at any isolation level -- even Read Uncommitted prevents dirty reads.
+
+**2. Non-Repeatable Read**
+
+Reading the same row twice in a transaction and getting different values because another transaction committed an UPDATE between reads.
+
+```sql
+-- Session A (Read Committed)          -- Session B
+BEGIN;
+SELECT balance FROM accounts
+  WHERE id = 1;  -- returns 500
+                                       BEGIN;
+                                       UPDATE accounts SET balance = 400
+                                         WHERE id = 1;
+                                       COMMIT;
+SELECT balance FROM accounts
+  WHERE id = 1;  -- returns 400
+COMMIT;
+```
+
+Possible under Read Committed. Prevented by Repeatable Read and Serializable.
+
+**3. Phantom Read**
+
+Re-executing a query and getting different rows because another transaction committed an INSERT or DELETE matching the WHERE clause.
+
+```sql
+-- Session A (Read Committed)          -- Session B
+BEGIN;
+SELECT COUNT(*) FROM orders
+  WHERE status = 'pending';
+  -- returns 5
+                                       BEGIN;
+                                       INSERT INTO orders (status)
+                                         VALUES ('pending');
+                                       COMMIT;
+SELECT COUNT(*) FROM orders
+  WHERE status = 'pending';
+  -- returns 6 (phantom row appeared)
+COMMIT;
+```
+
+Possible under Read Committed. In PostgreSQL, Repeatable Read also prevents phantom reads (MVCC snapshot). In MySQL InnoDB, Repeatable Read uses gap locking for phantom prevention.
+
+**4. Write Skew**
+
+Two transactions read overlapping data, make decisions based on what they read, and write non-overlapping data that together violate a constraint. Only prevented by Serializable.
+
+```sql
+-- Hospital on-call: at least one doctor must be on-call
+-- doctors table: (name, on_call) = [('Alice', true), ('Bob', true)]
+
+-- Session A (Repeatable Read)         -- Session B (Repeatable Read)
+BEGIN;                                 BEGIN;
+SELECT COUNT(*) FROM doctors
+  WHERE on_call = true;
+  -- returns 2, safe to drop
+                                       SELECT COUNT(*) FROM doctors
+                                         WHERE on_call = true;
+                                         -- returns 2, safe to drop
+UPDATE doctors SET on_call = false
+  WHERE name = 'Alice';
+                                       UPDATE doctors SET on_call = false
+                                         WHERE name = 'Bob';
+COMMIT;                                COMMIT;
+-- Both succeed! No one is on-call -- constraint violated.
+```
+
+**Phenomena vs Isolation Level Matrix (SQL Standard)**
+
+| Phenomenon     | Read Uncommitted | Read Committed | Repeatable Read | Serializable |
+| -------------- | ---------------- | -------------- | --------------- | ------------ |
+| Dirty Read     | Possible         | Prevented      | Prevented       | Prevented    |
+| Non-Repeatable | Possible         | Possible       | Prevented       | Prevented    |
+| Phantom Read   | Possible         | Possible       | Possible\*      | Prevented    |
+| Write Skew     | Possible         | Possible       | Possible        | Prevented    |
+
+\*PostgreSQL's Repeatable Read also prevents phantom reads (stricter than the SQL standard minimum).
+
+### Worked Example
+
+Write skew under Repeatable Read vs Serializable -- the on-call scheduling problem:
+
+```sql
+-- Setup
+CREATE TABLE doctors (
+  name TEXT PRIMARY KEY,
+  on_call BOOLEAN NOT NULL DEFAULT false
+);
+INSERT INTO doctors VALUES ('Alice', true), ('Bob', true);
+
+-- Under REPEATABLE READ: both transactions succeed (write skew)
+-- Session A
+BEGIN ISOLATION LEVEL REPEATABLE READ;
+SELECT COUNT(*) FROM doctors WHERE on_call = true;  -- 2
+UPDATE doctors SET on_call = false WHERE name = 'Alice';
+COMMIT;  -- succeeds
+
+-- Session B (concurrent)
+BEGIN ISOLATION LEVEL REPEATABLE READ;
+SELECT COUNT(*) FROM doctors WHERE on_call = true;  -- 2 (snapshot)
+UPDATE doctors SET on_call = false WHERE name = 'Bob';
+COMMIT;  -- succeeds -- nobody on-call!
+
+-- Under SERIALIZABLE: one transaction aborts
+-- Session A
+BEGIN ISOLATION LEVEL SERIALIZABLE;
+SELECT COUNT(*) FROM doctors WHERE on_call = true;  -- 2
+UPDATE doctors SET on_call = false WHERE name = 'Alice';
+COMMIT;  -- succeeds
+
+-- Session B (concurrent)
+BEGIN ISOLATION LEVEL SERIALIZABLE;
+SELECT COUNT(*) FROM doctors WHERE on_call = true;  -- 2
+UPDATE doctors SET on_call = false WHERE name = 'Bob';
+COMMIT;  -- ERROR: could not serialize access
+```
+
+### Anti-Patterns
+
+1. **Ignoring non-repeatable reads in Read Committed.** Reading a value, doing computation, then using it for a write without rechecking. The value may have changed between the SELECT and the UPDATE. Use `SELECT FOR UPDATE` or a higher isolation level.
+
+2. **Believing Repeatable Read prevents all anomalies.** Write skew is still possible in PostgreSQL Repeatable Read. If your transaction reads data that influences a write to different rows, Repeatable Read is not sufficient.
+
+3. **Using SELECT then INSERT without protection.** Classic TOCTOU (time-of-check-time-of-use) race: check if a username exists, then insert. Between the check and insert, another transaction can insert the same username. Use unique constraints, `SELECT FOR UPDATE`, or Serializable isolation.
+
+### PostgreSQL Specifics
+
+- PostgreSQL prevents dirty reads at all levels, even Read Uncommitted
+- PostgreSQL Repeatable Read also prevents phantom reads (stricter than SQL standard)
+- The phenomena matrix for PostgreSQL differs from the SQL standard minimum guarantees
+- `SET TRANSACTION READ ONLY` with Serializable gives a deferrable snapshot that never aborts
+- Monitor hot rows with `pg_stat_user_tables.n_tup_hot_updated` as a signal for non-repeatable read risk
+
+## Details
+
+### Advanced Topics
+
+**Read-only transactions:** `SET TRANSACTION READ ONLY, DEFERRABLE` under Serializable gives a snapshot that never needs to abort for serialization failures. Ideal for reporting queries that must see a consistent snapshot without retry logic overhead.
+
+**Detecting phenomena in production:** Non-repeatable reads are difficult to observe directly. Look for symptoms: inconsistent aggregations, business logic that produces impossible states, and constraint violations that appear intermittently under load.
+
+### Engine Differences
+
+MySQL InnoDB uses **gap locking** to prevent phantom reads under Repeatable Read -- a fundamentally different mechanism than PostgreSQL's MVCC snapshots. Gap locks block concurrent inserts into index ranges, which can cause unexpected blocking and even deadlocks.
+
+MySQL does not support SSI. Its Serializable level uses shared read locks on all SELECT statements, which prevents write skew by blocking concurrent modifications but significantly reduces throughput compared to PostgreSQL's optimistic SSI approach.
+
+### Real-World Case Studies
+
+An inventory system allowed concurrent orders. Two customers each checked that stock > 0 for the last item, then both placed orders. Under Repeatable Read, both transactions saw stock = 1, both decremented to 0, resulting in stock = -1. This write skew was invisible at Repeatable Read because each transaction wrote to a different row (the order table) based on reading a shared row (inventory). The fix was either Serializable isolation for the checkout flow or an explicit `SELECT stock FROM inventory WHERE product_id = $1 FOR UPDATE` to serialize access to the inventory row.
+
+## Source
+
+- [PostgreSQL Transaction Isolation](https://www.postgresql.org/docs/current/transaction-iso.html)
+- Berenson, H. et al. "A Critique of ANSI SQL Isolation Levels" (SIGMOD 1995)
+
+## Process
+
+1. Identify which read phenomenon is causing the data inconsistency in your system.
+2. Consult the phenomena matrix to determine the minimum isolation level that prevents it.
+3. Implement the isolation level change with appropriate retry logic, or use explicit locking if upgrading isolation is too expensive.
+
+## Harness Integration
+
+- **Type:** knowledge -- this skill is a reference document, not a procedural workflow.
+- **No tools or state** -- consumed as context by other skills and agents.
+- **related_skills:** db-isolation-levels, db-isolation-selection, db-acid-properties, db-mvcc, db-optimistic-locking, prisma-transactions, drizzle-transactions
+
+## Success Criteria
+
+- Developers identify which read phenomenon causes their concurrency bug
+- The minimum isolation level that prevents the identified phenomenon is chosen
+- Write skew scenarios are recognized and addressed with Serializable or explicit locking

package/dist/agents/skills/gemini-cli/db-read-phenomena/skill.yaml

@@ -0,0 +1,42 @@
+name: db-read-phenomena
+version: '1.0.0'
+description: Dirty reads, non-repeatable reads, phantom reads, and serialization anomalies explained with concrete examples
+stability: static
+cognitive_mode: advisory-guide
+type: knowledge
+tier: 3
+triggers:
+  - manual
+platforms:
+  - claude-code
+  - gemini-cli
+  - cursor
+  - codex
+tools: []
+paths: []
+related_skills:
+  - db-isolation-levels
+  - db-isolation-selection
+  - db-acid-properties
+  - db-mvcc
+  - db-optimistic-locking
+  - prisma-transactions
+  - drizzle-transactions
+stack_signals:
+  - postgresql
+  - mysql
+  - database
+keywords:
+  - dirty-read
+  - non-repeatable-read
+  - phantom-read
+  - write-skew
+  - serialization-anomaly
+  - read-phenomena
+metadata:
+  author: community
+  upstream: postgresql.org/docs/current/transaction-iso.html#XACT-READ-PHENOMENA
+state:
+  persistent: false
+  files: []
+depends_on: []