@sylix/coworker 2.0.11 → 2.0.12

package/dist/skills/defaults/data-engineering/spark.md

@@ -0,0 +1,411 @@
+---
+name: spark-optimization
+description: Optimize Apache Spark jobs with partitioning, caching, shuffle optimization, and memory tuning. Use when improving Spark performance, debugging slow jobs, or scaling data processing pipelines.
+---
+
+# Apache Spark Optimization
+
+Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
+
+## When to Use This Skill
+
+- Optimizing slow Spark jobs
+- Tuning memory and executor configuration
+- Implementing efficient partitioning strategies
+- Debugging Spark performance issues
+- Scaling Spark pipelines for large datasets
+- Reducing shuffle and data skew
+
+## Core Concepts
+
+### 1. Spark Execution Model
+
+```
+Driver Program
+    ↓
+Job (triggered by action)
+    ↓
+Stages (separated by shuffles)
+    ↓
+Tasks (one per partition)
+```
+
+### 2. Key Performance Factors
+
+| Factor            | Impact                | Solution                      |
+| ----------------- | --------------------- | ----------------------------- |
+| **Shuffle**       | Network I/O, disk I/O | Minimize wide transformations |
+| **Data Skew**     | Uneven task duration  | Salting, broadcast joins      |
+| **Serialization** | CPU overhead          | Use Kryo, columnar formats    |
+| **Memory**        | GC pressure, spills   | Tune executor memory          |
+| **Partitions**    | Parallelism           | Right-size partitions         |
+
+## Quick Start
+
+```python
+from pyspark.sql import SparkSession
+from pyspark.sql import functions as F
+
+# Create optimized Spark session
+spark = (SparkSession.builder
+    .appName("OptimizedJob")
+    .config("spark.sql.adaptive.enabled", "true")
+    .config("spark.sql.adaptive.coalescePartitions.enabled", "true")
+    .config("spark.sql.adaptive.skewJoin.enabled", "true")
+    .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
+    .config("spark.sql.shuffle.partitions", "200")
+    .getOrCreate())
+
+# Read with optimized settings
+df = (spark.read
+    .format("parquet")
+    .option("mergeSchema", "false")
+    .load("s3://bucket/data/"))
+
+# Efficient transformations
+result = (df
+    .filter(F.col("date") >= "2024-01-01")
+    .select("id", "amount", "category")
+    .groupBy("category")
+    .agg(F.sum("amount").alias("total")))
+
+result.write.mode("overwrite").parquet("s3://bucket/output/")
+```
+
+## Patterns
+
+### Pattern 1: Optimal Partitioning
+
+```python
+# Calculate optimal partition count
+def calculate_partitions(data_size_gb: float, partition_size_mb: int = 128) -> int:
+    """
+    Optimal partition size: 128MB - 256MB
+    Too few: Under-utilization, memory pressure
+    Too many: Task scheduling overhead
+    """
+    return max(int(data_size_gb * 1024 / partition_size_mb), 1)
+
+# Repartition for even distribution
+df_repartitioned = df.repartition(200, "partition_key")
+
+# Coalesce to reduce partitions (no shuffle)
+df_coalesced = df.coalesce(100)
+
+# Partition pruning with predicate pushdown
+df = (spark.read.parquet("s3://bucket/data/")
+    .filter(F.col("date") == "2024-01-01"))  # Spark pushes this down
+
+# Write with partitioning for future queries
+(df.write
+    .partitionBy("year", "month", "day")
+    .mode("overwrite")
+    .parquet("s3://bucket/partitioned_output/"))
+```
+
+### Pattern 2: Join Optimization
+
+```python
+from pyspark.sql import functions as F
+from pyspark.sql.types import *
+
+# 1. Broadcast Join - Small table joins
+# Best when: One side < 10MB (configurable)
+small_df = spark.read.parquet("s3://bucket/small_table/")  # < 10MB
+large_df = spark.read.parquet("s3://bucket/large_table/")  # TBs
+
+# Explicit broadcast hint
+result = large_df.join(
+    F.broadcast(small_df),
+    on="key",
+    how="left"
+)
+
+# 2. Sort-Merge Join - Default for large tables
+# Requires shuffle, but handles any size
+result = large_df1.join(large_df2, on="key", how="inner")
+
+# 3. Bucket Join - Pre-sorted, no shuffle at join time
+# Write bucketed tables
+(df.write
+    .bucketBy(200, "customer_id")
+    .sortBy("customer_id")
+    .mode("overwrite")
+    .saveAsTable("bucketed_orders"))
+
+# Join bucketed tables (no shuffle!)
+orders = spark.table("bucketed_orders")
+customers = spark.table("bucketed_customers")  # Same bucket count
+result = orders.join(customers, on="customer_id")
+
+# 4. Skew Join Handling
+# Enable AQE skew join optimization
+spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
+spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor", "5")
+spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "256MB")
+
+# Manual salting for severe skew
+def salt_join(df_skewed, df_other, key_col, num_salts=10):
+    """Add salt to distribute skewed keys"""
+    # Add salt to skewed side
+    df_salted = df_skewed.withColumn(
+        "salt",
+        (F.rand() * num_salts).cast("int")
+    ).withColumn(
+        "salted_key",
+        F.concat(F.col(key_col), F.lit("_"), F.col("salt"))
+    )
+
+    # Explode other side with all salts
+    df_exploded = df_other.crossJoin(
+        spark.range(num_salts).withColumnRenamed("id", "salt")
+    ).withColumn(
+        "salted_key",
+        F.concat(F.col(key_col), F.lit("_"), F.col("salt"))
+    )
+
+    # Join on salted key
+    return df_salted.join(df_exploded, on="salted_key", how="inner")
+```
+
+### Pattern 3: Caching and Persistence
+
+```python
+from pyspark import StorageLevel
+
+# Cache when reusing DataFrame multiple times
+df = spark.read.parquet("s3://bucket/data/")
+df_filtered = df.filter(F.col("status") == "active")
+
+# Cache in memory (MEMORY_AND_DISK is default)
+df_filtered.cache()
+
+# Or with specific storage level
+df_filtered.persist(StorageLevel.MEMORY_AND_DISK_SER)
+
+# Force materialization
+df_filtered.count()
+
+# Use in multiple actions
+agg1 = df_filtered.groupBy("category").count()
+agg2 = df_filtered.groupBy("region").sum("amount")
+
+# Unpersist when done
+df_filtered.unpersist()
+
+# Storage levels explained:
+# MEMORY_ONLY - Fast, but may not fit
+# MEMORY_AND_DISK - Spills to disk if needed (recommended)
+# MEMORY_ONLY_SER - Serialized, less memory, more CPU
+# DISK_ONLY - When memory is tight
+# OFF_HEAP - Tungsten off-heap memory
+
+# Checkpoint for complex lineage
+spark.sparkContext.setCheckpointDir("s3://bucket/checkpoints/")
+df_complex = (df
+    .join(other_df, "key")
+    .groupBy("category")
+    .agg(F.sum("amount")))
+df_complex.checkpoint()  # Breaks lineage, materializes
+```
+
+### Pattern 4: Memory Tuning
+
+```python
+# Executor memory configuration
+# spark-submit --executor-memory 8g --executor-cores 4
+
+# Memory breakdown (8GB executor):
+# - spark.memory.fraction = 0.6 (60% = 4.8GB for execution + storage)
+#   - spark.memory.storageFraction = 0.5 (50% of 4.8GB = 2.4GB for cache)
+#   - Remaining 2.4GB for execution (shuffles, joins, sorts)
+# - 40% = 3.2GB for user data structures and internal metadata
+
+spark = (SparkSession.builder
+    .config("spark.executor.memory", "8g")
+    .config("spark.executor.memoryOverhead", "2g")  # For non-JVM memory
+    .config("spark.memory.fraction", "0.6")
+    .config("spark.memory.storageFraction", "0.5")
+    .config("spark.sql.shuffle.partitions", "200")
+    # For memory-intensive operations
+    .config("spark.sql.autoBroadcastJoinThreshold", "50MB")
+    # Prevent OOM on large shuffles
+    .config("spark.sql.files.maxPartitionBytes", "128MB")
+    .getOrCreate())
+
+# Monitor memory usage
+def print_memory_usage(spark):
+    """Print current memory usage"""
+    sc = spark.sparkContext
+    for executor in sc._jsc.sc().getExecutorMemoryStatus().keySet().toArray():
+        mem_status = sc._jsc.sc().getExecutorMemoryStatus().get(executor)
+        total = mem_status._1() / (1024**3)
+        free = mem_status._2() / (1024**3)
+        print(f"{executor}: {total:.2f}GB total, {free:.2f}GB free")
+```
+
+### Pattern 5: Shuffle Optimization
+
+```python
+# Reduce shuffle data size
+spark.conf.set("spark.sql.shuffle.partitions", "auto")  # With AQE
+spark.conf.set("spark.shuffle.compress", "true")
+spark.conf.set("spark.shuffle.spill.compress", "true")
+
+# Pre-aggregate before shuffle
+df_optimized = (df
+    # Local aggregation first (combiner)
+    .groupBy("key", "partition_col")
+    .agg(F.sum("value").alias("partial_sum"))
+    # Then global aggregation
+    .groupBy("key")
+    .agg(F.sum("partial_sum").alias("total")))
+
+# Avoid shuffle with map-side operations
+# BAD: Shuffle for each distinct
+distinct_count = df.select("category").distinct().count()
+
+# GOOD: Approximate distinct (no shuffle)
+approx_count = df.select(F.approx_count_distinct("category")).collect()[0][0]
+
+# Use coalesce instead of repartition when reducing partitions
+df_reduced = df.coalesce(10)  # No shuffle
+
+# Optimize shuffle with compression
+spark.conf.set("spark.io.compression.codec", "lz4")  # Fast compression
+```
+
+### Pattern 6: Data Format Optimization
+
+```python
+# Parquet optimizations
+(df.write
+    .option("compression", "snappy")  # Fast compression
+    .option("parquet.block.size", 128 * 1024 * 1024)  # 128MB row groups
+    .parquet("s3://bucket/output/"))
+
+# Column pruning - only read needed columns
+df = (spark.read.parquet("s3://bucket/data/")
+    .select("id", "amount", "date"))  # Spark only reads these columns
+
+# Predicate pushdown - filter at storage level
+df = (spark.read.parquet("s3://bucket/partitioned/year=2024/")
+    .filter(F.col("status") == "active"))  # Pushed to Parquet reader
+
+# Delta Lake optimizations
+(df.write
+    .format("delta")
+    .option("optimizeWrite", "true")  # Bin-packing
+    .option("autoCompact", "true")  # Compact small files
+    .mode("overwrite")
+    .save("s3://bucket/delta_table/"))
+
+# Z-ordering for multi-dimensional queries
+spark.sql("""
+    OPTIMIZE delta.`s3://bucket/delta_table/`
+    ZORDER BY (customer_id, date)
+""")
+```
+
+### Pattern 7: Monitoring and Debugging
+
+```python
+# Enable detailed metrics
+spark.conf.set("spark.sql.codegen.wholeStage", "true")
+spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true")
+
+# Explain query plan
+df.explain(mode="extended")
+# Modes: simple, extended, codegen, cost, formatted
+
+# Get physical plan statistics
+df.explain(mode="cost")
+
+# Monitor task metrics
+def analyze_stage_metrics(spark):
+    """Analyze recent stage metrics"""
+    status_tracker = spark.sparkContext.statusTracker()
+
+    for stage_id in status_tracker.getActiveStageIds():
+        stage_info = status_tracker.getStageInfo(stage_id)
+        print(f"Stage {stage_id}:")
+        print(f"  Tasks: {stage_info.numTasks}")
+        print(f"  Completed: {stage_info.numCompletedTasks}")
+        print(f"  Failed: {stage_info.numFailedTasks}")
+
+# Identify data skew
+def check_partition_skew(df):
+    """Check for partition skew"""
+    partition_counts = (df
+        .withColumn("partition_id", F.spark_partition_id())
+        .groupBy("partition_id")
+        .count()
+        .orderBy(F.desc("count")))
+
+    partition_counts.show(20)
+
+    stats = partition_counts.select(
+        F.min("count").alias("min"),
+        F.max("count").alias("max"),
+        F.avg("count").alias("avg"),
+        F.stddev("count").alias("stddev")
+    ).collect()[0]
+
+    skew_ratio = stats["max"] / stats["avg"]
+    print(f"Skew ratio: {skew_ratio:.2f}x (>2x indicates skew)")
+```
+
+## Configuration Cheat Sheet
+
+```python
+# Production configuration template
+spark_configs = {
+    # Adaptive Query Execution (AQE)
+    "spark.sql.adaptive.enabled": "true",
+    "spark.sql.adaptive.coalescePartitions.enabled": "true",
+    "spark.sql.adaptive.skewJoin.enabled": "true",
+
+    # Memory
+    "spark.executor.memory": "8g",
+    "spark.executor.memoryOverhead": "2g",
+    "spark.memory.fraction": "0.6",
+    "spark.memory.storageFraction": "0.5",
+
+    # Parallelism
+    "spark.sql.shuffle.partitions": "200",
+    "spark.default.parallelism": "200",
+
+    # Serialization
+    "spark.serializer": "org.apache.spark.serializer.KryoSerializer",
+    "spark.sql.execution.arrow.pyspark.enabled": "true",
+
+    # Compression
+    "spark.io.compression.codec": "lz4",
+    "spark.shuffle.compress": "true",
+
+    # Broadcast
+    "spark.sql.autoBroadcastJoinThreshold": "50MB",
+
+    # File handling
+    "spark.sql.files.maxPartitionBytes": "128MB",
+    "spark.sql.files.openCostInBytes": "4MB",
+}
+```
+
+## Best Practices
+
+### Do's
+
+- **Enable AQE** - Adaptive query execution handles many issues
+- **Use Parquet/Delta** - Columnar formats with compression
+- **Broadcast small tables** - Avoid shuffle for small joins
+- **Monitor Spark UI** - Check for skew, spills, GC
+- **Right-size partitions** - 128MB - 256MB per partition
+
+### Don'ts
+
+- **Don't collect large data** - Keep data distributed
+- **Don't use UDFs unnecessarily** - Use built-in functions
+- **Don't over-cache** - Memory is limited
+- **Don't ignore data skew** - It dominates job time
+- **Don't use `.count()` for existence** - Use `.take(1)` or `.isEmpty()`

package/dist/skills/defaults/database/postgresql.md

@@ -0,0 +1,202 @@
+---
+name: postgresql-table-design
+description: Design a PostgreSQL-specific schema. Covers best-practices, data types, indexing, constraints, performance patterns, and advanced features
+---
+
+# PostgreSQL Table Design
+
+## Core Rules
+
+- Define a **PRIMARY KEY** for reference tables (users, orders, etc.). Not always needed for time-series/event/log data. When used, prefer `BIGINT GENERATED ALWAYS AS IDENTITY`; use `UUID` only when global uniqueness/opacity is needed.
+- **Normalize first (to 3NF)** to eliminate data redundancy and update anomalies; denormalize **only** for measured, high-ROI reads where join performance is proven problematic. Premature denormalization creates maintenance burden.
+- Add **NOT NULL** everywhere it's semantically required; use **DEFAULT**s for common values.
+- Create **indexes for access paths you actually query**: PK/unique (auto), **FK columns (manual!)**, frequent filters/sorts, and join keys.
+- Prefer **TIMESTAMPTZ** for event time; **NUMERIC** for money; **TEXT** for strings; **BIGINT** for integer values, **DOUBLE PRECISION** for floats (or `NUMERIC` for exact decimal arithmetic).
+
+## PostgreSQL "Gotchas"
+
+- **Identifiers**: unquoted → lowercased. Avoid quoted/mixed-case names. Convention: use `snake_case` for table/column names.
+- **Unique + NULLs**: UNIQUE allows multiple NULLs. Use `UNIQUE (...) NULLS NOT DISTINCT` (PG15+) to restrict to one NULL.
+- **FK indexes**: PostgreSQL **does not** auto-index FK columns. Add them.
+- **No silent coercions**: length/precision overflows error out (no truncation). Example: inserting 999 into `NUMERIC(2,0)` fails with error, unlike some databases that silently truncate or round.
+- **Sequences/identity have gaps** (normal; don't "fix"). Rollbacks, crashes, and concurrent transactions create gaps in ID sequences (1, 2, 5, 6...). This is expected behavior—don't try to make IDs consecutive.
+- **Heap storage**: no clustered PK by default (unlike SQL Server/MySQL InnoDB); `CLUSTER` is one-off reorganization, not maintained on subsequent inserts. Row order on disk is insertion order unless explicitly clustered.
+- **MVCC**: updates/deletes leave dead tuples; vacuum handles them—design to avoid hot wide-row churn.
+
+## Data Types
+
+- **IDs**: `BIGINT GENERATED ALWAYS AS IDENTITY` preferred (`GENERATED BY DEFAULT` also fine); `UUID` when merging/federating/used in a distributed system or for opaque IDs. Generate with `uuidv7()` (preferred if using PG18+) or `gen_random_uuid()` (if using an older PG version).
+- **Integers**: prefer `BIGINT` unless storage space is critical; `INTEGER` for smaller ranges; avoid `SMALLINT` unless constrained.
+- **Floats**: prefer `DOUBLE PRECISION` over `REAL` unless storage space is critical. Use `NUMERIC` for exact decimal arithmetic.
+- **Strings**: prefer `TEXT`; if length limits needed, use `CHECK (LENGTH(col) <= n)` instead of `VARCHAR(n)`; avoid `CHAR(n)`. Use `BYTEA` for binary data. Large strings/binary (>2KB default threshold) automatically stored in TOAST with compression. TOAST storage: `PLAIN` (no TOAST), `EXTENDED` (compress + out-of-line), `EXTERNAL` (out-of-line, no compress), `MAIN` (compress, keep in-line if possible). Default `EXTENDED` usually optimal. Control with `ALTER TABLE tbl ALTER COLUMN col SET STORAGE strategy` and `ALTER TABLE tbl SET (toast_tuple_target = 4096)` for threshold. Case-insensitive: for locale/accent handling use non-deterministic collations; for plain ASCII use expression indexes on `LOWER(col)` (preferred unless column needs case-insensitive PK/FK/UNIQUE) or `CITEXT`.
+- **Money**: `NUMERIC(p,s)` (never float).
+- **Time**: `TIMESTAMPTZ` for timestamps; `DATE` for date-only; `INTERVAL` for durations. Avoid `TIMESTAMP` (without timezone). Use `now()` for transaction start time, `clock_timestamp()` for current wall-clock time.
+- **Booleans**: `BOOLEAN` with `NOT NULL` constraint unless tri-state values are required.
+- **Enums**: `CREATE TYPE ... AS ENUM` for small, stable sets (e.g. US states, days of week). For business-logic-driven and evolving values (e.g. order statuses) → use TEXT (or INT) + CHECK or lookup table.
+- **Arrays**: `TEXT[]`, `INTEGER[]`, etc. Use for ordered lists where you query elements. Index with **GIN** for containment (`@>`, `<@`) and overlap (`&&`) queries. Access: `arr[1]` (1-indexed), `arr[1:3]` (slicing). Good for tags, categories; avoid for relations—use junction tables instead. Literal syntax: `'{val1,val2}'` or `ARRAY[val1,val2]`.
+- **Range types**: `daterange`, `numrange`, `tstzrange` for intervals. Support overlap (`&&`), containment (`@>`), operators. Index with **GiST**. Good for scheduling, versioning, numeric ranges. Pick a bounds scheme and use it consistently; prefer `[)` (inclusive/exclusive) by default.
+- **Network types**: `INET` for IP addresses, `CIDR` for network ranges, `MACADDR` for MAC addresses. Support network operators (`<<`, `>>`, `&&`).
+- **Geometric types**: `POINT`, `LINE`, `POLYGON`, `CIRCLE` for 2D spatial data. Index with **GiST**. Consider **PostGIS** for advanced spatial features.
+- **Text search**: `TSVECTOR` for full-text search documents, `TSQUERY` for search queries. Index `tsvector` with **GIN**. Always specify language: `to_tsvector('english', col)` and `to_tsquery('english', 'query')`. Never use single-argument versions. This applies to both index expressions and queries.
+- **Domain types**: `CREATE DOMAIN email AS TEXT CHECK (VALUE ~ '^[^@]+@[^@]+$')` for reusable custom types with validation. Enforces constraints across tables.
+- **Composite types**: `CREATE TYPE address AS (street TEXT, city TEXT, zip TEXT)` for structured data within columns. Access with `(col).field` syntax.
+- **JSONB**: preferred over JSON; index with **GIN**. Use only for optional/semi-structured attrs. ONLY use JSON if the original ordering of the contents MUST be preserved.
+- **Vector types**: `vector` type by `pgvector` for vector similarity search for embeddings.
+
+### Do not use the following data types
+
+- DO NOT use `timestamp` (without time zone); DO use `timestamptz` instead.
+- DO NOT use `char(n)` or `varchar(n)`; DO use `text` instead.
+- DO NOT use `money` type; DO use `numeric` instead.
+- DO NOT use `timetz` type; DO use `timestamptz` instead.
+- DO NOT use `timestamptz(0)` or any other precision specification; DO use `timestamptz` instead
+- DO NOT use `serial` type; DO use `generated always as identity` instead.
+
+## Table Types
+
+- **Regular**: default; fully durable, logged.
+- **TEMPORARY**: session-scoped, auto-dropped, not logged. Faster for scratch work.
+- **UNLOGGED**: persistent but not crash-safe. Faster writes; good for caches/staging.
+
+## Row-Level Security
+
+Enable with `ALTER TABLE tbl ENABLE ROW LEVEL SECURITY`. Create policies: `CREATE POLICY user_access ON orders FOR SELECT TO app_users USING (user_id = current_user_id())`. Built-in user-based access control at the row level.
+
+## Constraints
+
+- **PK**: implicit UNIQUE + NOT NULL; creates a B-tree index.
+- **FK**: specify `ON DELETE/UPDATE` action (`CASCADE`, `RESTRICT`, `SET NULL`, `SET DEFAULT`). Add explicit index on referencing column—speeds up joins and prevents locking issues on parent deletes/updates. Use `DEFERRABLE INITIALLY DEFERRED` for circular FK dependencies checked at transaction end.
+- **UNIQUE**: creates a B-tree index; allows multiple NULLs unless `NULLS NOT DISTINCT` (PG15+). Standard behavior: `(1, NULL)` and `(1, NULL)` are allowed. With `NULLS NOT DISTINCT`: only one `(1, NULL)` allowed. Prefer `NULLS NOT DISTINCT` unless you specifically need duplicate NULLs.
+- **CHECK**: row-local constraints; NULL values pass the check (three-valued logic). Example: `CHECK (price > 0)` allows NULL prices. Combine with `NOT NULL` to enforce: `price NUMERIC NOT NULL CHECK (price > 0)`.
+- **EXCLUDE**: prevents overlapping values using operators. `EXCLUDE USING gist (room_id WITH =, booking_period WITH &&)` prevents double-booking rooms. Requires appropriate index type (often GiST).
+
+## Indexing
+
+- **B-tree**: default for equality/range queries (`=`, `<`, `>`, `BETWEEN`, `ORDER BY`)
+- **Composite**: order matters—index used if equality on leftmost prefix (`WHERE a = ? AND b > ?` uses index on `(a,b)`, but `WHERE b = ?` does not). Put most selective/frequently filtered columns first.
+- **Covering**: `CREATE INDEX ON tbl (id) INCLUDE (name, email)` - includes non-key columns for index-only scans without visiting table.
+- **Partial**: for hot subsets (`WHERE status = 'active'` → `CREATE INDEX ON tbl (user_id) WHERE status = 'active'`). Any query with `status = 'active'` can use this index.
+- **Expression**: for computed search keys (`CREATE INDEX ON tbl (LOWER(email))`). Expression must match exactly in WHERE clause: `WHERE LOWER(email) = 'user@example.com'`.
+- **GIN**: JSONB containment/existence, arrays (`@>`, `?`), full-text search (`@@`)
+- **GiST**: ranges, geometry, exclusion constraints
+- **BRIN**: very large, naturally ordered data (time-series)—minimal storage overhead. Effective when row order on disk correlates with indexed column (insertion order or after `CLUSTER`).
+
+## Partitioning
+
+- Use for very large tables (>100M rows) where queries consistently filter on partition key (often time/date).
+- Alternate use: use for tables where data maintenance tasks dictates e.g. data pruned or bulk replaced periodically
+- **RANGE**: common for time-series (`PARTITION BY RANGE (created_at)`). Create partitions: `CREATE TABLE logs_2024_01 PARTITION OF logs FOR VALUES FROM ('2024-01-01') TO ('2024-02-01')`. **TimescaleDB** automates time-based or ID-based partitioning with retention policies and compression.
+- **LIST**: for discrete values (`PARTITION BY LIST (region)`). Example: `FOR VALUES IN ('us-east', 'us-west')`.
+- **HASH**: for even distribution when no natural key (`PARTITION BY HASH (user_id)`). Creates N partitions with modulus.
+- **Constraint exclusion**: requires `CHECK` constraints on partitions for query planner to prune. Auto-created for declarative partitioning (PG10+).
+- Prefer declarative partitioning or hypertables. Do NOT use table inheritance.
+- **Limitations**: no global UNIQUE constraints—include partition key in PK/UNIQUE. FKs from partitioned tables not supported; use triggers.
+
+## Special Considerations
+
+### Update-Heavy Tables
+
+- **Separate hot/cold columns**—put frequently updated columns in separate table to minimize bloat.
+- **Use `fillfactor=90`** to leave space for HOT updates that avoid index maintenance.
+- **Avoid updating indexed columns**—prevents beneficial HOT updates.
+- **Partition by update patterns**—separate frequently updated rows in a different partition from stable data.
+
+### Insert-Heavy Workloads
+
+- **Minimize indexes**—only create what you query; every index slows inserts.
+- **Use `COPY` or multi-row `INSERT`** instead of single-row inserts.
+- **UNLOGGED tables** for rebuildable staging data—much faster writes.
+- **Defer index creation** for bulk loads—>drop index, load data, recreate indexes.
+- **Partition by time/hash** to distribute load. **TimescaleDB** automates partitioning and compression of insert-heavy data.
+- **Use a natural key for primary key** such as a (timestamp, device_id) if enforcing global uniqueness is important many insert-heavy tables don't need a primary key at all.
+- If you do need a surrogate key, **Prefer `BIGINT GENERATED ALWAYS AS IDENTITY` over `UUID`**.
+
+### Upsert-Friendly Design
+
+- **Requires UNIQUE index** on conflict target columns—`ON CONFLICT (col1, col2)` needs exact matching unique index (partial indexes don't work).
+- **Use `EXCLUDED.column`** to reference would-be-inserted values; only update columns that actually changed to reduce write overhead.
+- **`DO NOTHING` faster** than `DO UPDATE` when no actual update needed.
+
+### Safe Schema Evolution
+
+- **Transactional DDL**: most DDL operations can run in transactions and be rolled back—`BEGIN; ALTER TABLE...; ROLLBACK;` for safe testing.
+- **Concurrent index creation**: `CREATE INDEX CONCURRENTLY` avoids blocking writes but can't run in transactions.
+- **Volatile defaults cause rewrites**: adding `NOT NULL` columns with volatile defaults (e.g., `now()`, `gen_random_uuid()`) rewrites entire table. Non-volatile defaults are fast.
+- **Drop constraints before columns**: `ALTER TABLE DROP CONSTRAINT` then `DROP COLUMN` to avoid dependency issues.
+- **Function signature changes**: `CREATE OR REPLACE` with different arguments creates overloads, not replacements. DROP old version if no overload desired.
+
+## Generated Columns
+
+- `... GENERATED ALWAYS AS (<expr>) STORED` for computed, indexable fields. PG18+ adds `VIRTUAL` columns (computed on read, not stored).
+
+## Extensions
+
+- **`pgcrypto`**: `crypt()` for password hashing.
+- **`uuid-ossp`**: alternative UUID functions; prefer `pgcrypto` for new projects.
+- **`pg_trgm`**: fuzzy text search with `%` operator, `similarity()` function. Index with GIN for `LIKE '%pattern%'` acceleration.
+- **`citext`**: case-insensitive text type. Prefer expression indexes on `LOWER(col)` unless you need case-insensitive constraints.
+- **`btree_gin`/`btree_gist`**: enable mixed-type indexes (e.g., GIN index on both JSONB and text columns).
+- **`hstore`**: key-value pairs; mostly superseded by JSONB but useful for simple string mappings.
+- **`timescaledb`**: essential for time-series—automated partitioning, retention, compression, continuous aggregates.
+- **`postgis`**: comprehensive geospatial support beyond basic geometric types—essential for location-based applications.
+- **`pgvector`**: vector similarity search for embeddings.
+- **`pgaudit`**: audit logging for all database activity.
+
+## JSONB Guidance
+
+- Prefer `JSONB` with **GIN** index.
+- Default: `CREATE INDEX ON tbl USING GIN (jsonb_col);` → accelerates:
+  - **Containment** `jsonb_col @> '{"k":"v"}'`
+  - **Key existence** `jsonb_col ? 'k'`, **any/all keys** `?|`, `?&`
+  - **Path containment** on nested docs
+  - **Disjunction** `jsonb_col @> ANY(ARRAY['{"status":"active"}', '{"status":"pending"}'])`
+- Heavy `@>` workloads: consider opclass `jsonb_path_ops` for smaller/faster containment-only indexes:
+  - `CREATE INDEX ON tbl USING GIN (jsonb_col jsonb_path_ops);`
+  - **Trade-off**: loses support for key existence (`?`, `?|`, `?&`) queries—only supports containment (`@>`)
+- Equality/range on a specific scalar field: extract and index with B-tree (generated column or expression):
+  - `ALTER TABLE tbl ADD COLUMN price INT GENERATED ALWAYS AS ((jsonb_col->>'price')::INT) STORED;`
+  - `CREATE INDEX ON tbl (price);`
+  - Prefer queries like `WHERE price BETWEEN 100 AND 500` (uses B-tree) over `WHERE (jsonb_col->>'price')::INT BETWEEN 100 AND 500` without index.
+- Arrays inside JSONB: use GIN + `@>` for containment (e.g., tags). Consider `jsonb_path_ops` if only doing containment.
+- Keep core relations in tables; use JSONB for optional/variable attributes.
+- Use constraints to limit allowed JSONB values in a column e.g. `config JSONB NOT NULL CHECK(jsonb_typeof(config) = 'object')`
+
+## Examples
+
+### Users
+
+```sql
+CREATE TABLE users (
+  user_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+  email TEXT NOT NULL UNIQUE,
+  name TEXT NOT NULL,
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+CREATE UNIQUE INDEX ON users (LOWER(email));
+CREATE INDEX ON users (created_at);
+```
+
+### Orders
+
+```sql
+CREATE TABLE orders (
+  order_id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
+  user_id BIGINT NOT NULL REFERENCES users(user_id),
+  status TEXT NOT NULL DEFAULT 'PENDING' CHECK (status IN ('PENDING','PAID','CANCELED')),
+  total NUMERIC(10,2) NOT NULL CHECK (total > 0),
+  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
+);
+CREATE INDEX ON orders (user_id);
+CREATE INDEX ON orders (created_at);
+```
+
+### JSONB
+
+```sql
+CREATE TABLE profiles (
+  user_id BIGINT PRIMARY KEY REFERENCES users(user_id),
+  attrs JSONB NOT NULL DEFAULT '{}',
+  theme TEXT GENERATED ALWAYS AS (attrs->>'theme') STORED
+);
+CREATE INDEX profiles_attrs_gin ON profiles USING GIN (attrs);
+```